From 1fdb681e82fbcf204072508b61a6a4951dea0cdf Mon Sep 17 00:00:00 2001 From: mfw78 Date: Sat, 25 Jul 2026 03:18:11 +0000 Subject: [PATCH 1/2] docs: prune and de-drift deployment and production docs Trim the deployment and production corpus to the current contract and fix ownership: docs/deployment.md owns the engine.toml reference, module artefact builds, and local runs; docs/production.md owns the production deploy (systemd, backup, observability wiring); docs/deployment/docker.md owns containers; docs/deployment/multi-chain.md owns multi-chain config; docs/06-production-hardening.md owns the hardening design facts and stops restating deploy steps. Each of the others links the owner rather than repeating. Verify every config key, default, metric name, port, and binary against the code on dev/m1: the production binary is shepherd (not nexum-cli), resource caps live under [limits] (not [engine.limits]), the redb file is local-store.redb, the metric prefix is shepherd_* with the eleven-metric surface the runtime actually emits, and the orderbook URL override is the cow-venue adapter's own [config] orderbook-url rather than an engine-side extensions.cow table. Drop the fabricated cli subcommands, epoch mechanism, health endpoint, and nexum_* metric table. Purge em dashes and repo-name and planning drift. --- README.md | 87 ++--- docs/06-production-hardening.md | 468 ++---------------------- docs/deployment.md | 247 +++---------- docs/deployment/docker.md | 181 ++------- docs/deployment/multi-chain.md | 252 ++----------- docs/production.md | 627 +++++--------------------------- 6 files changed, 277 insertions(+), 1585 deletions(-) diff --git a/README.md b/README.md index 21aa9c19..ff627fd7 100644 --- a/README.md +++ b/README.md @@ -3,55 +3,41 @@ [![CI](https://github.com/nullislabs/shepherd/actions/workflows/ci.yml/badge.svg)](https://github.com/nullislabs/shepherd/actions/workflows/ci.yml) [![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE) -**Shepherd is a CoW Protocol-extended [Nexum Runtime](https://github.com/nullislabs): on-chain automation that runs as sandboxed WebAssembly, not scripts.** +Shepherd is a CoW Protocol extension of the [Nexum Runtime](https://github.com/nullislabs): on-chain automation that runs as sandboxed WebAssembly. -The Nexum Runtime executes untrusted automation as WASM components against the `nexum:host` WIT contract. Every module receives exactly the host capabilities it declares in its manifest and nothing more - no ambient filesystem or network. Execution is metered by fuel and epoch, memory-capped, and transactional per event: state commits on success and rolls back on trap. Modules are distributed content-addressed and verified by hash. There is no central service to depend on; you run the node. +The Nexum Runtime executes untrusted automation as WASM components against the `nexum:host` WIT contract. A module receives only the host capabilities it declares in its manifest: no ambient filesystem or network. Execution is metered by fuel and epoch, memory-capped, and transactional per event: state commits on success and rolls back on trap. Modules are content-addressed and verified by hash. There is no central service; you run the node. -Shepherd extends that runtime with `shepherd:cow` - CoW Protocol order APIs and submission - so a TWAP, EthFlow, or ComposableCoW watch-tower is an ordinary module, not a special case baked into the engine. Write the strategy once as a component; the runtime supervises, restarts, meters, and sandboxes it. +Shepherd registers the `videre:venue` platform and bundles the `cow-venue` adapter, so a keeper module submits CoW Protocol orders through the venue registry rather than through engine-baked logic. A keeper watching `ComposableCoW` or `EthFlow` is an ordinary module. -A module built against the universal `nexum:host` world runs on any Nexum-compatible host. A module built against `shepherd:cow` additionally gains CoW Protocol access and requires a Shepherd host. +A module built against `nexum:host` runs on any Nexum-compatible host. CoW order submission additionally needs a host that registers the venue platform, which the `shepherd` binary does. -> **Pre-release** and under active development. Testnets and lab environments only. - -Looking for the org? See **[github.com/nullislabs](https://github.com/nullislabs)**. - ---- - -## Why - -- **WASM Component Model, not a plugin API** - a WIT-typed host/guest contract with structural isolation and multi-language guests (Rust today; anything that compiles to a component next). -- **Capability-scoped by construction** - a module sees only the host primitives it declares. No filesystem; outbound HTTP only against a per-module allowlist (`wasi:http`); WASI clocks and randomness are linked in ambiently. -- **Metered and transactional** - per-event fuel and epoch limits, a memory cap, and all-or-nothing state. A runaway module cannot starve its neighbours or corrupt its store. -- **Declarative subscriptions** - modules declare their block, log, and cron events in a manifest; the runtime wires and multiplexes the sources. -- **Content-addressed distribution** - modules are fetched by hash (Swarm, IPFS, OCI, HTTPS) and integrity-checked before they load. -- **Self-hosted** - one binary, your keys, your RPC. No centralised dependency. - ---- +> Pre-release, under active development. Testnets and lab environments only. ## Layout | Path | Purpose | | --- | --- | -| `crates/nexum-runtime/` | The **engine** - the Nexum Runtime's reference host: a wasmtime implementation of the `nexum:host` contract. | -| `crates/nexum-launch/` | The generic launcher library - shared CLI, config load, tracing, and the preset-driven launch. | -| `crates/nexum-cli/` | The bare `nexum` binary - the core lattice with no extension payload. | -| `crates/shepherd/` | The `shepherd` binary - the cow composition root registering the videre venue platform. | -| `crates/nexum-sdk/` | Generic guest SDK - the host trait seam, bind macro, chain/config/address helpers, wasi:http `fetch`, and tracing facade for any module. | -| `wit/nexum-host/` | The **`nexum:host`** WIT package - the host/guest contract every engine implements and every module imports. | -| `wit/shepherd-cow/` | The `shepherd:cow` WIT package - the CoW event ABIs of record. | -| `modules/` | Guest modules - TWAP and EthFlow watch-towers, examples, and test fixtures. | +| `crates/nexum-runtime/` | The engine: a wasmtime host implementing the `nexum:host` contract. | +| `crates/nexum-launch/` | Launcher library: shared CLI, config load, tracing, preset launch. | +| `crates/nexum-cli/` | The bare `nexum` binary: the core lattice, no extension payload. | +| `crates/shepherd/` | The `shepherd` binary: the cow composition root registering the videre venue platform and the Prometheus add-on. | +| `crates/nexum-sdk/` | Guest SDK: host trait seam, bind macro, chain/config/address helpers, `wasi:http` fetch, tracing facade. | +| `crates/videre-sdk/` | Venue-platform SDK: the `videre:venue` client and adapter contracts. | +| `crates/cow-venue/` | The bundled CoW venue adapter component. | +| `wit/nexum-host/` | The `nexum:host` WIT package: the host/guest contract. | +| `wit/videre-venue/` | The `videre:venue` WIT package: the venue-adapter contract. | +| `wit/shepherd-cow/` | `cow-events.wit`: the CoW event ABIs of record. | +| `modules/` | Guest modules: TWAP and EthFlow keepers, examples, and test fixtures. | | `docs/` | Architecture and design notes. Start with [`docs/00-overview.md`](docs/00-overview.md). | -> **Engine vs. host.** An *engine* is a concrete implementation that runs WASM components (today `nexum`, a wasmtime daemon). The `nexum:host` WIT package is the *contract* - the host imports a guest sees. Other engines (mobile, browser) can implement the same contract, and modules built against it run on any compliant engine. - ---- +An engine is a concrete implementation that runs WASM components (`nexum`, a wasmtime daemon). The `nexum:host` WIT package is the contract. Modules built against it run on any compliant engine. ## Build from source Shepherd uses [Nix](https://nixos.org/) flakes to pin the toolchain and [just](https://github.com/casey/just) as the task runner. ```sh -nix develop # enter the dev shell (Rust, wasm-tools, just, ...) +nix develop # dev shell (Rust, wasm-tools, just) just build # build the engine and the example module just run # run the engine against the example module just test # unit tests @@ -59,55 +45,30 @@ just test # unit tests Without Nix you need Rust (edition 2024), the `wasm32-wasip2` target, and `wasm-tools`. ---- - ## Running -Single module (development): +Development, a single module against a synthetic event: ```sh nexum [] ``` -Multi-module (production) - `engine.toml` declares RPC endpoints, the state directory, and a `[[modules]]` list: +Production, an `engine.toml` declaring chains, state directory, modules, and adapters: ```sh -nexum --engine-config engine.toml -``` - -A module's own `module.toml` declares its capabilities and event subscriptions: - -```toml -[module] -name = "twap-monitor" -version = "0.1.0" - -[capabilities] -required = ["chain", "local-store", "client"] -optional = ["http"] - -[[subscription]] -kind = "chain-log" -chain_id = 1 -address = "0xfdaFc9d1902f4e0b84f65F49f244b32b31013b74" # ComposableCoW +shepherd --engine-config engine.toml ``` -See [`docs/`](docs) for the full schema and the design corpus - start with [`docs/00-overview.md`](docs/00-overview.md). - ---- +See [`docs/deployment.md`](docs/deployment.md) for the `engine.toml` reference and [`docs/production.md`](docs/production.md) for the production deploy. ## Contributing -Open an issue before non-trivial PRs - this is a pre-release codebase under active churn. Conventional Commits. CI runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`, and per-module `wasm32-wasip2` builds. +Open an issue before non-trivial PRs. Conventional Commits. CI runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`, and per-module `wasm32-wasip2` builds. ## Security -Capability sandboxing, key handling, and order signing are security-critical. Please report vulnerabilities privately rather than in public issues. +Capability sandboxing, key handling, and order signing are security-critical. Report vulnerabilities privately rather than in public issues. ## License AGPL-3.0-or-later © Nullis Labs LLC and contributors. See [LICENSE](LICENSE). - -``` -● AGPL-3.0 · pre-release · Nexum Runtime -``` diff --git a/docs/06-production-hardening.md b/docs/06-production-hardening.md index e1944384..1b66b4c9 100755 --- a/docs/06-production-hardening.md +++ b/docs/06-production-hardening.md @@ -1,473 +1,59 @@ -# Production Hardening & Observability +# Production hardening -## Resource Enforcement +The design facts behind the runtime's resource enforcement, restart policy, RPC resilience, and error model. Deploy procedures live in [`docs/production.md`](./production.md); the metric surface and alert rules in [`docs/production.md`](./production.md) §6-7; containers in [`docs/deployment/docker.md`](./deployment/docker.md). -Four resource dimensions are capped per module. **In 0.2, caps come from compile-time global defaults** (`DEFAULT_FUEL_PER_EVENT = 1_000_000_000`, `DEFAULT_MEMORY_LIMIT = 64 MiB` in `crates/nexum-runtime/src/runtime/limits.rs`); per-module overrides via the manifest's `[module.resources]` section are a future direction (0.3). The mechanism and shape below describe what the 0.2 engine actually enforces - using global values where this doc previously read `module_config.max_*`. +## Resource enforcement -### CPU: Fuel +Four dimensions are capped per module. Caps come from `[limits]` in `engine.toml`, resolving to built-in defaults (`crates/nexum-runtime/src/engine_config.rs`). They apply uniformly to every module; per-module overrides land in 0.3. -Each `on_event` call is budgeted `DEFAULT_FUEL_PER_EVENT` fuel units. Exhaustion traps the call (state rolled back -- see doc 04). The budget prevents infinite loops and excessive computation. +### Fuel -```rust -store.set_fuel(DEFAULT_FUEL_PER_EVENT)?; -``` +Each `on_event` call is granted `fuel_per_event` fuel (default 1_000_000_000, ~1s of compute). Exhaustion traps the call and rolls back its state (doc 04). Fuel is deterministic: the same WASM consumes the same fuel regardless of host speed. It meters only guest instructions. -Fuel is deterministic -- the same WASM code consumes the same fuel regardless of host machine speed. +### Wall-clock deadline -### CPU: Epoch (wall-clock) - -A background Tokio task increments the engine epoch on a fixed interval (e.g. 100ms). Each module store has a deadline: if a callback exceeds N epochs, it yields back to the Tokio runtime. This prevents one module from starving others even if fuel is generous. - -```rust -// Runtime startup -let engine = engine.clone(); -tokio::spawn(async move { - let mut interval = tokio::time::interval(Duration::from_millis(100)); - loop { - interval.tick().await; - engine.increment_epoch(); - } -}); - -// Per-module store setup -store.epoch_deadline_async_yield_and_update(10); // yield after 10 epochs (~1s) -``` +Fuel does not meter time spent in host calls (chain RPC, HTTP, redb). A per-dispatch wall-clock deadline (`event_deadline_secs`, default 120, floor 1) is the backstop: the supervisor runs each dispatch under a `tokio::time::timeout`, and a dispatch that outlives it, guest plus every awaited host call, is cancelled and the module marked dead. Fuel and the deadline are complementary, not redundant. ### Memory -The engine builds a `wasmtime::StoreLimitsBuilder` with `DEFAULT_MEMORY_LIMIT` and attaches it to each module's store at dispatch time (see `crates/nexum-runtime/src/supervisor.rs`). `memory.grow` is denied past the cap. Future direction: a per-module `ResourceLimiter` driven by `module_config.max_memory_bytes` from the manifest; not in 0.2 scope. +The supervisor attaches a `wasmtime::StoreLimitsBuilder` built from `memory_bytes` (default 64 MiB) to each module store; `memory.grow` past the cap is denied. ### Storage -Local-store quota is enforced on the `local-store::set` host path. The 0.2 engine treats the cap as a host-side constant; a manifest-driven `max_state_bytes` is future direction. Rejected with a clear `fault` (see doc 04), not a trap -- the module can handle it gracefully. - -### Summary +The local-store byte quota (`state_bytes`, default 50 MiB) is enforced on the `local-store::set` host path. Overrun is rejected with a `fault.invalid-input`, not a trap, so the module can handle it. | Resource | Mechanism | Failure mode | |----------|-----------|-------------| -| CPU (deterministic) | Fuel | Trap -> rollback -> restart | -| CPU (wall-clock) | Epoch interruption | Yield -> resume or trap | -| Memory | `ResourceLimiter` | `memory.grow` returns -1 | +| CPU (guest instructions) | Fuel | Trap, rollback, restart | +| Wall-clock | Per-dispatch tokio timeout | Cancel, module marked dead | +| Memory | `StoreLimits` | `memory.grow` denied | | Storage | Host-side byte tracking | `local-store::set` returns `fault.invalid-input` | -## Crash Handling & Restart Policy - -### Restart with Exponential Backoff - -When a module's `on_event` (or `init`) traps or returns `Err`: - -``` -Attempt 1: restart after 1s -Attempt 2: restart after 2s -Attempt 3: restart after 4s -Attempt 4: restart after 8s -... -Attempt N: restart after min(2^(N-1), 300)s <- capped at 5 minutes -``` - -A successful `on_event` resets the backoff counter to zero. - -```rust -struct RestartPolicy { - consecutive_failures: u32, - max_backoff: Duration, // 5 minutes - base: Duration, // 1 second -} - -impl RestartPolicy { - fn next_backoff(&self) -> Duration { - let backoff = self.base * 2u32.saturating_pow(self.consecutive_failures - 1); - backoff.min(self.max_backoff) - } - - fn record_success(&mut self) { - self.consecutive_failures = 0; - } - - fn record_failure(&mut self) { - self.consecutive_failures += 1; - } -} -``` - -### Poison Pill Detection - -A module that crashes on every event is a poison pill. After `max_consecutive_failures` (default: 10), the module transitions to `Dead` state: - -```mermaid -flowchart TD - A["Consecutive failures >= 10"] --> B["Module state -> Dead"] - B --> C["All event dispatch stops"] - B --> D["Alert emitted (metric + log)"] - B --> E["Requires manual intervention"] - E --> F["nexum module restart twap-monitor"] - E --> G["nexum module reload twap-monitor\n(re-fetch + recompile)"] -``` - -The threshold is configurable per-module in the manifest under `[module.restart]` (separate from resource caps): - -```toml -[module.restart] -max_consecutive_failures = 10 -``` - -### Restart Scope - -A restart creates a fresh `Store` (clean WASM memory) but reuses the `InstancePre` (no recompilation). The module's `init` is called again. Local-store data persists (doc 04). - -## RPC Resilience - -All RPC I/O flows through alloy providers configured by the runtime operator. The `chain::request` host function (see doc 07) forwards to the provider, which is wrapped with resilience layers using alloy's tower-based middleware. The additive `chain::request-batch` (0.2) routes alloy's `RequestPacket::Batch` to actually batch on the wire. - -### Provider Stack +## Restart policy -```mermaid -flowchart TD - A["Module calls chain::request\n(via alloy Provider in SDK)"] --> B["Host chain::request impl\n-> alloy Provider"] - B --> C["Timeout\n(10s default)"] - C --> D["Retry\n(3 attempts, exponential\nbackoff + jitter)"] - D --> E["Rate Limit\n(per-endpoint)"] - E --> F["Fallback\n(primary -> secondary)"] - F --> G["RPC Endpoint"] +When a module's `init` or `on_event` traps or returns `Err`, the supervisor restarts it after an exponential backoff: 1s, 2s, 4s, 8s, doubling to a 300s (5 min) cap (`runtime/restart_policy.rs`). A restart creates a fresh `Store` (clean WASM memory) but reuses the compiled `InstancePre`; `init` runs again and local-store data persists (doc 04). A successful dispatch resets the counter. - subgraph Tower Layer Stack - C - D - E - F - end -``` +A module that keeps trapping is a poison pill. After `max_failures` traps within a sliding `window_secs` (`[limits.poison]`, default 5 / 600s in `runtime/poison_policy.rs`), the module is quarantined: dispatch to it stops, the `shepherd_module_poisoned` gauge goes to `1`, and a WARN is logged. Quarantine clears only on an engine restart. Venue adapters follow the same policy under the `shepherd_adapter_poisoned` gauge. -### Operator Configuration +## RPC resilience -```toml -[[chains]] -chain_id = 42161 -name = "arbitrum" +RPC I/O flows through one alloy provider per chain, opened from `engine.toml` at boot (`crates/nexum-runtime/src/host/provider_pool.rs`). Each chain has a single `rpc_url`; there is no secondary-endpoint failover. The `chain::request` host function forwards the typed method to the provider. -[[chains.endpoints]] -url = "wss://arb-mainnet.g.alchemy.com/v2/KEY" -priority = 1 +Two layers harden it: -[[chains.endpoints]] -url = "https://arb1.arbitrum.io/rpc" -priority = 2 # fallback +- A transport `RetryBackoffLayer` (10 retries, 300ms base backoff, 100 CU/s pacing) heals transient node blips below the poller, so a momentary hiccup does not force a stream re-open. +- A per-request timeout (`request_timeout_secs`, default 30) bounds each `chain::request`; it does not apply to the long-lived subscription and log-poller streams. -[chains.rpc_policy] -timeout_ms = 10_000 -max_retries = 3 -rate_limit_per_second = 50 -``` +Block following uses `eth_subscribe(newHeads)` on a WebSocket URL and polls `eth_getBlockByNumber` on HTTP; logs poll `eth_getLogs` on either transport. On a dropped WebSocket the event loop reconnects with backoff, then backfills the gap between the last dispatched block and head so modules do not silently miss events. A `resume` chain-log subscription persists its cursor under `last_dispatched_block:{chain_id}` and resumes from it across restarts. -### Subscription Reconnection +## Error model -WebSocket subscriptions (`eth_subscribe`) drop when the connection is lost. The event source manager detects this and reconnects: +Host interfaces surface a common `fault` variant (`wit/nexum-host/types.wit`): `unsupported`, `unavailable`, `denied`, `rate-limited` (carrying `retry-after-ms` guidance), `timeout`, `invalid-input`, `internal`. A fault is a typed, recoverable return the guest can handle; a trap (fuel, memory, panic) is not, and drives the restart path above. -```mermaid -flowchart TD - A["Subscription drops"] --> B["Reconnect with backoff\n(1s, 2s, 4s, ... 30s cap)"] - B --> C["On reconnect: check for missed blocks"] - C --> D["Query eth_blockNumber"] - C --> E["Compare with last dispatched block"] - C --> F["Backfill missed blocks\nvia eth_getBlockByNumber"] - F --> G["Resume live subscription"] -``` +## Structured logging -This ensures modules don't silently miss events during RPC outages. - -## Structured Logging - -### Stack: `tracing` + `tracing-subscriber` - -All runtime logging uses the `tracing` crate with structured fields, output as JSON in production: - -```rust -use tracing::{info, warn, error, instrument, Span}; - -#[instrument(skip(store), fields(module = %module_id, chain_id))] -async fn dispatch_event(module_id: &str, event: &Event, store: &mut Store) { - info!(event_type = %event.type_name(), "dispatching event"); - // ... -} -``` - -### Log Contexts - -Every log line includes: - -| Field | Source | -|-------|--------| -| `module` | Module name from manifest | -| `chain_id` | Chain the event originated from | -| `event_type` | `block` / `logs` / `tick` / `message` | -| `block_number` | For block/log events | -| `level` | trace / debug / info / warn / error | -| `timestamp` | ISO 8601 | -| `span_id` | Tracing span (correlates related logs) | - -### Module Guest Logs - -When a module calls `logging::log(level, message)`, the host writes a `tracing` event tagged with the module's context: - -```rust -impl logging::Host for NexumHostState { - fn log(&mut self, level: Level, message: String) { - let span = tracing::info_span!("module", module = %self.module_id); - let _enter = span.enter(); - match level { - Level::Trace => tracing::trace!("{message}"), - Level::Debug => tracing::debug!("{message}"), - Level::Info => tracing::info!("{message}"), - Level::Warn => tracing::warn!("{message}"), - Level::Error => tracing::error!("{message}"), - } - } -} -``` - -### Output Formats - -| Environment | Format | Config | -|-------------|--------|--------| -| Development | Pretty, coloured | `RUST_LOG=nexum=debug` | -| Production | JSON, one line per event | `--log-format json` | - -```json -{ - "timestamp": "2026-02-18T12:00:00.123Z", - "level": "INFO", - "module": "twap-monitor", - "chain_id": 42161, - "event_type": "block", - "block_number": 19000001, - "message": "posted TWAP part 3/10", - "span_id": "abc123" -} -``` +All runtime logging uses `tracing` with structured fields, emitted as JSON in production (or the pretty format under `--pretty-logs`). Every per-module event carries the module name and, on chain events, the `chain_id` and `block_number`. When a guest calls `logging::log`, the host writes a `tracing` event tagged with the module's context. Log operations (retention, aggregation) are in [`docs/production.md`](./production.md) §5. ## Metrics -### Stack: `metrics` crate + Prometheus exporter - -The `metrics` crate provides a facade (like `log` for logging). We use `metrics-exporter-prometheus` to expose a `/metrics` HTTP endpoint. - -> Note: The OpenTelemetry Prometheus exporter crate is deprecated. The `metrics` + `metrics-exporter-prometheus` combo remains the simplest, most stable path for Prometheus scraping. - -### Metric Definitions - -#### Runtime-level - -| Metric | Type | Labels | Description | -|--------|------|--------|-------------| -| `nexum_modules_loaded` | Gauge | -- | Number of modules currently in Run state | -| `nexum_modules_dead` | Gauge | -- | Number of modules in Dead state | -| `nexum_uptime_seconds` | Counter | -- | Runtime uptime | -| `nexum_content_fetch_total` | Counter | `scheme` | Content store fetches by scheme | -| `nexum_content_fetch_errors` | Counter | `scheme` | Content store fetch failures | - -#### Per-module - -| Metric | Type | Labels | Description | -|--------|------|--------|-------------| -| `nexum_events_dispatched_total` | Counter | `module`, `event_type` | Events dispatched | -| `nexum_events_processed_total` | Counter | `module`, `event_type` | Events successfully processed | -| `nexum_events_failed_total` | Counter | `module`, `event_type` | Events that trapped or returned Err | -| `nexum_event_duration_seconds` | Histogram | `module`, `event_type` | Wall-clock time per on_event call | -| `nexum_fuel_consumed` | Histogram | `module` | Fuel consumed per on_event call | -| `nexum_restarts_total` | Counter | `module` | Total restart count | -| `nexum_consecutive_failures` | Gauge | `module` | Current consecutive failure count | -| `nexum_state_bytes_used` | Gauge | `module` | Current local-store usage in bytes | -| `nexum_memory_bytes_used` | Gauge | `module` | Current WASM linear memory size | - -#### Per-chain RPC - -| Metric | Type | Labels | Description | -|--------|------|--------|-------------| -| `nexum_rpc_requests_total` | Counter | `chain_id`, `method` | RPC calls made | -| `nexum_rpc_errors_total` | Counter | `chain_id`, `method`, `endpoint` | RPC errors | -| `nexum_rpc_duration_seconds` | Histogram | `chain_id`, `method` | RPC call latency | -| `nexum_rpc_fallbacks_total` | Counter | `chain_id` | Times a fallback endpoint was used | -| `nexum_subscription_reconnects_total` | Counter | `chain_id` | Subscription reconnection count | -| `nexum_blocks_behind` | Gauge | `chain_id` | Blocks behind head (0 = caught up) | - -#### Identity - -| Metric | Type | Labels | Description | -|--------|------|--------|-------------| -| `nexum_identity_sign_total` | Counter | `module`, `account` | Signing operations performed | -| `nexum_identity_errors_total` | Counter | `module`, `error_code` | Identity operation failures | - -### Exposition - -```toml -[metrics] -enabled = true -listen = "0.0.0.0:9090" -path = "/metrics" -``` - -```bash -curl http://localhost:9090/metrics -# HELP nexum_events_dispatched_total Events dispatched to modules -# TYPE nexum_events_dispatched_total counter -nexum_events_dispatched_total{module="twap-monitor",event_type="block"} 150234 -``` - -## Health Checks - -> **Future direction, not in 0.2 scope.** A dedicated `:8080/health` JSON endpoint is described below as design intent for 0.3. The 0.2 engine does **not** bind a separate health port; liveness is signalled by the metrics scrape (`:9100/metrics` returns 200 iff the engine is running and the Prometheus exporter is up) and by the structured `tracing` JSON on stdout (per-module state transitions and quarantine events). Docker / compose configurations use a TCP/bash health probe against the metrics port (see [`docs/deployment/docker.md`](deployment/docker.md)). - -### HTTP Health Endpoint (future direction) - -``` -GET /health -> 200 OK | 503 Service Unavailable -``` - -Intended response shape: - -```json -{ - "status": "healthy", - "uptime_seconds": 86400, - "modules": { - "twap-monitor": { "state": "running", "last_event_age_seconds": 2 }, - "ethflow-watcher": { "state": "running", "last_event_age_seconds": 5 } - }, - "chains": { - "42161": { "connected": true, "head_block": 19000500, "blocks_behind": 0 }, - "1": { "connected": true, "head_block": 21500000, "blocks_behind": 0 } - } -} -``` - -When this endpoint lands, health would be `unhealthy` if: -- Any required chain's RPC is disconnected. -- Any module is in `Dead` state. -- Last event age exceeds a configurable staleness threshold (suggests subscription dropped and backfill failed). - -```toml -[health] -listen = "0.0.0.0:8080" -stale_event_threshold_seconds = 60 -``` - -### Docker / Kubernetes (0.2 today) - -Today the metrics endpoint and the supervisor `tracing` stream cover the liveness signal. A TCP probe against the metrics port works as a liveness check (see `docker-compose.yml` and `docs/deployment/docker.md` for the shipped configuration). Once the dedicated `:8080/health` endpoint lands, the recommended probe shape would be: - -```yaml -livenessProbe: - httpGet: - path: /health - port: 8080 - periodSeconds: 10 -readinessProbe: - httpGet: - path: /health - port: 8080 - initialDelaySeconds: 5 -``` - -## Alerting Conditions - -The runtime itself doesn't send alerts -- it exposes metrics and health for external systems (Prometheus + Alertmanager, Grafana, PagerDuty, etc). Recommended alert rules: - -| Condition | Severity | Prometheus Expression | -|-----------|----------|-----------------------| -| Module entered Dead state | Critical | `nexum_modules_dead > 0` | -| High event failure rate | Warning | `rate(nexum_events_failed_total[5m]) / rate(nexum_events_dispatched_total[5m]) > 0.1` | -| Event processing latency spike | Warning | `histogram_quantile(0.99, nexum_event_duration_seconds) > 5` | -| RPC endpoint down | Critical | `nexum_rpc_errors_total` sustained increase with no successes | -| Chain falling behind | Warning | `nexum_blocks_behind > 10` | -| State store near quota | Warning | `nexum_state_bytes_used / nexum_state_bytes_limit > 0.9` | -| Subscription reconnect storm | Warning | `rate(nexum_subscription_reconnects_total[5m]) > 1` | -| Identity signing failures | Warning | `rate(nexum_identity_errors_total[5m]) > 0.5` | - -## Runtime Configuration Summary - -```toml -# -- Logging -- -[logging] -format = "json" # "json" | "pretty" -level = "info" # default filter level -module_level = "debug" # filter for module guest logs - -# -- Metrics -- -[metrics] -enabled = true -listen = "0.0.0.0:9090" -path = "/metrics" - -# -- Health (future direction; not bound in 0.2) -- -# [health] -# listen = "0.0.0.0:8080" -# stale_event_threshold_seconds = 60 - -# -- Epoch ticker -- -[runtime] -epoch_interval_ms = 100 -epoch_deadline = 10 # epochs before yield (~1s) - -# -- Resource defaults -- -# In 0.2 these come from compile-time constants in -# crates/nexum-runtime/src/runtime/limits.rs (DEFAULT_FUEL_PER_EVENT = 1B, -# DEFAULT_MEMORY_LIMIT = 64 MiB). Manifest-driven per-module overrides are -# a future direction (0.3). - -# -- Global restart defaults -- -# In 0.2 the restart policy is the global exponential backoff (1s -> 2s -> -# ... cap 5 min) defined in runtime/restart_policy.rs and the poison -# threshold (POISON_MAX_FAILURES = 5 within 600 s) in runtime/poison_policy.rs. -# Per-module overrides via [module.restart] are a future direction. -``` - -## Deployment: Docker - -The shipped Dockerfile lives at the repo root; see [`docs/deployment/docker.md`](deployment/docker.md) for the canonical operator-facing configuration. The shape is a multi-stage build with `tini` PID1, a non-root `shepherd` user, the five reference modules baked in, and the metrics port exposed: - -```dockerfile -FROM rust:1.96-slim-bookworm AS builder -# ... cargo build --release -p nexum-cli + module wasm builds ... - -FROM debian:bookworm-slim -COPY --from=builder /build/target/release/nexum /usr/local/bin/ -EXPOSE 9100 # metrics -ENTRYPOINT ["tini", "--", "nexum"] -``` - -```bash -docker run -d \ - -v /etc/shepherd:/etc/shepherd \ - -v /var/shepherd:/var/shepherd \ - -p 9100:9100 \ - shepherd:latest -``` - -Volumes: -- `/etc/shepherd/` -- engine config, module manifests. -- `/var/shepherd/` -- local-store (`state.redb`), content cache, logs. - -## Operational Runbook (CLI) - -```bash -# List loaded modules and their state -nexum module list - -# Restart a dead or failed module -nexum module restart twap-monitor - -# Reload a module (re-fetch wasm, recompile) -nexum module reload twap-monitor - -# Purge a module's local-store -nexum state purge --module twap-monitor - -# Compact the state database -nexum state compact - -# Check runtime health -nexum health - -# Dump metrics -nexum metrics -``` +The runtime records through the `metrics` crate facade. The `shepherd` binary installs a `metrics-exporter-prometheus` exporter (the Prometheus add-on) that binds `/metrics` on `[engine.metrics] bind_addr` when `enabled = true`; the bare `nexum` binary installs the recorder but binds no listener. The runtime emits no alerts itself. The metric surface and recommended alert rules are in [`docs/production.md`](./production.md) §6-7. diff --git a/docs/deployment.md b/docs/deployment.md index 88213615..c430ceaa 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -1,255 +1,124 @@ # Deploying Shepherd -This guide covers the **operator** side - running a `nexum` -instance against a fleet of WASM modules. For module-author topics -(building a module from scratch, writing tests, packaging) see the -[SDK overview](./sdk.md) and the [first-module -tutorial](./tutorial-first-module.md). +The operator side: the `engine.toml` reference, building module artefacts, and local runs. For the production deploy (systemd, backup, observability) see [`docs/production.md`](./production.md); for containers see [`docs/deployment/docker.md`](deployment/docker.md); for multiple chains see [`docs/deployment/multi-chain.md`](deployment/multi-chain.md). Module-author topics are in the [SDK overview](./sdk.md) and the [first-module tutorial](./tutorial-first-module.md). ## What an operator runs -A Shepherd deployment is one or more `nexum` processes, each -pointed at: +A deployment is one or more engine processes, each pointed at: -1. an `engine.toml` describing the local environment (chain RPCs, - resource caps, where state lives); -2. one or more `[[modules]]` entries listing `.wasm` artefacts and - their `module.toml` manifests; -3. a `state_dir` the engine creates / owns (the redb local-store - database). +1. an `engine.toml` describing the local environment (chain RPCs, resource caps, state directory); +2. `[[modules]]` entries listing `.wasm` artefacts and their `module.toml` manifests; +3. `[[adapters]]` entries listing venue-adapter components (the bundled `cow-venue` adapter); +4. a `state_dir` the engine creates and owns (the redb local-store). -Modules are statically declared in `engine.toml`. The engine does -not pull them from a registry today; you ship the `.wasm` files -alongside the binary and reference them by path. +CoW order submission needs the venue platform, so run the `shepherd` binary, not the bare `nexum`. Modules and adapters are declared statically; the engine does not pull them from a registry today. ## `engine.toml` reference +`engine.example.toml` at the repo root is the annotated template. The shape: + ```toml [engine] -# Directory the local-store redb file (and future engine artefacts) -# will be created under. Created automatically at boot. +# Local-store redb directory, created at boot. state_dir = "./data" - -# `tracing_subscriber::EnvFilter`-compatible directive. `RUST_LOG` -# overrides at process start. +# `tracing_subscriber::EnvFilter` directive; `RUST_LOG` overrides at start. log_level = "info" -# Resource caps applied to every module store at instantiation. -# wasmtime traps a module that overruns either; the supervisor then -# logs and continues on the next event. -[engine.limits] -# Fuel budget granted before every `on_event` invocation. -# 1 unit ~ 1 wasm instruction. 1 billion ~ ~1 second of pure compute. -fuel_per_event = 1_000_000_000 -# Per-dispatch wall-clock backstop, in seconds; bounds host-call time fuel does not meter. Default 120. -event_deadline_secs = 120 -# Linear-memory ceiling per module, in bytes. Default 64 MiB. -memory_bytes = 67_108_864 - -# One [chains.] table per chain the engine should be able to -# reach. Chain ids are EVM decimal. -# -# ws:// + wss:// — alloy pubsub transport (REQUIRED for the -# eth_subscribe-backed [[subscription]] kinds: -# `block`, `log`). -# http:// + https:// — HTTP transport; request/response only, -# no subscriptions. -# -# Mix and match: a chain used only for eth_call (e.g. a Chainlink -# oracle module) can be HTTP; chains carrying log subscriptions -# need WebSocket. - +[engine.metrics] +# Prometheus exporter. Disabled unless enabled = true (bare `nexum` never binds it). +enabled = true +bind_addr = "127.0.0.1:9100" + +# Per-module wasmtime resource caps. Every field is optional; omitted +# values resolve to built-in defaults. Applies uniformly to every module. +[limits] +fuel_per_event = 1_000_000_000 # ~1s of pure compute; wasmtime traps on exhaustion +event_deadline_secs = 120 # wall-clock backstop for unmetered host-call time (min 1) +memory_bytes = 67_108_864 # 64 MiB linear-memory ceiling +state_bytes = 52_428_800 # 50 MiB local-store quota + +# One [chains.] per chain, keyed by EVM decimal id. `ws://`/`wss://` +# engage the pubsub transport (blocks push via eth_subscribe); `http://`/ +# `https://` poll (blocks via eth_getBlockByNumber, logs via eth_getLogs). +# Both work; prefer wss:// where the provider offers it. [chains.1] rpc_url = "https://ethereum-rpc.publicnode.com" - -[chains.100] -rpc_url = "https://rpc.gnosischain.com" +# request_timeout_secs = 30 # per-request JSON-RPC timeout (default 30, 0 rejected at boot) [chains.11155111] rpc_url = "wss://ethereum-sepolia-rpc.publicnode.com" - -[chains.42161] -rpc_url = "https://arb1.arbitrum.io/rpc" - -# Extension-owned tables. The engine hands each [extensions.] -# table to the matching extension verbatim; the engine itself never -# interprets them. The cow-api extension reads per-chain orderbook -# base URL overrides here - chains without an entry use the canonical -# api.cow.fi URL. Point this at a staging/barn instance or a local -# mock (tools/orderbook-mock for the load test). -[extensions.cow.orderbook_urls] -# 11155111 = "http://localhost:9999" ``` -### `[[modules]]` entries +The full `[limits.*]` subtables (`http`, `chain`, `logs`, `poison`, `dispatch`, `quota`, `watch`) are documented inline in `engine.example.toml`. -> 0.2 takes the module path + manifest as positional CLI args (a -> single module per engine process). The multi-module -> `[[modules]]` array is shipped by the supervisor work in nullislabs/shepherd PR #9. - -Once the supervisor PR lands, the syntax is: +### `[[modules]]` and `[[adapters]]` ```toml [[modules]] -name = "twap-monitor" -wasm = "modules/twap-monitor.wasm" -manifest = "modules/twap-monitor/module.toml" +path = "modules/twap-monitor.wasm" +manifest = "modules/twap-monitor/module.toml" # defaults to module.toml beside `path` -[[modules]] -name = "ethflow-watcher" -wasm = "modules/ethflow-watcher.wasm" -manifest = "modules/ethflow-watcher/module.toml" +[[adapters]] +path = "target/wasm32-wasip2/release/cow_venue.wasm" +manifest = "crates/cow-venue/module.toml" +http_allow = ["api.cow.fi"] # outbound wasi:http allowlist the operator grants ``` +The orderbook base URL is not an engine setting: the `cow-venue` adapter reads it from its own `module.toml` `[config]` (`chain`, and optional `orderbook-url` to point at a barn or mock). See [`docs/deployment/multi-chain.md`](deployment/multi-chain.md). + ## Building module `.wasm` artefacts -Modules compile to the `wasm32-wasip2` target. Add the target once -per dev machine: +Modules compile to `wasm32-wasip2`. Add the target once: ```sh rustup target add wasm32-wasip2 ``` -Then build release artefacts from the workspace root: +Build release artefacts from the workspace root: ```sh cargo build --target wasm32-wasip2 --release \ -p twap-monitor -p ethflow-watcher +cargo build --target wasm32-wasip2 --release -p cow-venue --features adapter ``` -The `.wasm` files land in -`target/wasm32-wasip2/release/{twap_monitor,ethflow_watcher}.wasm`. -Copy them to wherever your `engine.toml` points (typical: -`./modules/` next to the binary). - -Size sanity check after a build (CI guards regression): +Artefacts land in `target/wasm32-wasip2/release/*.wasm`. Copy them to wherever `engine.toml` points. CI guards a size regression: ```sh ls -lh target/wasm32-wasip2/release/*.wasm ``` -The M2 modules sit at 270–310 KB optimised. Sudden +10× growth -usually means a fresh dependency landed in the wasm graph — review -`cargo tree -p --target wasm32-wasip2` to confirm. - -## Single-binary local runs +## Local runs -The 0.2 engine ships as the `nexum` binary. From the -workspace root, dispatch a module against a test event: +Build and run the `shepherd` binary against an `engine.toml`: ```sh -cargo run -p nexum-cli -- \ - target/wasm32-wasip2/release/twap_monitor.wasm \ - modules/twap-monitor/module.toml +cargo run -p shepherd -- --engine-config engine.toml ``` -On a fresh checkout, the engine creates `./data/local-store.redb`, -opens RPC providers for the chains in `engine.toml`, loads the -component, calls `init`, and dispatches a synthetic block event. -Console output is `tracing` JSON (or pretty if you set -`RUST_LOG=info,nexum_runtime=debug`). - -For systemd-style production runs, see `docs/production.md`. - -## Docker - -A `Dockerfile` + `docker-compose.yml` ship at the repo root. See -[`docs/deployment/docker.md`](deployment/docker.md) for the full -container workflow. The quick start: +The single-module shortcut takes positional paths and synthesizes a one-module config: ```sh -cp .env.example .env -$EDITOR .env # paste wss:// RPC URLs -docker compose up -d -``` - -The image is published to -`ghcr.io/nullislabs/shepherd:` on every merged commit to `main`. - -Mount the `state_dir` as a volume so the redb file survives container -restarts. - -## Observability - -### Logs - -Every host backend logs through `tracing`. Set `RUST_LOG` to filter: - -```sh -RUST_LOG=info,nexum_runtime=debug,nexum_runtime::host::cow_orderbook=trace \ - cargo run -p nexum-cli -- ... -``` - -Recommended baseline for production: - -``` -RUST_LOG=info,nexum_runtime::host=debug -``` - -The structured-logging audit consolidates the field set -across every dispatch / state change / submission path so a single -JSON grep reconstructs each order's timeline. - -### Prometheus metrics - -A planned metrics exporter wires a `metrics-exporter-prometheus` endpoint at -`engine.toml::[engine.metrics].bind_addr` (default -`127.0.0.1:9100`). Once it lands, scrape with: - -```yaml -scrape_configs: - - job_name: shepherd - static_configs: - - targets: ['shepherd-host:9100'] +cargo run -p shepherd -- \ + target/wasm32-wasip2/release/twap_monitor.wasm \ + modules/twap-monitor/module.toml ``` -Suggested Grafana panels (dashboard JSON planned): - -- Module uptime — `shepherd_module_uptime_seconds{module}` -- Event latency p50 / p95 / p99 — - `shepherd_event_latency_seconds{module}` -- Submit success rate — - `rate(shepherd_cow_api_submit_total{outcome="success"}[5m])` - / - `rate(shepherd_cow_api_submit_total[5m])` -- Fuel headroom — - `1 - (shepherd_fuel_consumed / 1_000_000_000)` -- Memory pressure — - `shepherd_memory_peak_bytes / 67_108_864` - -## Backups - -`state_dir/local-store.redb` is the only durable state the engine -holds. redb's WAL means a file-level snapshot taken while the -engine is running is consistent; for safety, either: - -- Pause the engine (`systemctl stop shepherd`), copy the file, then - restart. Sub-second downtime on a small store. -- Use `redb::Database::backup` from a sidecar. - -The store is per-module-namespaced (32-byte keccak prefix per -`module.name`), so a fresh deployment can re-import partial backups -without cross-module bleed. +At boot the engine creates `state_dir/local-store.redb`, opens RPC providers, loads components, calls `init`, and begins dispatching. Console output is `tracing` JSON, or pretty with `--pretty-logs`. For systemd runs see [`docs/production.md`](./production.md). ## Troubleshooting | Symptom | Likely cause | Fix | |---|---|---| -| `init failed: unsupported` | Module imports a capability that needs a chain RPC not configured. | Add the missing `[chains.]` entry to `engine.toml`. | -| `unknown chain ... (no engine.toml RPC entry)` | Module dispatched `chain::request` for a chain not in `engine.toml`. | Same — add the chain. | -| `OutOfFuel` trap, immediate restart loop | Module's `on_event` exceeds `[engine.limits].fuel_per_event`. | Bump `fuel_per_event`, or audit the module's loop bounds. | -| `MemoryOutOfBounds` trap | Module's linear-memory growth exceeds `[engine.limits].memory_bytes`. | Bump `memory_bytes`; profile the module for runaway allocations. | -| `dispatch exceeded its ...s wall-clock deadline`, module marked dead | A host call (RPC / HTTP) blocked, or a chain of slow calls, ran past `[engine.limits].event_deadline_secs`. | Audit the module's host calls; tighten the per-call chain/HTTP timeouts, or bump `event_deadline_secs` if the workload is legitimately long. | -| `submit failed (... InvalidAppData)` | Module sent an `OrderCreation` with a non-empty app-data hash but `app_data = "{}"`. | Out of M2 scope — modules currently only support `EMPTY_APP_DATA_JSON`. Patch is on the M3 follow-up board. | +| `init failed: unsupported` | Module needs a chain RPC not configured. | Add the missing `[chains.]` entry. | +| `unknown chain ... (no engine.toml RPC entry)` | `chain::request` for a chain not in `engine.toml`. | Add the chain. | +| `OutOfFuel` trap, restart loop | `on_event` exceeds `[limits] fuel_per_event`. | Bump `fuel_per_event`, or audit the module's loop bounds. | +| `MemoryOutOfBounds` trap | Linear-memory growth exceeds `[limits] memory_bytes`. | Bump `memory_bytes`; profile the module. | +| dispatch exceeded its wall-clock deadline, module marked dead | A host call blocked past `[limits] event_deadline_secs`. | Tighten the module's host-call timeouts, or bump `event_deadline_secs`. | ## Reference - [SDK overview](./sdk.md) - [First-module tutorial](./tutorial-first-module.md) -- ADR-0001 (`docs/adr/0001-engine-toml-separate-from-nexum-toml.md`) - — why `engine.toml` and `module.toml` are split. -- ADR-0003 (`docs/adr/0003-local-store-namespacing.md`) — how the - `state_dir/local-store.redb` file partitions across modules. -- ADR-0005 (`docs/adr/0005-cow-api-via-cached-orderbookapi.md`) — - how the `cow-api` host backend caches per-chain `OrderBookApi` - clients. +- ADR-0001 (`docs/adr/0001-engine-toml-separate-from-nexum-toml.md`): why `engine.toml` and `module.toml` are split. +- ADR-0003 (`docs/adr/0003-local-store-namespacing.md`): how `state_dir/local-store.redb` partitions across modules. diff --git a/docs/deployment/docker.md b/docs/deployment/docker.md index e6d1220d..6c48958f 100644 --- a/docs/deployment/docker.md +++ b/docs/deployment/docker.md @@ -1,194 +1,87 @@ -# Docker deployment runbook +# Docker deployment -Operator-facing quickstart for running Shepherd in production via the -published container image. For the full hardening surface (systemd -unit, backup recipes, RPC selection, alerting rules) read -`docs/production.md`. +Running Shepherd from the published container image. The hardening surface (systemd, backup, RPC selection, alerting) is in [`docs/production.md`](../production.md); the `engine.toml` reference is in [`docs/deployment.md`](../deployment.md). -The image is published on every push to `main` and on every -`v*` tag: +The image is published to `ghcr.io/nullislabs/shepherd` on every push to `main` and every `v*` tag: ``` -ghcr.io/bleu/nullis-shepherd:latest # main branch HEAD -ghcr.io/bleu/nullis-shepherd:sha- # exact-build pin -ghcr.io/bleu/nullis-shepherd:v0.2.0 # tag +ghcr.io/nullislabs/shepherd:latest # main HEAD +ghcr.io/nullislabs/shepherd:sha- # exact-build pin +ghcr.io/nullislabs/shepherd:v0.2.0 # release tag ``` -`linux/amd64` only for now (the soak VM is x86_64; add `arm64` once -an operator surfaces a real need). +The image is a multi-stage build: a `debian:bookworm-slim` runtime, `tini` as PID 1 (forwards SIGINT/SIGTERM), a non-root `shepherd` user, the `shepherd` binary, and the four production modules plus the `cow-venue` adapter baked under `/opt/shepherd/`. `linux/amd64` only. ---- - -## 1. First boot on a fresh VM +## 1. First boot ```bash -# On the VM: -git clone https://github.com/bleu/nullis-shepherd /opt/shepherd +git clone https://github.com/nullislabs/shepherd /opt/shepherd cd /opt/shepherd -# Operator-supplied RPC URLs. `.env` is gitignored; the template -# committed at `.env.example` lists every variable the engine -# substitutes into `engine.docker.toml` via `${VAR}` placeholders. +# Operator RPC URLs. `.env` is gitignored; `.env.example` lists every +# variable the engine substitutes into engine.docker.toml via ${VAR}. cp .env.example .env -${EDITOR:-vi} .env # paste your paid wss:// URLs +${EDITOR:-vi} .env -# Pull the published image (no local build needed). docker compose pull - -# Start the engine. Compose reads `.env` automatically and passes -# the listed variables into the container, where the engine -# substitutes them at config-load time. docker compose up -d - -# Logs (JSON line-per-event, see `docs/production.md §5`). docker compose logs -f shepherd ``` -If you want the observability stack on the same host: +The observability profile adds Prometheus on the same host: ```bash -docker compose --profile observability up -d -# Prometheus UI: http://127.0.0.1:9090 +docker compose --profile observability up -d # Prometheus UI: http://127.0.0.1:9090 ``` -The metrics endpoint binds the **host's loopback** by default -(`127.0.0.1:9100`); the Prometheus container scrapes via the -compose-internal DNS name `shepherd:9100`. Never expose `:9100` to -the public internet without authn — see `docs/production.md §7`. - ---- - -## 2. Configuring `engine.toml` - -The image bind-mounts the committed `engine.docker.toml` at -`/etc/shepherd/engine.toml` read-only. It uses `${VAR}` placeholders -for every paid-RPC URL, which the engine substitutes at load time -from environment (Docker compose forwards them in from `.env`). -A missing variable fails the boot fast with the exact name. - -To run with a custom config (different module mix, extra chains) -instead of `engine.docker.toml`, point compose at it via -`SHEPHERD_ENGINE_CONFIG=./engine.local.toml` in `.env` — the bind -mount picks up whichever path is set. - -Minimum production shape if you write your own: - -```toml -[engine] -state_dir = "/var/lib/shepherd" # mapped to the `shepherd-state` named volume -log_level = "info" - -[engine.metrics] -enabled = true -bind_addr = "0.0.0.0:9100" # inside the container; compose maps to 127.0.0.1 - -# One per chain you subscribe to. `${VAR}` placeholders are -# substituted at load time from environment — keep the actual URL -# in `.env`, not in any committed file. Must be `wss://`; the -# engine emits a boot-time ERROR otherwise (see docs/production.md §6). -[chains.11155111] -rpc_url = "${SEPOLIA_RPC_URL}" - -[chains.42161] -rpc_url = "${ARBITRUM_RPC_URL}" - -# One [[modules]] per .wasm baked into /opt/shepherd/modules/. -# `manifest` defaults to /module.toml if omitted. -[[modules]] -path = "/opt/shepherd/modules/twap_monitor.wasm" -manifest = "/opt/shepherd/manifests/twap-monitor.toml" - -[[modules]] -path = "/opt/shepherd/modules/ethflow_watcher.wasm" -manifest = "/opt/shepherd/manifests/ethflow-watcher.toml" -# Add price-alert / balance-tracker the same way. -``` +The metrics endpoint binds the host loopback (`127.0.0.1:9100`); the Prometheus container scrapes via the compose DNS name `shepherd:9100`. Never expose `:9100` publicly without authn (see [`docs/production.md`](../production.md)). -If you want compose to use this file instead of the bundled -`engine.docker.toml`, set `SHEPHERD_ENGINE_CONFIG=./engine.local.toml` -in `.env` and put your file there (the `*.local.toml` pattern is -already gitignored). +## 2. Configuring the engine -Public RPCs throttle `eth_subscribe` + `eth_getLogs` under sustained -load (independently confirmed by the baseline-latency tool -- see -`docs/operations/baselines/`). The soak explicitly requires paid -endpoints. +The image bind-mounts the committed `engine.docker.toml` at `/etc/shepherd/engine.toml` read-only. It uses `${VAR}` placeholders for every RPC URL, substituted at load time from the environment; a missing variable fails the boot with the exact name. To run a custom config, set `SHEPHERD_ENGINE_CONFIG=./engine.local.toml` in `.env` (the `*.local.toml` pattern is gitignored). ---- +Inside the container the metrics exporter binds `0.0.0.0:9100` so the compose port mapping reaches it; the mapping keeps it on the host loopback. Public RPCs throttle `eth_subscribe` and `eth_getLogs` under load, so use paid endpoints. -## 3. Upgrade / rollback +## 3. Upgrade and rollback ```bash -# Roll forward to the latest main-branch build. -docker compose pull -docker compose up -d # picks up the new image; graceful - # shutdown drains in-flight dispatch - # before the new container takes over. +# Roll forward to the latest main build. Graceful shutdown drains the +# in-flight dispatch before the new container takes over. +docker compose pull && docker compose up -d -# Roll back to a specific build. -export SHEPHERD_IMAGE=ghcr.io/bleu/nullis-shepherd:sha-abc1234 +# Pin a specific build. +export SHEPHERD_IMAGE=ghcr.io/nullislabs/shepherd:sha-abc1234 docker compose up -d - -# Cold roll: stop, prune image, pull fresh. -docker compose down -docker image rm ghcr.io/bleu/nullis-shepherd:latest -docker compose pull && docker compose up -d ``` -The `shepherd-state` named volume survives container recreation — -the redb file with all `submitted:` / `dropped:` / `backoff:` markers -persists across upgrades by design (idempotency lives there). +The `shepherd-state` named volume survives container recreation, so the redb file and its idempotency markers persist across upgrades. ---- +## 4. Building locally -## 4. Building the image locally - -The CI publishes on every push, so the local build path is only for -testing un-merged changes: - -```bash -docker compose build # uses repo-root Dockerfile -docker compose up -d # runs the locally-built image -``` - -To pin the locally-built tag and avoid accidentally pulling `:latest`: +For testing unmerged changes: ```bash +docker compose build # repo-root Dockerfile export SHEPHERD_IMAGE=shepherd:local docker build -t "$SHEPHERD_IMAGE" . docker compose up -d ``` ---- - -## 5. Verifying the deploy +## 5. Verifying ```bash -# Engine is up, modules are loaded, no module is quarantined. +# Engine up, no module or adapter quarantined. curl -s http://127.0.0.1:9100/metrics \ - | grep -E '^shepherd_(module_poisoned|module_restarts_total|stream_reconnects_total)' + | grep -E '^shepherd_(module_poisoned|adapter_poisoned|stream_reconnects_total)' -# Tail the structured logs. -docker compose logs -f shepherd | grep -E '"level":(("ERROR")|("WARN"))' - -# In a separate shell: confirm the engine wrote a last-dispatched- -# block marker after the first 30s of uptime (proof the supervisor -# is dispatching events, not just idle-looping). +docker compose logs -f shepherd | grep -E '"level":("ERROR"|"WARN")' docker compose exec shepherd ls -la /var/lib/shepherd/ ``` -Green: `shepherd_module_poisoned == 0`, no ERROR/WARN lines beyond -boot, and a non-empty redb file under `/var/lib/shepherd/`. - ---- +Green: poisoned gauges `0`, no ERROR/WARN beyond boot, and a non-empty `local-store.redb` under `/var/lib/shepherd/`. -## 6. Cross-references +## 6. See also -- `docs/production.md` — full process-level deploy (systemd path), - backup recipes, RPC selection, alerting rules, runbook. -- `docs/06-production-hardening.md` — resource-limit design (fuel, - memory, storage), restart policy, RPC resilience, observability - design. -- `docs/operations/m3-testnet-runbook.md` — staging validation - playbook; reuse the same steps before the production soak. -- `engine.example.toml` — annotated reference for the engine config. +- [`docs/production.md`](../production.md): systemd, backup, RPC selection, alerting. +- [`docs/06-production-hardening.md`](../06-production-hardening.md): resource-limit design, restart policy, RPC resilience. +- `engine.example.toml`: annotated engine config reference. diff --git a/docs/deployment/multi-chain.md b/docs/deployment/multi-chain.md index b15e3a80..36a1e138 100644 --- a/docs/deployment/multi-chain.md +++ b/docs/deployment/multi-chain.md @@ -1,50 +1,24 @@ -# Multi-chain deployment patterns +# Multi-chain deployment -This guide covers running Shepherd against multiple EVM chains simultaneously. -The engine dispatches each module only to the chains it subscribes to, so a -single `nexum` process can serve modules watching Mainnet, Gnosis Chain, -Arbitrum One, and Base at the same time. +Running Shepherd against multiple EVM chains. The engine dispatches each module only to the chains it subscribes to, so one process can serve modules watching Mainnet, Gnosis Chain, Arbitrum One, and Base at once. For the base `engine.toml` reference see [`docs/deployment.md`](../deployment.md). -That covers keeper modules, which subscribe per chain. It does not extend to -venue adapters. The CoW adapter fixes its orderbook chain at `init` from its -own manifest `[config] chain`, and registers under the fixed venue id `cow` -(`CowVenue::ID`), which carries no chain component. Two cow adapters installed -in one process would therefore register under the same id, with nothing at the -venue registry to tell them apart. - -Single-process multi-chain *submission* is an explicit non-goal for M4. Run one -engine process per submitting chain, each paired with the matching adapter -manifest (`module.toml` for Mainnet, `module.sepolia.toml` for Sepolia). -Modules that only watch chains are unaffected and may span chains freely. - ---- +Keeper modules subscribe per chain and may span chains freely. Venue adapters do not: the `cow-venue` adapter fixes its orderbook chain at `init` from its own `[config] chain`, and registers under the fixed venue id `cow` (`CowVenue::ID`), which carries no chain component. Two cow adapters in one process would register under the same id. Single-process multi-chain submission is a non-goal: run one engine process per submitting chain, each paired with the matching adapter manifest (`module.toml` for Mainnet, `module.sepolia.toml` for Sepolia). ## Chain support matrix -The table lists the chains this repo's modules target. The CoW Protocol -orderbook supports more (the `cowprotocol` crate's `Chain` enum also carries -BNB, Polygon, Avalanche, Linea, and Plasma); the same wiring pattern applies -to any of them. Shepherd can subscribe to block and log events on any EVM -chain; add only the chains your modules actually use. - -| Chain | Chain ID | Orderbook slug | Barn (staging) | Notes | -|-------|----------|----------------|----------------|-------| -| Ethereum Mainnet | 1 | `mainnet` | ✓ | Primary production chain | -| Gnosis Chain | 100 | `xdai` | ✓ | Second-most-active CoW deployment | -| Base | 8453 | `base` | ✗ | | -| Arbitrum One | 42161 | `arbitrum_one` | ✓ | | -| Sepolia | 11155111 | `sepolia` | ✓ | Recommended testnet for soak runs | - -The CoW Protocol orderbook is not deployed on every EVM chain. A `[chains.]` -entry in `engine.toml` for a chain with no orderbook deployment will open block -and log subscriptions, but any module that calls `cow-api` will fail at runtime. +The chains this repo's modules target. The CoW orderbook supports more (the `cowprotocol` crate's `Chain` enum also carries BNB, Polygon, Avalanche, Linea, Plasma); the same wiring applies. A `[chains.]` entry for a chain with no orderbook deployment still opens block and log subscriptions, but any submission on it fails at runtime. ---- +| Chain | Chain ID | Orderbook slug | Barn (staging) | +|-------|----------|----------------|----------------| +| Ethereum Mainnet | 1 | `mainnet` | yes | +| Gnosis Chain | 100 | `xdai` | yes | +| Base | 8453 | `base` | no | +| Arbitrum One | 42161 | `arbitrum_one` | yes | +| Sepolia | 11155111 | `sepolia` | yes | -## `engine.toml` — per-chain RPC wiring +## Per-chain RPC wiring -Add one `[chains.]` table per chain. Log-subscription modules require a -WebSocket (`wss://`) transport; request-only modules can use HTTP. +One `[chains.]` table per chain. `ws://`/`wss://` engage the pubsub transport (blocks push via `eth_subscribe(newHeads)`); `http://`/`https://` poll (blocks via `eth_getBlockByNumber`, logs via `eth_getLogs`). Both transports carry block and log subscriptions; there is no WebSocket requirement. Prefer `wss://` where the provider offers it, since push is lower-latency than polling. ```toml [chains.1] # Ethereum Mainnet @@ -59,16 +33,13 @@ rpc_url = "${BASE_RPC_URL}" [chains.42161] # Arbitrum One rpc_url = "${ARBITRUM_RPC_URL}" -[chains.11155111] # Sepolia (soak / staging) +[chains.11155111] # Sepolia rpc_url = "${SEPOLIA_RPC_URL}" ``` -`${VAR}` tokens are substituted at engine boot from environment variables. A -missing variable fails fast with the exact variable name. In Docker Compose, -forward the variables from the host `.env` file: +`${VAR}` tokens are substituted at boot from the environment; a missing variable fails fast with the exact name. Under Docker Compose, forward the variables from the host `.env`: ```yaml -# docker-compose.yml (engine service environment section) environment: MAINNET_RPC_URL: GNOSIS_RPC_URL: @@ -77,56 +48,28 @@ environment: SEPOLIA_RPC_URL: ``` -### Opt out of the WebSocket requirement +## Orderbook URLs -By default the engine logs an ERROR at boot when a chain is configured with an -HTTP URL because `block` and `log` subscriptions need WebSocket. If a chain is -used only for `chain::request` (poll-only modules with no `[[subscription]]`), -suppress it: +The `cow-venue` adapter resolves the orderbook base URL from its `[config] chain` using the canonical `https://api.cow.fi//` pattern. Override it per adapter in the adapter's `module.toml` to point at a barn (staging) instance or a local mock: ```toml -[chains.1] -rpc_url = "https://eth.llamarpc.com" -require_ws = false +# crates/cow-venue/module.toml +[config] +chain = 11155111 +orderbook-url = "https://barn.api.cow.fi/sepolia/" ``` ---- - -## CoW Protocol orderbook URLs - -The engine's `cow-api` extension resolves the orderbook URL automatically from -the chain ID using the canonical `https://api.cow.fi//` pattern. No -extra config is needed for production. - -Override individual chains to point at a staging ("barn") instance or a local -mock: - -```toml -[extensions.cow.orderbook_urls] -# Point chain 11155111 at the barn (staging) orderbook: -11155111 = "https://barn.api.cow.fi/sepolia/" - -# Point chain 1 at a local Wiremock during integration testing: -1 = "http://localhost:9999/" -``` - -**Canonical URLs by chain:** - | Chain | Production URL | Barn (staging) URL | |-------|---------------|-------------------| | Mainnet (1) | `https://api.cow.fi/mainnet/` | `https://barn.api.cow.fi/mainnet/` | | Gnosis (100) | `https://api.cow.fi/xdai/` | `https://barn.api.cow.fi/xdai/` | -| Base (8453) | `https://api.cow.fi/base/` | — | +| Base (8453) | `https://api.cow.fi/base/` | none | | Arbitrum One (42161) | `https://api.cow.fi/arbitrum_one/` | `https://barn.api.cow.fi/arbitrum_one/` | | Sepolia (11155111) | `https://api.cow.fi/sepolia/` | `https://barn.api.cow.fi/sepolia/` | ---- - ## Contract addresses -### CREATE2-stable (identical on every chain) - -These addresses are the same across all supported chains: +CREATE2-stable, identical on every supported chain: | Contract | Address | |----------|---------| @@ -134,176 +77,57 @@ These addresses are the same across all supported chains: | `GPv2VaultRelayer` | `0xC92E8bdf79f0507f65a392b0ab4667716BFE0110` | | `ComposableCoW` | `0xfdaFc9d1902f4e0b84f65F49f244b32b31013b74` | -### EthFlow - -The **current** production EthFlow deployment is address-identical on every -chain CoW Protocol supports (the `cowprotocol` crate pins it as -`ETH_FLOW_PRODUCTION`, sourced from -[`cowprotocol/ethflowcontract`](https://github.com/cowprotocol/ethflowcontract) -`networks.prod.json`): +EthFlow is not CREATE2-stable. The current production deployment is address-identical on every chain CoW supports (the `cowprotocol` crate pins it as `ETH_FLOW_PRODUCTION` from `cowprotocol/ethflowcontract` `networks.prod.json`): ``` -0xbA3cB449bD2B4ADddBc894D8697F5170800EAdeC (all supported chains) +0xbA3cB449bD2B4ADddBc894D8697F5170800EAdeC ``` -This is what `modules/ethflow-watcher/module.toml` wires for Sepolia, and the -same value carries to Mainnet, Gnosis Chain, Arbitrum One, and Base today. - -Unlike ComposableCoW, however, the sameness is **not a CREATE2 guarantee** - -EthFlow has legacy per-version deployments at other addresses (e.g. the -v1.0.0 contracts), and a future version may land at a new address on a -per-chain schedule. When porting `ethflow-watcher` to a new chain or bumping -the contract version, verify the `[[subscription]] address` against -`networks.prod.json` for that chain rather than assuming the constant holds. - ---- - -## Module manifests — the `[[subscription]]` duplication pattern +Legacy per-version EthFlow deployments live at other addresses, and a future version may land per-chain. When porting `ethflow-watcher` to a new chain, verify the `[[subscription]] address` against `networks.prod.json` for that chain. -A module subscribes per chain. To watch the same event on multiple chains, -declare one `[[subscription]]` block per chain. The engine opens a separate -stream for each and routes dispatches independently. +## Subscription duplication -### twap-monitor on Mainnet + Gnosis +A module subscribes per chain: to watch the same event on multiple chains, declare one `[[subscription]]` block per chain. The engine opens a separate stream for each and routes dispatches independently, tagging each event with its `chain_id`. ```toml -# module.toml for twap-monitor (multi-chain) - -# ComposableCoW.ConditionalOrderCreated on Mainnet +# twap-monitor on Mainnet + Gnosis: ComposableCoW.ConditionalOrderCreated [[subscription]] kind = "chain-log" chain_id = 1 address = "0xfdaFc9d1902f4e0b84f65F49f244b32b31013b74" event_signature = "0x2cceac5555b0ca45a3744ced542f54b56ad2eb45e521962372eef212a2cbf361" -# New-block ticks on Mainnet (drives the poll loop) [[subscription]] kind = "block" chain_id = 1 -# ComposableCoW.ConditionalOrderCreated on Gnosis Chain [[subscription]] kind = "chain-log" chain_id = 100 address = "0xfdaFc9d1902f4e0b84f65F49f244b32b31013b74" event_signature = "0x2cceac5555b0ca45a3744ced542f54b56ad2eb45e521962372eef212a2cbf361" -# New-block ticks on Gnosis Chain [[subscription]] kind = "block" chain_id = 100 ``` -The module's `on_event` receives every event tagged with its `chain_id`; use -the chain ID to dispatch to the correct `chain::request` target and the correct -`cow-api` submission path. - -### ethflow-watcher on Mainnet + Sepolia - -```toml -# module.toml for ethflow-watcher (multi-chain) +## Event topics -# CoWSwapEthFlow.OrderPlacement on Mainnet -# IMPORTANT: verify this address against cowprotocol/ethflowcontract -# networks.prod.json before deploying — EthFlow is NOT CREATE2-stable. -[[subscription]] -kind = "chain-log" -chain_id = 1 -address = "" -event_signature = "0xcf5f9de2984132265203b5c335b25727702ca77262ff622e136baa7362bf1da9" - -# CoWSwapEthFlow.OrderPlacement on Sepolia -[[subscription]] -kind = "chain-log" -chain_id = 11155111 -address = "0xbA3cB449bD2B4ADddBc894D8697F5170800EAdeC" -event_signature = "0xcf5f9de2984132265203b5c335b25727702ca77262ff622e136baa7362bf1da9" -``` - ---- - -## Event topic reference - -These are keccak256 hashes of the event signatures. They are the same on every -chain; only the contract `address` changes for EthFlow. Package of record: -`wit/shepherd-cow/cow-events.wit`. +keccak256 of the event signatures, identical on every chain; only the EthFlow `address` changes. Package of record: `wit/shepherd-cow/cow-events.wit`. | Event | Topic-0 | |-------|---------| | `ComposableCoW.ConditionalOrderCreated(address,(address,bytes32,bytes))` | `0x2cceac5555b0ca45a3744ced542f54b56ad2eb45e521962372eef212a2cbf361` | -| `CoWSwapEthFlow.OrderPlacement(address,(address,address,address,uint256,uint256,uint32,bytes32,uint256,bytes32,bool,bytes32,bytes32),(uint8,bytes),bytes)` | `0xcf5f9de2984132265203b5c335b25727702ca77262ff622e136baa7362bf1da9` | - ---- - -## Resource sizing (per additional chain) - -Each new chain adds: - -- **1 block subscription** (always-on WS stream, ~zero CPU when idle). -- **N log subscriptions**, where N = number of modules with a `chain-log` - subscription on that chain. -- **M `eth_call`s per block** for polling modules (e.g. TWAP), where M scales - linearly with the number of active registered orders on that chain. - -RPC provider sizing: budget **≥ 25 sustained req/s per chain** (a paid -Alchemy / Infura / QuickNode tier - free tiers throttle `eth_subscribe` -under exactly this load). Dedicated endpoints per chain are preferable to -shared-rate plans when running `block` subscriptions simultaneously. - -Monitor `shepherd_chain_request_total{outcome="err"}` per `chain_id` — a -sustained rate above 5% on any chain indicates RPC degradation. - ---- - -## Full multi-chain `engine.docker.toml` example - -```toml -[engine] -state_dir = "/var/lib/shepherd" -log_level = "info" +| `CoWSwapEthFlow.OrderPlacement(...)` | `0xcf5f9de2984132265203b5c335b25727702ca77262ff622e136baa7362bf1da9` | -[engine.metrics] -enabled = true -bind_addr = "0.0.0.0:9100" - -[chains.1] -rpc_url = "${MAINNET_RPC_URL}" - -[chains.100] -rpc_url = "${GNOSIS_RPC_URL}" - -[chains.8453] -rpc_url = "${BASE_RPC_URL}" - -[chains.42161] -rpc_url = "${ARBITRUM_RPC_URL}" - -[chains.11155111] -rpc_url = "${SEPOLIA_RPC_URL}" - -[extensions.cow.orderbook_urls] -# Uncomment to override individual chains with barn or a local mock: -# 11155111 = "https://barn.api.cow.fi/sepolia/" - -[[modules]] -path = "/opt/shepherd/modules/twap_monitor.wasm" -manifest = "/opt/shepherd/manifests/twap-monitor.toml" - -[[modules]] -path = "/opt/shepherd/modules/ethflow_watcher.wasm" -manifest = "/opt/shepherd/manifests/ethflow-watcher.toml" -``` +## Resource sizing ---- +Each new chain adds one always-on block subscription, N log subscriptions (one per module subscribing on that chain), and M `eth_call` per block for polling modules (M scales with active orders). Budget a paid RPC tier per chain; see [`docs/production.md`](../production.md) for capacity and the `shepherd_chain_request_total{outcome="err"}` degradation signal. ## See also -- [`docs/deployment.md`](../deployment.md) — `engine.toml` reference and - single-module quickstart -- [`docs/production.md`](../production.md) — systemd, Docker, RPC provider - recommendations, and alerting rules -- [`docs/deployment/docker.md`](./docker.md) — container image layout -- [`modules/twap-monitor/module.toml`](../../modules/twap-monitor/module.toml) - — canonical subscription example -- [`modules/ethflow-watcher/module.toml`](../../modules/ethflow-watcher/module.toml) - — EthFlow subscription with per-chain address caveat +- [`docs/deployment.md`](../deployment.md): `engine.toml` reference and single-module quickstart. +- [`docs/production.md`](../production.md): systemd, RPC selection, alerting. +- [`docs/deployment/docker.md`](./docker.md): container image layout. +- `modules/twap-monitor/module.toml`, `modules/ethflow-watcher/module.toml`: canonical subscription examples. diff --git a/docs/production.md b/docs/production.md index 96f57755..ab2d73d7 100644 --- a/docs/production.md +++ b/docs/production.md @@ -1,62 +1,24 @@ -# Production deployment guide - -Operator handbook for running `nexum` (Shepherd) in -production. Focused on **concrete artefacts** — unit files, -backup recipes, alert rules — not the design rationale, which -lives in `docs/06-production-hardening.md` (resource enforcement, -restart policy, RPC resilience, logging + metrics design). - -Audience: someone deploying Shepherd onto a Linux host or a -container orchestrator for the first time, with the assumption -that the runtime, modules, and module manifests are already -known-good (M3 + M4 milestones complete; module developer's -handbook is `docs/tutorial-first-module.md`). - ---- - -## 1. Pre-flight checklist - -Before launching: - -- [ ] **Engine binary built in `--release`** mode. - `cargo build -p nexum-cli --release` → `target/release/nexum`. -- [ ] **All module artefacts present** under - `target/wasm32-wasip2/release/` and content-addressable - (the operator pins the sha256 in each module's manifest - `[module] component = "sha256:..."` once 0.3 verification - lands; for 0.2 the field exists but is not enforced). -- [ ] **`engine.toml`** (the production-shape config) exists with: - - `[engine] state_dir = "/var/lib/shepherd"` (or equivalent - persistent path; never `/tmp`). - - `[engine] log_level = "info"` (NOT debug — see §5). - - `[engine.metrics] enabled = true` and `bind_addr` on - `127.0.0.1:9100` (NOT `0.0.0.0` — see §7). - - One `[chains.]` entry per chain you intend to - subscribe to, with a **paid** WS URL (Alchemy / Infura / - QuickNode — public nodes will throttle under sustained - load, see §6). - - One `[[modules]]` entry per module to load. -- [ ] **`/var/lib/shepherd`** exists, writable by the engine's - service user, and on a volume large enough for the local-store - growth budget (§4). -- [ ] **A Prometheus instance** scraping the engine's `/metrics` - endpoint (§7) and an alert pipeline pointed at the rules in §9. -- [ ] **A log aggregator** ingesting the engine's JSON stdout - (§5) — stdout, not a file written by the engine. -- [ ] **An on-call runbook reference** — link to this document - and to `docs/operations/m3-testnet-runbook.md` (testnet - validation, useful for staging deploys). - ---- - -## 2. Process-level deploy: systemd unit +# Production deployment + +Operator handbook for running the `shepherd` binary in production: systemd unit, state backup, observability wiring. The hardening design (resource enforcement, restart policy, RPC resilience, error model) is in [`docs/06-production-hardening.md`](./06-production-hardening.md); the `engine.toml` reference is in [`docs/deployment.md`](./deployment.md); containers in [`docs/deployment/docker.md`](./deployment/docker.md). + +## 1. Pre-flight + +- Engine built in release: `cargo build -p shepherd --release` gives `target/release/shepherd`. +- Module and adapter `.wasm` artefacts present under `target/wasm32-wasip2/release/`. +- `engine.toml` with `state_dir` on a persistent path (never `/tmp`), `log_level = "info"`, `[engine.metrics] enabled = true` and `bind_addr = "127.0.0.1:9100"`, one `[chains.]` per subscribed chain with a paid RPC URL, one `[[modules]]` per module, and the `[[adapters]]` cow entry. +- The `state_dir` exists and is writable by the service user. +- A Prometheus instance scraping `/metrics` (§6) with the alert rules in §7. +- A log aggregator ingesting the engine's JSON stdout (§5). + +## 2. systemd unit `/etc/systemd/system/shepherd.service`: ```ini [Unit] -Description=Shepherd (nexum) - CoW Protocol off-chain automation runtime -Documentation=https://github.com/bleu/nullis-shepherd +Description=Shepherd CoW Protocol automation runtime +Documentation=https://github.com/nullislabs/shepherd After=network-online.target Wants=network-online.target @@ -64,52 +26,39 @@ Wants=network-online.target Type=simple User=shepherd Group=shepherd - -# Working directory + binary. WorkingDirectory=/opt/shepherd -ExecStart=/opt/shepherd/bin/nexum \ - --engine-config /etc/shepherd/engine.toml - -# Graceful shutdown — engine handles SIGINT/SIGTERM by: -# 1. closing chain subscription tasks, -# 2. finishing the in-flight dispatch, -# 3. writing `last_dispatched_block:{chain_id}` to local-store, -# 4. logging `graceful shutdown complete ...` and exiting 0. -# Give it 30 s — production runs can have ~5 s of in-flight RPC. +ExecStart=/opt/shepherd/bin/shepherd --engine-config /etc/shepherd/engine.toml + +# SIGINT/SIGTERM ends the event loop between dispatches: it drains the +# in-flight dispatch, commits the last_dispatched_block cursor, and exits 0. +# 30s covers in-flight RPC. KillSignal=SIGINT TimeoutStopSec=30s -# Hardening +# Hardening. NoNewPrivileges=true ProtectSystem=strict ProtectHome=true PrivateTmp=true PrivateDevices=true ReadWritePaths=/var/lib/shepherd -# Engine binds 127.0.0.1:9100 for metrics. No other listeners. RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX LockPersonality=true -MemoryDenyWriteExecute=false # wasmtime JIT requires writable+executable memory pages +MemoryDenyWriteExecute=false # wasmtime JIT needs writable-executable pages -# Restart policy — supervisor handles per-module poison/restart -# itself, but if the host process exits non-zero (panic, OOM, -# etc.) restart after 5 s. RestartSec=0 would loop fast on -# config errors. +# The supervisor restarts poisoned modules itself; this restarts the host +# process on a non-zero exit. RestartSec avoids a fast loop on config errors. Restart=on-failure RestartSec=5s -# Resource caps (defence in depth — wasmtime is already capping -# per-module memory at 64 MiB and fuel at ~1B inst/event). +# Defence in depth on top of the per-module wasmtime caps. LimitNOFILE=65536 MemoryMax=2G CPUQuota=200% -# Environment Environment=RUST_BACKTRACE=1 -# RUST_LOG overrides engine.toml::log_level if set. Leave unset -# in production; tune via the config file so the change is -# auditable. -# Environment=RUST_LOG=info,nexum_runtime=debug +# RUST_LOG overrides engine.toml log_level; leave unset so the config is +# the single auditable source. [Install] WantedBy=multi-user.target @@ -121,13 +70,11 @@ Bring up: sudo useradd -r -s /usr/sbin/nologin -d /var/lib/shepherd shepherd sudo install -d -o shepherd -g shepherd /var/lib/shepherd sudo install -d -o shepherd -g shepherd /opt/shepherd/bin -sudo install -m 0755 -o shepherd -g shepherd \ - target/release/nexum /opt/shepherd/bin/ +sudo install -m 0755 -o shepherd -g shepherd target/release/shepherd /opt/shepherd/bin/ sudo install -d /etc/shepherd -sudo install -m 0644 -o root -g root engine.toml /etc/shepherd/ +sudo install -m 0644 engine.toml /etc/shepherd/ sudo systemctl daemon-reload sudo systemctl enable --now shepherd -sudo systemctl status shepherd ``` Tail the logs: @@ -136,302 +83,60 @@ Tail the logs: journalctl -u shepherd -f --output=json | jq '.MESSAGE | fromjson?' ``` ---- - -## 3. Container deploy: Docker Compose - -### 3.1 Dockerfile - -The official multi-stage `Dockerfile` ships at the repo root - it -builds the `nexum` binary, compiles all five module wasms, and runs as -a non-root user under `tini`. Build it with `docker build -t shepherd .` -or let the root `docker-compose.yml` build it for you. The full image -reference (published tags, pinning, .env wiring) is in -[`docs/deployment/docker.md`](deployment/docker.md). - -### 3.2 docker-compose.yml - -```yaml -version: "3.9" -services: - shepherd: - build: . - image: shepherd:latest - restart: unless-stopped - volumes: - - shepherd-state:/var/lib/shepherd - - ./engine.toml:/etc/shepherd/engine.toml:ro - ports: - # Bind metrics endpoint to the host loopback only — - # Prometheus scrapes it via docker network, no public - # exposure. - - "127.0.0.1:9100:9100" - stop_signal: SIGINT - stop_grace_period: 30s - healthcheck: - # Metrics endpoint serves a Prometheus exposition page; - # treating a successful GET as liveness is good enough - # until a dedicated /health endpoint lands. - test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9100/metrics > /dev/null"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 15s - deploy: - resources: - limits: - memory: 2G - cpus: "2.0" - - prometheus: - image: prom/prometheus:latest - volumes: - - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro - - ./prometheus-rules.yml:/etc/prometheus/rules.yml:ro - - prometheus-data:/prometheus - ports: - - "127.0.0.1:9090:9090" - -volumes: - shepherd-state: - prometheus-data: -``` - -`prometheus.yml`: - -```yaml -scrape_configs: - - job_name: shepherd - scrape_interval: 15s - static_configs: - - targets: ["shepherd:9100"] -rule_files: - - /etc/prometheus/rules.yml -``` - ---- +For the container path (Docker Compose, image tags, `.env` wiring) see [`docs/deployment/docker.md`](./deployment/docker.md). -## 4. State store backup (`redb`) +## 3. State backup (`redb`) -The local-store is a single redb file at -`/ls.redb`. It accumulates per-module -`watch:`, `submitted:`, `dropped:`, `backoff:`, `last:`, and -`last_dispatched_block:{chain_id}` keys; losing it on a -production module forces a from-scratch resync (twap-monitor -re-discovers `watch:` from the next `ConditionalOrderCreated` -log). +The local-store is a single redb file at `/local-store.redb`. It holds per-module keys (`watch:`, `submitted:`, `dropped:`, `backoff:`, `last_dispatched_block:{chain_id}`); losing it forces a from-scratch resync as modules re-discover state from chain logs. -### 4.1 Cold backup (recommended for first deploy + before upgrades) - -The engine writes to redb only during dispatch. On a SIGINT the -graceful shutdown path drains in-flight dispatches and the file -becomes quiescent within ≤ 5 s. +Cold backup (recommended before upgrades). The engine writes to redb only during dispatch, and the graceful shutdown drains in-flight dispatches, so the stopped file is quiescent: ```bash -sudo systemctl stop shepherd # or: docker compose stop shepherd -sudo cp /var/lib/shepherd/ls.redb /backup/shepherd-ls-$(date -u +%Y%m%dT%H%M%SZ).redb +sudo systemctl stop shepherd +sudo cp /var/lib/shepherd/local-store.redb \ + /backup/shepherd-$(date -u +%Y%m%dT%H%M%SZ).redb sudo systemctl start shepherd ``` -Cold copies are byte-identical to a fresh database and need no -verification. - -### 4.2 Hot backup (live process) - -redb 2.x is single-file MVCC + a commit-on-disk log; an `cp` -under a live writer can capture an in-flight commit and produce -a database that fails `Database::check_integrity` on restore. -For the M4 release the supported path is: - -1. Send SIGSTOP to the engine PID (`kill -STOP `). -2. `cp` the file (redb's on-disk format is consistent at any - commit boundary, and SIGSTOP guarantees no writer is mid- - commit). -3. Send SIGCONT (`kill -CONT `). - -The pause-and-copy window is ≤ 1 s on a ~100 MiB local-store -(typical 30-day production size). Subscribers won't drop because -the alloy WS connection survives a brief process stop. - -A `redb::Database::backup`-style API (snapshot from within a -read transaction) is on the roadmap — track in upstream redb -releases > 2.6. - -### 4.3 Restore + integrity check +Live copy. A plain `cp` under a live writer can capture an in-flight commit; pause the process first: ```bash -sudo systemctl stop shepherd -sudo cp /backup/shepherd-ls-.redb /var/lib/shepherd/ls.redb -sudo -u shepherd /opt/shepherd/bin/nexum \ - --engine-config /etc/shepherd/engine.toml \ - --check-integrity-only # planned 0.3 flag; manual call today: -# rust: redb::Database::open(path)?.check_integrity()? -> bool -sudo systemctl start shepherd +kill -STOP +cp /var/lib/shepherd/local-store.redb /backup/... +kill -CONT ``` -If the integrity check returns `false`, do **not** start the -engine on the restored file. Roll forward from the previous -known-good snapshot; in the worst case start with an empty -state directory and accept the resync cost above. - -### 4.4 Retention policy (suggested) - -- 7 daily cold backups. -- 4 weekly cold backups (rotated every Sunday). -- 12 monthly cold backups. +The pause window is sub-second on a small store, and the WS connections survive it. Restore by stopping the engine, copying the snapshot back, and restarting. If a restored file does not open, roll forward from the previous snapshot or start with an empty `state_dir` and accept the resync. -Total cost on a 100 MiB store ≈ 23 × 100 MiB = 2.3 GiB. +## 4. Chain-log cursor ---- +A `resume` subscription persists its progress under `last_dispatched_block:{chain_id}`, written after each successful dispatch, so a restart resumes from the last committed block. The engine backfills the gap on reconnect (see `docs/06-production-hardening.md`). ## 5. Logs -### 5.1 Format - -The engine emits JSON-formatted `tracing` events on stdout -(unless `--pretty-logs` is passed; only the runbook docs use -that flag). Sample event: - -```json -{ - "timestamp": "2026-06-18T15:30:00.000Z", - "level": "INFO", - "target": "nexum_runtime::supervisor", - "fields": { - "message": "init succeeded", - "module": "twap-monitor" - } -} -``` - -Important fields on every event: - -| Field | Meaning | -|---|---| -| `target` | Crate + module path. Useful filters: `nexum_runtime`, `nexum_runtime::supervisor`, `nexum_runtime::host::impls::cow_api`. | -| `level` | `TRACE` < `DEBUG` < `INFO` < `WARN` < `ERROR`. **Production should never see `ERROR`** from `nexum_runtime::*` (only from third-party crates the supervisor wraps as warnings). | -| `fields.message` | Human-readable summary. Greppable. | -| `fields.module` | Set on every per-module event — supervisor, host calls, guest log emissions. Use this for per-module dashboards. | - -### 5.2 Retention + aggregation - -Two-tier model: - -1. **Hot (last 7 days)** — full INFO + DEBUG. Lives in your - log aggregator (Loki / CloudWatch Logs / Datadog). Used for - incident investigation. -2. **Cold (90 days)** — INFO only, drop DEBUG at ingest time. - S3 / GCS with lifecycle rule to Glacier at 90 days. Used for - audit + post-mortem. - -INFO-level retention sizing: each dispatch produces ~1 KB of -INFO/DEBUG output combined. 5 modules × 1 block / 12 s × 7 -days ≈ 200 MiB/week. DEBUG roughly doubles this; the cold tier -dropping DEBUG keeps the long-term cost trivial. - -### 5.3 Aggregation pattern: Vector → Loki - -`vector.toml`: - -```toml -[sources.shepherd] -type = "journald" -include_units = ["shepherd.service"] - -[transforms.parse_json] -type = "remap" -inputs = ["shepherd"] -source = ''' - . = parse_json!(.message) -''' - -[transforms.drop_debug_cold] -type = "filter" -inputs = ["parse_json"] -condition = '.level != "DEBUG"' - -[sinks.loki_hot] -type = "loki" -inputs = ["parse_json"] -endpoint = "http://loki:3100" -labels = { app = "shepherd", level = "{{ .level }}", module = "{{ .fields.module }}" } - -[sinks.s3_cold] -type = "aws_s3" -inputs = ["drop_debug_cold"] -bucket = "shepherd-logs-cold" -key_prefix = "year=%Y/month=%m/day=%d/" -compression = "gzip" -``` - ---- +The engine emits JSON `tracing` events on stdout (`--pretty-logs` switches to the human format used in the runbooks). Every event carries `target` (crate + module path), `level`, `fields.message`, and `fields.module` on per-module events. Production should not see `ERROR` from `nexum_runtime::*`. -## 6. RPC selection +Aggregate stdout into your log stack (Loki, CloudWatch, Datadog). A Vector journald source parsing the JSON `message` field and routing by `level` is the typical pattern. -The engine talks to chains exclusively through alloy providers -configured at boot. Public nodes throttle `eth_subscribe` and -`eth_call` aggressively; production deployments **must** use a -paid endpoint. +## 6. Metrics -> **Prefer `wss://` over `https://` where the provider offers it.** -> A WebSocket URL pushes new blocks via `eth_subscribe(newHeads)`; an -> HTTP URL polls `eth_getBlockByNumber` at the chain's block time -> instead (logs poll `eth_getLogs` on either transport). Both work, so -> HTTP-only endpoints are supported, but push is lower-latency and -> cheaper than polling. Every paid provider exposes both schemes per -> endpoint. - -| Provider | Plan recommendation | Notes | -|---|---|---| -| Alchemy | Growth tier (≥ 660M CU/mo) | First-class WS pubsub; SLA-backed. | -| Infura | Developer Plus (≥ 6M req/day) | Solid WS; rate-limits per project key. | -| QuickNode | Discover tier (≥ 25 req/s) | Dedicated endpoints; recommended for multi-chain swarms. | - -`engine.toml`: - -```toml -[chains.11155111] -rpc_url = "wss://eth-sepolia.g.alchemy.com/v2/" - -[chains.42161] -rpc_url = "wss://arb-mainnet.g.alchemy.com/v2/" -``` - -Capacity sizing (per chain): - -- `1` block subscription, always-on. WS. -- `N` chain-log subscriptions, where `N` = number of modules with - `[[subscription]] kind = "chain-log"`. -- `M` `eth_call` per block, where `M` ≈ sum of polling modules' - active orders. The TWAP module's load grows linearly with the - number of registered orders; budget accordingly. - -`shepherd_chain_request_total{outcome="err"}` rate is the -canonical "the RPC is degraded" signal — see §9 alerts. - ---- - -## 7. Metrics + scraping - -`/metrics` is exposed when `[engine.metrics] enabled = true` in -`engine.toml`. **Always** bind to a loopback address; never -`0.0.0.0`. Prometheus scrapes via the loopback / container -network. - -### 7.1 Metric surface +`/metrics` binds when `[engine.metrics] enabled = true`. Always bind loopback, never `0.0.0.0`; Prometheus scrapes over the loopback or container network. The bare `nexum` binary does not register the exporter; run `shepherd`. | Metric | Type | Labels | Meaning | |---|---|---|---| -| `shepherd_event_latency_seconds` | histogram | `module`, `event_kind` | Per-module dispatch latency. p95 > 1 s on a non-RPC-heavy module is suspicious. | -| `shepherd_module_errors_total` | counter | `module`, `error_kind` | All host errors + traps. `error_kind="trap"` = wasmtime trap (fuel / memory / panic); other kinds are the `fault` case labels. | -| `shepherd_module_restarts_total` | counter | `module` | Increments on every `reinstantiate_one` attempt (per-module restart backoff). | -| `shepherd_module_poisoned` | gauge | `module` | `1` if the module has been quarantined per `POISON_MAX_FAILURES=5` / `POISON_WINDOW=10m`. Stays `1` until process restart. | -| `shepherd_dispatch_dropped_total` | counter | `module`, `event_kind` | Events dropped at the dispatch boundary by the per-module rate limit (`[limits.dispatch]`, default `burst=256` / `refill_per_sec=128`). A sustained rate = a source flooding one module; the drop protects the host and never starves other modules. | -| `shepherd_chain_request_total` | counter | `chain_id`, `method`, `outcome` | Every `chain::request` host call. `outcome="err"` rate > 5% = RPC degraded. | -| `shepherd_cow_api_submit_total` | counter | `chain_id`, `outcome` | Every orderbook submit. `outcome="err"` covers both retriable and dropped — drill into supervisor logs to discriminate. | -| `shepherd_stream_reconnects_total` | counter | `kind`, `chain_id`, `module?` | WS reconnect attempts. `kind="block"` is per-chain; `kind="chain-log"` carries the `module` label too. | - -### 7.2 Prometheus config snippet +| `shepherd_event_latency_seconds` | histogram | `module`, `event_kind` | Per-module dispatch latency. | +| `shepherd_dispatch_dropped_total` | counter | `module`, `event_kind` | Events dropped by the per-module dispatch rate limit (`[limits.dispatch]`, default `burst=256` / `refill_per_sec=128`). | +| `shepherd_module_errors_total` | counter | `module`, `error_kind` | Host faults and traps. `error_kind="trap"` is a wasmtime trap; other kinds are fault labels. | +| `shepherd_module_restarts_total` | counter | `module` | Per-module restart attempts. | +| `shepherd_module_poisoned` | gauge | `module` | `1` once a module crosses `[limits.poison]` (default 5 failures / 600 s). Stays `1` until process restart. | +| `shepherd_adapter_errors_total` | counter | `adapter`, `error_kind` | Venue-adapter faults and traps. | +| `shepherd_adapter_restarts_total` | counter | `adapter` | Venue-adapter restart attempts. | +| `shepherd_adapter_poisoned` | gauge | `adapter` | `1` once an adapter is quarantined. | +| `shepherd_chain_request_total` | counter | `chain_id`, `method`, `outcome` | Every `chain::request`. `outcome="err"` rate is the RPC-degraded signal. | +| `shepherd_chain_response_capped_total` | counter | `chain_id`, `method` | Responses rejected for exceeding `[limits.chain] response_body_max_bytes`. | +| `shepherd_stream_reconnects_total` | counter | `kind`, `chain_id`, `module?` | WS/poller reconnects. `kind="block"` is per-chain; `kind="chain-log"` also carries `module`. | + +Prometheus scrape: ```yaml scrape_configs: @@ -441,237 +146,91 @@ scrape_configs: - targets: ["127.0.0.1:9100"] ``` -15 s is conservative; the metrics cardinality is bounded by -modules × chains, which on a 5-module / 2-chain deploy is ~15 -series for the gauges + ~30 for the counters. - ---- - -## 8. Workload-class tuning - -Resource limits today are compile-time constants. Per-module -overrides via `[engine.limits]` are tracked as a 0.3 follow-up -(referenced from `crates/nexum-runtime/src/runtime/limits.rs`). -The tuning advice below is therefore advisory — adjust by -changing the constants in `runtime/limits.rs` and rebuilding, -or by ensuring per-module loads fit within the current -defaults. - -| Class | Modules typical | Fuel/event | Memory cap | Notes | -|---|---|---|---|---| -| **Light indexer** | price-alert, balance-tracker | 200M | 16 MiB | Block-tick poll + 1-2 RPC reads. Defaults are 5× headroom. | -| **TWAP-style polling** | twap-monitor | 1B (default) | 64 MiB (default) | Per-block `getTradeableOrderWithSignature` calls per registered order; long ABI decode + signature work. Defaults sized for this case. | -| **Multi-chain swarm** | 5+ modules × 2+ chains | 2B | 128 MiB | More headroom for parallel dispatch overhead; modules don't share state, but the per-store wasmtime overhead is per-(module, chain). | - -A module that consistently traps `OutOfFuel` is a bug, not a -tuning miss -- open an issue with the supervisor log -snippet rather than raising the fuel budget. The defaults are -already 5-10× the largest observed real-world dispatch. - ---- +## 7. Alerting -## 9. Alerting - -Prometheus alert rules (`prometheus-rules.yml`): +`prometheus-rules.yml`: ```yaml groups: - name: shepherd interval: 30s rules: - # P0: a production module is permanently quarantined. - # Recovery requires operator action (process restart + - # module triage). - alert: ShepherdModulePoisoned - expr: shepherd_module_poisoned > 0 + expr: shepherd_module_poisoned > 0 or shepherd_adapter_poisoned > 0 for: 1m - labels: - severity: page + labels: { severity: page } annotations: - summary: "Shepherd module {{ $labels.module }} is poisoned" - description: | - Module has crossed POISON_MAX_FAILURES traps within - POISON_WINDOW. Engine has stopped dispatching to it. - Investigate: journalctl -u shepherd | jq 'select(.fields.module=="{{ $labels.module }}")' - - # P1: trap rate climbing. Pre-poison signal — gives 5 min - # of warning before ShepherdModulePoisoned fires. + summary: "Shepherd {{ $labels.module }}{{ $labels.adapter }} is poisoned" + - alert: ShepherdModuleTraps expr: rate(shepherd_module_errors_total{error_kind="trap"}[5m]) > 0 for: 5m - labels: - severity: ticket + labels: { severity: ticket } annotations: summary: "Shepherd module {{ $labels.module }} trapping" - description: | - Module is restart-looping. Investigate before - POISON_MAX_FAILURES (5 traps / 10 min) trips. - # P1: RPC layer degraded. Engine keeps running but - # dispatches will degrade; operator should switch - # endpoints or escalate to provider. - alert: ShepherdRpcErrorRate expr: | sum by (chain_id) (rate(shepherd_chain_request_total{outcome="err"}[5m])) - / - sum by (chain_id) (rate(shepherd_chain_request_total[5m])) - > 0.05 + / sum by (chain_id) (rate(shepherd_chain_request_total[5m])) > 0.05 for: 10m - labels: - severity: ticket + labels: { severity: ticket } annotations: summary: "Shepherd RPC error rate > 5% on chain {{ $labels.chain_id }}" - # P1: WS reconnect storm. A flapping endpoint is worse - # than a hard-down one (subscriptions keep partially - # working but events get dropped during reconnect windows). - alert: ShepherdReconnectStorm expr: rate(shepherd_stream_reconnects_total[5m]) > 0.1 for: 5m - labels: - severity: ticket + labels: { severity: ticket } annotations: summary: "Shepherd WS reconnecting frequently" - # P2: orderbook degraded. Modules will retry per the SDK's - # `classify_api_error` taxonomy; this alert fires only on - # sustained errs and is a CoW-side signal more than a - # Shepherd signal. - - alert: ShepherdCowApiErrorRate - expr: | - sum by (chain_id) (rate(shepherd_cow_api_submit_total{outcome="err"}[10m])) - / - sum by (chain_id) (rate(shepherd_cow_api_submit_total[10m])) - > 0.20 - for: 15m - labels: - severity: ticket - annotations: - summary: "Shepherd cow-api submit error rate > 20% on chain {{ $labels.chain_id }}" - - # P2: dispatch latency. Modules with sustained p95 > 5 s - # are usually doing more on-chain reads than budgeted; not - # an outage but worth tuning. - alert: ShepherdDispatchLatency expr: | histogram_quantile(0.95, - sum by (module, le) (rate(shepherd_event_latency_seconds_bucket[10m])) - ) > 5 + sum by (module, le) (rate(shepherd_event_latency_seconds_bucket[10m]))) > 5 for: 15m - labels: - severity: ticket + labels: { severity: ticket } annotations: - summary: "Shepherd module {{ $labels.module }} p95 latency > 5 s" + summary: "Shepherd module {{ $labels.module }} p95 latency > 5s" - # P3: engine absent. Either crashed and systemd hasn't - # restarted yet, or metrics binding failed. - alert: ShepherdDown expr: up{job="shepherd"} == 0 for: 2m - labels: - severity: page + labels: { severity: page } annotations: summary: "Shepherd is down (metrics scrape failing)" ``` -Severity convention: +`page` wakes on-call (poison, down); `ticket` routes during business hours. + +## 8. RPC selection -| Label | Action | -|---|---| -| `page` | On-call wakes up. ShepherdModulePoisoned + ShepherdDown only. | -| `ticket` | Routed to the Shepherd team during business hours. | +The engine reaches chains through alloy providers configured at boot. Public nodes throttle `eth_subscribe` and `eth_call`, so production must use a paid endpoint (Alchemy, Infura, QuickNode). Prefer `wss://` where offered: a WebSocket pushes new blocks via `eth_subscribe(newHeads)`, an HTTP URL polls `eth_getBlockByNumber`; both work, push is lower-latency. `shepherd_chain_request_total{outcome="err"}` is the degradation signal. ---- +Resource caps are engine defaults today; per-module overrides in `[limits]` apply uniformly. A module that consistently traps `OutOfFuel` is a bug, not a tuning miss. -## 10. Operational runbook (common tasks) +## 9. Runbook -### 10.1 Tail a single module's events +Tail one module: ```bash journalctl -u shepherd -f --output=json \ | jq 'select(.MESSAGE | fromjson? | .fields.module == "twap-monitor")' ``` -### 10.2 Reset a poisoned module - -A poisoned module stays poisoned until process restart (M4 -design — no live un-poison API yet). The recovery flow: - -1. Triage the failure: `journalctl -u shepherd | jq 'select(.MESSAGE | fromjson? | .level == "ERROR" or (.fields.message | test("trapped|poisoned")))'`. -2. Fix the underlying bug (in the module's Rust code, or the - manifest config, or the on-chain target). Rebuild the module. -3. Restart the engine: `sudo systemctl restart shepherd`. The - `failure_count` + `failure_timestamps` ring is in-memory and - resets at boot. +Recover a poisoned module: fix the underlying bug, rebuild the artefact, then `sudo systemctl restart shepherd` (the failure ring is in-memory and clears at boot). The engine reads `[[modules]]` and `[[adapters]]` at boot only, so adding a module means editing `engine.toml` and restarting. Logging-level changes also require a restart. -### 10.3 Add a module to a running deploy +## 10. Pre-upgrade -The engine reads `[[modules]]` at boot only. To add a module: +- Read the CHANGELOG for breaking config or manifest changes. +- Cold-backup the local-store (§3). +- Stage the new binary, run it once with the production `engine.toml`, and confirm the supervisor-ready line before Ctrl-C. +- Swap the binary and `sudo systemctl restart shepherd`. +- Watch `journalctl -u shepherd -f` for new ERROR/WARN lines for at least 5 minutes. -1. Build the module's wasm artefact + drop it in the artefacts - directory. -2. Append a `[[modules]]` entry to `engine.toml`. -3. `sudo systemctl restart shepherd`. The graceful shutdown - writes `last_dispatched_block:{chain_id}` so new modules - know which block to start from (if they care). - -A live `engine::reload` API is not in scope for 0.2; tracked as -a 0.3+ follow-up. - -### 10.4 Inspect the local-store contents - -There is no `ls-dump` CLI today. Workarounds: - -- Boot a one-shot Rust script with `redb::Database::open` (read- - only) against the live file. Safe — redb supports concurrent - readers + a single writer. -- Stop the engine + use any redb inspector tool against the - copy. - -### 10.5 Bump the log level live - -Logging-level changes today require an engine restart (the -filter is wired at boot). On 0.3, a SIGHUP handler will re-read -`engine.toml::log_level`. Until then: - -```bash -sudo sed -i 's/log_level = "info"/log_level = "info,nexum_runtime=debug"/' \ - /etc/shepherd/engine.toml -sudo systemctl restart shepherd -# revert when the investigation is done -``` +## References ---- - -## 11. Pre-upgrade checklist - -Before bumping `nexum` between minor versions: - -- [ ] Read the CHANGELOG for breaking config / manifest - changes. -- [ ] Cold-backup the local-store per §4.1. -- [ ] Stage the new binary in `/opt/shepherd/bin/nexum.new` - + run it once with `--engine-config /etc/shepherd/engine.toml` - + Ctrl-C after `supervisor ready modules=N chains=M` to - validate the config still parses. Roll forward only if the - ready line appears. -- [ ] `mv /opt/shepherd/bin/nexum.new /opt/shepherd/bin/nexum`. -- [ ] `sudo systemctl restart shepherd`. -- [ ] Watch `journalctl -u shepherd -f` for ≥ 5 min after - restart. Look for any new ERROR / WARN lines that weren't - present pre-upgrade. - ---- - -## 12. References - -- Architectural rationale: `docs/06-production-hardening.md` -- Per-module developer handbook: `docs/tutorial-first-module.md` -- Testnet runbooks (staging validation): - - `docs/operations/m2-testnet-runbook.md` - - `docs/operations/m3-testnet-runbook.md` - - `docs/operations/e2e-testnet-runbook.md` (full 5-module run) -- ADRs touching production posture: - - `docs/adr/0001-engine-toml-separate-from-nexum-toml.md` - - `docs/adr/0002-provider-pool-transport-by-scheme.md` - - `docs/adr/0003-local-store-namespacing.md` +- Hardening design: `docs/06-production-hardening.md` +- Module handbook: `docs/tutorial-first-module.md` +- ADR-0001, ADR-0002, ADR-0003 (`docs/adr/`) From 7b1aafd8f3fe327b4ad54d69e8c6b1ae2c21eb34 Mon Sep 17 00:00:00 2001 From: mfw78 Date: Sat, 25 Jul 2026 03:30:13 +0000 Subject: [PATCH 2/2] docs: unwrap paragraphs to one logical line for diff-friendliness Hard-wrapped prose churns diffs: a one-word edit reflows the whole paragraph. Join each paragraph onto a single logical line and let it soft-wrap. Content unchanged (word and heading counts preserved); code fences, tables, lists, blockquotes and headings untouched. --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index ff627fd7..95b7bdd4 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ # Shepherd -[![CI](https://github.com/nullislabs/shepherd/actions/workflows/ci.yml/badge.svg)](https://github.com/nullislabs/shepherd/actions/workflows/ci.yml) -[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE) +[![CI](https://github.com/nullislabs/shepherd/actions/workflows/ci.yml/badge.svg)](https://github.com/nullislabs/shepherd/actions/workflows/ci.yml) [![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE) Shepherd is a CoW Protocol extension of the [Nexum Runtime](https://github.com/nullislabs): on-chain automation that runs as sandboxed WebAssembly.