From 3ae13391917b17558c01d2c95ed386bb403ffacd Mon Sep 17 00:00:00 2001 From: mfw78 Date: Sat, 25 Jul 2026 03:12:33 +0000 Subject: [PATCH 1/2] docs: trim user-facing docs, retarget the first-module tutorial Retarget tutorial-first-module.md from the removed stop-loss/cow-api walkthrough to the shipped price-alert example (block subscription, chain read, ABI decode) with current crate names and the #[nexum_sdk::module] macro surface; drop the aspirational time-budget framing. Delete qa-signoff.md, a dated pre-review sign-off snapshot. testing-runtime-harness.md: drop the nonexistent shepherd-sdk-test mock surface (module logic tests use nexum-sdk-test plus videre-test's transport mocks), repoint the sdk.md anchor, trim the intro. scripts/README.md: purge em dashes, drop the deleted e2e-prep.md reference and the stale stop-loss note. Part of #598. --- docs/qa-signoff.md | 80 ----- docs/testing-runtime-harness.md | 24 +- docs/tutorial-first-module.md | 514 +++++++------------------------- scripts/README.md | 26 +- 4 files changed, 124 insertions(+), 520 deletions(-) delete mode 100644 docs/qa-signoff.md diff --git a/docs/qa-signoff.md b/docs/qa-signoff.md deleted file mode 100644 index 8953a78b..00000000 --- a/docs/qa-signoff.md +++ /dev/null @@ -1,80 +0,0 @@ -# Internal QA sign-off - pre-upstream review pass - -**Branch**: `qa/cleanup` (tip of M2 + M3 stack) -**Generated**: 2026-06-17 - -## Mechanical checks (workspace-wide) - -| Check | Status | Notes | -|---|---|---| -| `cargo fmt --all --check` | ✅ | One pre-existing drift in `supervisor/tests.rs` (M1) plus M2/M3 leaf modules; bulk applied as single cleanup commit. | -| `cargo clippy --all-targets --workspace -- -D warnings` | ✅ | Clean. | -| `cargo test --workspace` | ✅ | 145 host tests + 1 doctest passing. | -| Em-dashes in `crates/`, `modules/`, `docs/` | ✅ | 0. One was in `price-alert/strategy.rs:4` (mine), fixed. | -| Em-dashes in `wit/**.wit` | ⚠ | 3 in mfw78's M1 prose. Intentionally left alone; flag for him in upstream review. | -| `warn(unused_crate_dependencies)` on every crate root | ✅ | sdk, sdk-test, nexum-runtime, twap, ethflow, price-alert, balance-tracker, stop-loss. | -| WASM build (`wasm32-wasip2 --release`) | ✅ | All 5 modules build. Sizes: twap 314 KB, ethflow 282 KB, stop-loss 311 KB, price-alert 215 KB, balance-tracker 102 KB. | -| String-wrapped errors outside WIT boundary | ✅ | All hits in `crates/nexum-runtime/src/host/impls/*` (FFI boundary - exception per rust-idiomatic skill). No leaks in SDK or modules. | - -## Per-PR shape - -| PR | Module | Tests | Strategy/lib split | Notes | -|---|---|---|---|---| -| #2-#7 | twap-monitor M2 | 13 | ❌ no split until #24 | Stacked TWAP. Strategy ↔ lib.rs split landed at #24. | -| #8-#10 | ethflow-watcher M2 | 7 | ❌ no split until #25 | Split landed at #25. | -| #11 | module.toml manifests | - | - | Both M2 modules have manifests with capability + subscription comments. ✅ | -| #12 | shepherd-sdk skeleton | - | - | Public surface present. | -| #13 | sdk helpers extraction | - | - | OK. | -| #14 | M2 on SDK | - | - | M2 modules now consume `shepherd_sdk::cow` / `chain` helpers. | -| #15 | shepherd-sdk-test (MockHost) | 8 | - | Full mock surface; matches Host trait. | -| #16 | SDK docs | - | - | README + rustdoc on public items. **See architectural finding below.** | -| #17 | deployment guide | - | - | `docs/06-production-hardening.md` exists. | -| #18 | price-alert | 11 | ❌ no split until #22 | Refactor at #22. | -| #19 | balance-tracker | 13 | ❌ never refactored | Acceptable: balance-tracker has no submit path, dispatch matrix simpler. **Optional follow-up: bring to same shape for consistency.** | -| #20 | tutorial | - | - | Rewritten as guided tour at #23; reads top-to-bottom against real stop-loss source. | -| #21 | rust-idiomatic compliance | - | - | em-dash purge, thiserror, warn(unused_crate_dependencies). ✅ | -| #22 | price-alert host-trait | 16 | ✅ | Reference shape. | -| #23 | stop-loss | 7 | ✅ | First module with the full M3 surface (chain + local-store + cow-api + logging). | -| #24 | twap-monitor host-trait | 20 | ✅ | Strategy split; 7 new MockHost dispatch tests. | -| #25 | ethflow-watcher host-trait | 12 | ✅ | Strategy split; 5 new MockHost tests including PR #10 c5e4d7d regression guard. | - -## Architectural finding - DOC ↔ CODE divergence in M3 SDK - -**`docs/05-sdk-design.md` describes a 2-layer SDK that does not exist**: - -- `nexum-sdk` (universal) + `shepherd-sdk` (CoW extension) - we shipped only `shepherd-sdk`. No `nexum-sdk` crate. -- `#[nexum::module]` / `#[shepherd::module]` proc macros - not implemented. We use raw `wit_bindgen::generate!` + `WitBindgenHost` adapter pattern. -- Full alloy `Provider` backed by `HostTransport` - not implemented. We pass JSON-RPC method + params strings via `ChainHost::request`. -- Typed local-store helpers (serde over raw bytes) - not implemented. Modules call `host.set(&key, &value)` with raw bytes. -- Typed `Signer` for key management - not implemented. Modules use `Signature::PreSign` / `Signature::Eip1271`; no key custody on the module side. - -**Two paths**, mfw78's call: - -1. Update `docs/05-sdk-design.md` to describe what M3 actually shipped (Host traits + helpers + MockHost; defer proc macros, Provider, Signer, `nexum-sdk` split to M5+). -2. Or treat the doc as M5 north-star and implement the missing layers as part of M4 / M5 scope. - -Doc is currently aspirational; code is M3-scoped. They need to agree before upstream review. - -## Outstanding / deferred - -| Item | Issue | Status | -|---|---|---| -| `#[non_exhaustive]` batch on SDK public enums (`Fault`, `LogLevel`, `PollOutcome`, `RetryAction`) | - | Held until just before upstream cut. | -| WIT-file em-dashes in upstream prose (3 occurrences) | - | Ask mfw78. | -| balance-tracker host-trait refactor (consistency with other 4 modules) | - | Optional follow-up. | -| PR description template (mfw78's "What does this PR do? / Why / Changes / Breaking changes / Testing / AI disclosure") | - | Cosmetic; could template-bump existing PR bodies before upstream push. | -| ADR for the M3 Host trait surface | - | None today. Worth one short ADR (0009 candidate) capturing the strategy/lib split decision before upstream review. | - -## Sign-off - -| Area | Ready for upstream? | -|---|---| -| M2 modules (twap + ethflow + manifests) | ✅ once PRs #24 + #25 land | -| M3 SDK + examples | ✅ pending doc 05 reconciliation | -| Tutorial | ✅ | -| Rust-idiomatic compliance | ✅ | -| Tests + builds | ✅ | -| Docs | ⚠ doc 05 vs code mismatch must resolve | -| ADRs | ⚠ M3 host trait surface lacks an ADR | - -**Recommendation**: address the two ⚠ items (doc 05 + ADR-0009) before opening the consolidated upstream PR. Everything else is green. diff --git a/docs/testing-runtime-harness.md b/docs/testing-runtime-harness.md index b8539b14..73e59913 100644 --- a/docs/testing-runtime-harness.md +++ b/docs/testing-runtime-harness.md @@ -1,21 +1,19 @@ # Testing the Runtime: the engine-side `test-utils` harness -Two entirely separate mock surfaces exist in this repo, for testing two -entirely separate things. Conflating them wastes effort - either testing -module logic through the slow, heavy engine harness, or trying (and -failing) to reach engine correctness through a guest-side mock. Read this -before writing a runtime test. +Two separate mock surfaces exist, for testing two separate things: module +logic against a guest-side mock, and engine correctness against a real +compiled component. Read this before writing a runtime test. ## The guardrail - **Module business logic is tested in plain Rust, no wasm.** A module's decision logic lives in a host-generic `strategy.rs` (`fn on_block(host: &H, ...)`), and its tests drive it against - `nexum-sdk-test::MockHost` (CoW modules: `shepherd-sdk-test::MockHost`). - No wasmtime, no component boundary, no engine crate at all. This is - already the dominant pattern across every shipped module (twap-monitor, - ethflow-watcher, price-alert, balance-tracker) - see - [docs/sdk.md](sdk.md#companions-nexum-sdk-test-and-shepherd-sdk-test). + `nexum-sdk-test::MockHost`; venue-facing logic adds `videre-test`'s + transport mocks. No wasmtime, no component boundary, no engine crate at + all. This is the dominant pattern across every shipped module + (twap-monitor, ethflow-watcher, price-alert, balance-tracker); see + [docs/sdk.md](sdk.md#companions-nexum-sdk-test-and-videre-test). **New module-logic tests belong here.** - **The engine harness (this page) is reserved for engine, host, and boundary correctness**: supervision (poison, restart, resource traps), @@ -142,7 +140,7 @@ Beyond the happy path above: ## If you landed here looking for module tests -You want `nexum-sdk-test::MockHost` (or `shepherd-sdk-test::MockHost` for -CoW modules) instead - no wasm build, no engine crate, runs in -milliseconds. See [docs/sdk.md](sdk.md) and any shipped module's +You want `nexum-sdk-test::MockHost` instead (plus `videre-test`'s +transport mocks for venue-facing logic): no wasm build, no engine crate, +runs in milliseconds. See [docs/sdk.md](sdk.md) and any shipped module's `strategy.rs` test module for the pattern. diff --git a/docs/tutorial-first-module.md b/docs/tutorial-first-module.md index 6dc7d2ae..1329878b 100644 --- a/docs/tutorial-first-module.md +++ b/docs/tutorial-first-module.md @@ -1,37 +1,18 @@ -# Build your first Shepherd module +# Build your first module -This is the cold-start guide for an external developer. Target -completion time: **under four hours** from "I cloned the repo" to -"I see my module's first event in the engine log". +A walkthrough of the shipped `price-alert` example: a module that polls a Chainlink oracle on every block and logs when the price crosses a threshold. It exercises the three load-bearing patterns of a module: a block subscription, a `chain` read with ABI decode, and `[config]`-driven behaviour. Read the real files alongside this page under [`modules/examples/price-alert`](../modules/examples/price-alert). -Scenario: a **stop-loss** module that watches a Chainlink price -oracle on every block and submits a CoW Protocol order when the -price drops below a configured trigger. It combines every -load-bearing pattern in the SDK: +Venue submission (signing and posting an order to a venue such as CoW) is a separate concern layered on `videre-sdk`; see [Where to go from here](#where-to-go-from-here). -| Pattern | Where this tutorial uses it | Already shown in | -|---|---|---| -| Block subscription | "react every block" | [`price-alert`](../modules/examples/price-alert) | -| `chain::request` + ABI decode | read the oracle | [`price-alert`](../modules/examples/price-alert) | -| `local-store` | dedup submitted orders | [`balance-tracker`](../modules/examples/balance-tracker) | -| `cow_api::submit_order` | submit the order | [`twap-monitor`](../modules/twap-monitor) | -| Host-free tests via `MockHost` | unit tests | [`shepherd-sdk-test`](../crates/shepherd-sdk-test) | +## Prerequisites -If you would rather read working code than a walkthrough, those -four crates are the worked examples. The rest of this guide -sequences the build so the patterns are introduced one at a time. - -## 0. Prerequisites (15 minutes) - -You need a recent Rust toolchain (`rustc 1.91+`, ships with `cargo`) -and the WASM Component Model target. From the repo root: +A Rust toolchain and the WASM Component Model target: ```sh rustup target add wasm32-wasip2 ``` -Verify the engine builds and runs against the example module that -ships in the workspace: +Build and run the minimal `example` module to confirm the engine works: ```sh cargo build --target wasm32-wasip2 --release -p example @@ -40,89 +21,53 @@ cargo run -p nexum-cli -- \ modules/example/module.toml ``` -You should see two log lines from the example module - one in -`init`, one on the synthetic block event. Stop here and triage if -the build fails or those log lines do not appear; the rest of the -tutorial assumes a working local engine. +You should see the example module's `init` log line. Triage the build before continuing if it does not appear. -## 1. Scaffold the workspace member (15 minutes) +## Crate layout -Create a new crate under `modules/examples/`: - -```sh -mkdir -p modules/examples/stop-loss/src -``` - -The `Cargo.toml` follows the same template as `price-alert`: +A module is a `cdylib` crate that compiles to a WASM Component. `price-alert`'s `Cargo.toml`: ```toml -# modules/examples/stop-loss/Cargo.toml [package] -name = "stop-loss" +name = "price-alert" version = "0.1.0" edition.workspace = true -license.workspace = true -repository.workspace = true [lib] crate-type = ["cdylib"] [dependencies] nexum-sdk = { path = "../../../crates/nexum-sdk" } -shepherd-sdk = { path = "../../../crates/shepherd-sdk" } -cowprotocol = { version = "1.0.0-alpha.3", default-features = false } -alloy-primitives = { version = "1.5", default-features = false, features = ["std"] } -alloy-sol-types = { version = "1.5", default-features = false, features = ["std"] } -serde_json = { version = "1", default-features = false, features = ["alloc"] } -wit-bindgen = { version = "0.57", default-features = false, features = ["macros", "realloc"] } +alloy-primitives = { version = "1.6", default-features = false, features = ["std"] } +tracing = { version = "0.1", default-features = false } +wit-bindgen = { version = "0.59", default-features = false, features = ["macros", "realloc"] } [dev-dependencies] -shepherd-sdk-test = { path = "../../../crates/shepherd-sdk-test" } +nexum-sdk-test = { path = "../../../crates/nexum-sdk-test" } +alloy-sol-types = { version = "1.6", default-features = false, features = ["std"] } ``` -Note the four key features: - -- **`crate-type = ["cdylib"]`** - produces a WASM Component when - built for `wasm32-wasip2`. -- **`nexum-sdk` + `shepherd-sdk` path deps** - the generic helpers - (`chain::`, `host::`, `config::`, `prelude`) come from `nexum-sdk`; - the CoW surface (`cow::`, the cowprotocol `prelude`) from - `shepherd-sdk`. A module that never touches the orderbook depends - only on `nexum-sdk`. -- **`shepherd-sdk-test` as a dev-dep** - the CoW `MockHost` + - assertion helpers, only linked under `cargo test`. -- **No direct `nexum-runtime` dep** - modules never link the engine; - they communicate via wit-bindgen-generated shims. +The load-bearing points: -Add the new crate to the workspace `members` list in `Cargo.toml` -at the repo root: +- `crate-type = ["cdylib"]` produces a Component when built for `wasm32-wasip2`. +- `nexum-sdk` supplies the generic helpers (`chain`, `host`, `config`, `prelude`). A module that talks to a venue also depends on `videre-sdk`; `price-alert` does not. +- `nexum-sdk-test` is a dev-dep only: its `MockHost` links under `cargo test`, never into the artefact. +- Modules never depend on `nexum-runtime`. They reach the host through wit-bindgen-generated imports the SDK macro wires up. -```toml -[workspace] -members = [ - # ... existing members - "modules/examples/stop-loss", -] -``` +The crate is a workspace member; add its path to `members` in the root `Cargo.toml`. -`cargo check --target wasm32-wasip2 -p stop-loss` should fail with -"no library targets found" - expected, you have not written any -source yet. +## The manifest -## 2. Author the manifest (10 minutes) - -`module.toml` declares the capabilities, subscriptions, and -operator-supplied config. Drop this next to `Cargo.toml`: +`module.toml` declares capabilities, subscriptions, and operator config: ```toml -# modules/examples/stop-loss/module.toml [module] -name = "stop-loss" +name = "price-alert" version = "0.1.0" component = "sha256:0000000000000000000000000000000000000000000000000000000000000000" [capabilities] -required = ["logging", "chain", "local-store", "cow-api"] +required = ["logging", "chain"] optional = [] [capabilities.http] @@ -133,410 +78,157 @@ kind = "block" chain_id = 11155111 # Sepolia [config] -# Chainlink AggregatorV3Interface address (ETH/USD on Sepolia). -oracle_address = "0x694AA1769357215DE4FAC081bf1f309aDC325306" +oracle_address = "0x694AA1769357215DE4FAC081bf1f309aDC325306" # ETH/USD on Sepolia decimals = "8" -# Trigger price in the oracle's native decimal units. Below this, -# we sell. -trigger_price = "2500.00" -# CoW order parameters (signed by the owner off-chain ahead of -# time, then the module submits the pre-signed body on trigger). -owner = "0x70997970C51812dc3A010C7d01b50e0d17dc79C8" -sell_token = "0x6810e776880C02933D47DB1b9fc05908e5386b96" # GNO on Sepolia -buy_token = "0xfff9976782d46cc05630d1f6ebab18b2324d6b14" # WETH on Sepolia -sell_amount_wei = "1000000000000000000" # 1 GNO -buy_amount_wei = "300000000000000000" # 0.3 ETH -valid_to_seconds = "4294967295" # u32::MAX (no expiry) +threshold = "2500.00" +direction = "below" +every_n_blocks = "1" ``` -Three patterns worth noting: - -- **`required` matches the WIT imports the module uses.** The - engine enforces this at instantiation - declaring a capability - the module does not use is fine; missing a capability the module - does use is a hard error. -- **`[capabilities.http].allow` is empty** because stop-loss makes - no outbound HTTP calls. A module that needs them declares the - `http` capability, lists the hosts it may contact in `allow`, - and calls `nexum_sdk::http::fetch` (which wraps the standard - wasi:http interface); a - request to an off-list host fails with the matchable - `FetchError::Denied`. See `modules/examples/http-probe` for a - working example. -- **`[config]` values are stringly-typed in 0.2.** Your `init` - parses them; the M3 SDK's `OnceLock` pattern (see - `price-alert`) is the recommended idiom. - -## 3. Write the strategy (60 minutes) - -The strategy logic splits into two layers: - -- A pure function that takes `&impl Host` and runs the decision - tree. This is what your tests exercise - no `wit-bindgen`, no - `wasmtime`, fast iteration. -- A thin `Guest` impl in `lib.rs` that adapts the wit-bindgen- - generated host imports into a struct implementing - `nexum_sdk::host::Host`. +- `required` lists the host capabilities the module imports. The engine enforces the list at instantiation: missing a used capability is a hard error. +- `[capabilities.http].allow` is empty because `price-alert` makes no outbound HTTP. A module that needs it declares the `http` capability, lists the hosts it may contact, and calls `nexum_sdk::http::fetch`; an off-list host returns the matchable `FetchError::Denied`. See [`modules/examples/http-probe`](../modules/examples/http-probe). +- `[config]` values are strings. `init` parses them into a typed `Settings`. -### 3a. The pure strategy (30 minutes) +## The pure strategy -Sketch in `src/strategy.rs`: +Decision logic lives in `strategy.rs` as a host-generic function. It never names `wit-bindgen` or `wasmtime`, so tests drive it directly: ```rust -use alloy_primitives::{Address, I256}; -use alloy_sol_types::{SolCall, sol}; -use nexum_sdk::chain::{eth_call_params, parse_eth_call_result}; -use nexum_sdk::host::Fault; -use nexum_sdk::prelude::*; -use shepherd_sdk::cow::{CowApiError, CowHost}; -use shepherd_sdk::prelude::*; - -sol! { - interface AggregatorV3 { - function latestRoundData() external view returns ( - uint80, int256 answer, uint256, uint256, uint80 - ); - } -} - -pub struct Settings { - pub oracle_address: Address, - pub trigger_price_scaled: I256, - pub owner: Address, - pub sell_token: Address, - pub buy_token: Address, - pub sell_amount: U256, - pub buy_amount: U256, - pub valid_to: u32, -} +use nexum_sdk::chain::chainlink::read_latest_answer; +use nexum_sdk::host::{ChainHost, Fault, LoggingHost}; -pub fn on_block( +pub fn on_block( host: &H, chain_id: u64, settings: &Settings, + block_number: u64, ) -> Result<(), Fault> { - // 1. Read the oracle. `host.request` returns a ChainError; `?` folds - // it into Fault via `From`. - let call = AggregatorV3::latestRoundDataCall {}; - let params = eth_call_params(&settings.oracle_address, &call.abi_encode()); - let result_json = host.request(chain_id, "eth_call", ¶ms)?; - let Some(bytes) = parse_eth_call_result(&result_json) else { - tracing::warn!("stop-loss: cannot decode oracle result"); - return Ok(()); - }; - let decoded = AggregatorV3::latestRoundDataCall::abi_decode_returns(&bytes) - .map_err(|e| Fault::InvalidInput(format!("oracle decode: {e}")))?; - let price = decoded.answer; - - // 2. Are we above trigger? Stay idle. - if price > settings.trigger_price_scaled { - tracing::info!(price = %price, "stop-loss idle"); + if !block_number.is_multiple_of(settings.every_n_blocks) { return Ok(()); } - - // 3. Dedup: did we already submit? - let dedup_key = format!("submitted:{:#x}", settings.owner); - if host.get(&dedup_key)?.is_some() { - tracing::info!("stop-loss: already submitted, skipping"); - return Ok(()); + let Some(answer) = + read_latest_answer(host, chain_id, settings.oracle_address, "price-alert") + else { + return Ok(()); // read_latest_answer already logged the failure + }; + if classify(answer, settings.threshold_scaled, settings.direction) { + tracing::warn!(answer = %answer, "price-alert: TRIGGERED"); + } else { + tracing::info!(answer = %answer, "price-alert: ok"); } - - // 4. Build the OrderCreation. (See `twap-monitor` for the full - // helper; for tutorial brevity we elide the JSON encoding.) - let body = build_order_body(settings)?; - // A real strategy matches on `CowApiError::Rejected` to classify the - // orderbook's typed rejection; here we fold to a Fault for brevity. - let uid = host.submit_order(chain_id, &body).map_err(|e| match e { - CowApiError::Fault(f) => f, - other => Fault::Internal(other.to_string()), - })?; - - // 5. Persist + log. - host.set(&dedup_key, uid.as_bytes())?; - tracing::warn!(uid = %uid, "stop-loss triggered"); Ok(()) } - -fn build_order_body(_s: &Settings) -> Result, Fault> { - // Cross-reference: `modules/twap-monitor/src/lib.rs::build_order_creation` - // shows the full assembly path using cowprotocol::OrderCreation:: - // from_signed_order_data + serde_json::to_vec. - todo!("see modules/twap-monitor for the canonical assembly") -} ``` The shape to internalise: -- **Every interaction with the world goes through `host`.** No - global wit-bindgen functions in the strategy; everything is a - method on `&impl Host`. -- **The function is pure-ish:** the only effects are through the - host trait. Tests in §3c run this function against `MockHost` - and assert on the side effects (calls + log lines + state writes). -- **Errors propagate but the loop should not abort on transient - failure.** Wrap upstream calls so a single bad event does not - poison the supervisor - see `price-alert`'s warn-and-return - pattern. - -### 3b. The Guest adapter (15 minutes) - -`src/lib.rs` adapts wit-bindgen's free functions into a struct that -implements `Host`. Every module needs the same glue here: a -`WitBindgenHost` adapter, the `Fault` conversions, and the -`Level` <-> wire-enum mapping for logging. Rather than hand-write -it, call the SDK's bind macro. Plain modules use -`nexum_sdk::bind_host_via_wit_bindgen!()`; stop-loss also submits -CoW orders, so it reaches for the CoW-aware variant, -`shepherd_sdk::bind_cow_host_via_wit_bindgen!()`: +- Every interaction with the world goes through `host`, bounded by the traits the module actually imports (`ChainHost + LoggingHost` here, matching the two declared capabilities). +- The function recovers from transient upstream failure by logging and returning `Ok`, so one bad event does not poison the supervisor. + +Config parsing follows the same one-shot style: `parse_config(&[(String, String)]) -> Result`, using the `nexum_sdk::config` helpers (`get_required`, `scale_decimal`). See the full source in `strategy.rs`. + +## The glue + +`lib.rs` declares the handlers and defers all per-cdylib glue to `#[nexum_sdk::module]`: ```rust #![allow(clippy::too_many_arguments)] -wit_bindgen::generate!({ - path: ["../../../wit/nexum-host", "../../../wit/shepherd-cow"], - world: "shepherd:cow/shepherd", - generate_all, -}); - mod strategy; use std::sync::OnceLock; - use nexum::host::types; -// `WitBindgenHost`, the fault and level `From` impls, `HostLogSink`, -// and `install_tracing` are generated below. Single source -// of truth in `nexum-sdk` + `shepherd-sdk`. -shepherd_sdk::bind_cow_host_via_wit_bindgen!(); - static SETTINGS: OnceLock = OnceLock::new(); -struct StopLoss; +struct PriceAlert; -impl Guest for StopLoss { +#[nexum_sdk::module] +impl PriceAlert { fn init(config: Vec<(String, String)>) -> Result<(), Fault> { install_tracing(); let cfg = strategy::parse_config(&config)?; - tracing::info!( - "stop-loss init: owner={:#x} trigger={} sell={:#x} buy={:#x}", - cfg.owner, - cfg.trigger_price_scaled, - cfg.sell_token, - cfg.buy_token, - ); let _ = SETTINGS.set(cfg); Ok(()) } - fn on_event(event: types::Event) -> Result<(), Fault> { - let Some(cfg) = SETTINGS.get() else { - return Ok(()); - }; - if let types::Event::Block(block) = event { - strategy::on_block(&WitBindgenHost, block.chain_id, cfg)?; - } - Ok(()) + fn on_block(block: types::Block) -> Result<(), Fault> { + let Some(cfg) = SETTINGS.get() else { return Ok(()) }; + strategy::on_block(&WitBindgenHost, block.chain_id, cfg, block.number) + .map_err(Into::into) } } - -export!(StopLoss); ``` -The macro generates `WitBindgenHost`, the `ChainHost` / -`LocalStoreHost` / `LoggingHost` / `CowApiHost` impls, the -`Fault` `From` impls (both directions), and -`install_tracing`, which installs the guest `tracing` facade so -the `tracing::info!`, `warn!`, and `error!` macros reach the host -log call with no `Host` value to thread through. Call it once at -the top of `Guest::init`. Only the `Guest` impl and `SETTINGS` -initialisation above are per-module code. - -Once `install_tracing()` has run, those macros reach the host from -anywhere with no `Host` value to thread through, -so both `init` and the strategy log through the macros. Prefer them: -they take structured fields (`tracing::warn!(code = err.code, "...")`) -that the host records as `key=value` pairs, rather than a -pre-formatted string. The wire-level `logging::log(...)` and the -`host.log(Level::INFO, ...)` trait method still exist for the rare -call site that runs before the subscriber is installed, but every -shipped module logs through the facade. See -`modules/examples/balance-tracker` for a strategy written against -the macros throughout. - -### 3c. Unit tests against `MockHost` (15 minutes) - -In `src/strategy.rs`, append: +Apply the attribute to an inherent `impl` whose functions are the named handlers (`init`, `on_block`, `on_chain_logs`, `on_tick`, `on_message`; absent handlers no-op). The macro generates the `wit_bindgen::generate!` call, the `WitBindgenHost` adapter, the `Fault` conversions, `install_tracing`, and the `Guest`/`export!` glue. Call `install_tracing()` once in `init`; after that the `tracing` macros reach the host log from anywhere with no `Host` value to thread through. + +## Tests against `MockHost` + +Because the strategy is host-generic, tests run in plain Rust with no wasm toolchain, driving it against `nexum_sdk_test::MockHost` and capturing the log output: ```rust #[cfg(test)] mod tests { use super::*; - use nexum_sdk::host::*; - use nexum_sdk_test::capture_tracing; - use shepherd_sdk_test::MockHost; - - fn settings(trigger_scaled: i64) -> Settings { - Settings { - oracle_address: "0x694AA1769357215DE4FAC081bf1f309aDC325306".parse().unwrap(), - trigger_price_scaled: I256::try_from(trigger_scaled).unwrap(), - owner: "0x70997970C51812dc3A010C7d01b50e0d17dc79C8".parse().unwrap(), - sell_token: Address::ZERO, - buy_token: Address::ZERO, - sell_amount: U256::ZERO, - buy_amount: U256::ZERO, - valid_to: 0xffff_ffff, - } - } - - /// Encode a Chainlink `latestRoundData` return for tests. - fn oracle_returns(answer: i64) -> String { - let returns = AggregatorV3::latestRoundDataCall::abi_encode_returns(&( - 0u128, - I256::try_from(answer).unwrap(), - U256::ZERO, - U256::ZERO, - 0u128, - )); - let hex = alloy_primitives::hex::encode_prefixed(returns); - format!("\"{hex}\"") - } + use nexum_sdk_test::{MockHost, capture_tracing}; #[test] - fn idle_when_price_above_trigger() { + fn triggers_below_threshold() { let host = MockHost::new(); - let s = settings(/*trigger*/ 1_000); - // Oracle returns 2000 (above the 1000 trigger). - host.chain.respond_to( - "eth_call", - &nexum_sdk::chain::eth_call_params( - &s.oracle_address, - &AggregatorV3::latestRoundDataCall {}.abi_encode(), - ), - Ok(oracle_returns(2000)), - ); - - let (result, logs) = capture_tracing(|| on_block(&host, 11_155_111, &s)); + let settings = sample_settings(250_050_000_000, Direction::Below); + programmed_eth_call(&host, settings.oracle_address, Ok(oracle_response_json(200_000_000_000))); + + let (result, logs) = capture_tracing(|| on_block(&host, 11_155_111, &settings, 100)); result.unwrap(); - assert_eq!(host.cow_api.call_count(), 0); - assert!(logs.any(|e| e.message.contains("stop-loss idle"))); - } - - #[test] - fn triggers_below_threshold_once() { - let host = MockHost::new(); - let s = settings(/*trigger*/ 1_000); - host.chain.respond_to( - "eth_call", - &nexum_sdk::chain::eth_call_params( - &s.oracle_address, - &AggregatorV3::latestRoundDataCall {}.abi_encode(), - ), - Ok(oracle_returns(500)), - ); - host.cow_api.respond(Ok("0xdeadbeef".into())); - - // First block: submits. - let (first, first_logs) = capture_tracing(|| on_block(&host, 11_155_111, &s)); - first.unwrap(); - assert_eq!(host.cow_api.call_count(), 1); - assert!(first_logs.any(|e| e.message.contains("triggered"))); - - // Second block at the same price: dedup'd by the - // `submitted:` key. - let (second, second_logs) = capture_tracing(|| on_block(&host, 11_155_111, &s)); - second.unwrap(); - assert_eq!(host.cow_api.call_count(), 1); - assert!(second_logs.any(|e| e.message.contains("already submitted"))); + let ev = logs.expect_one(|e| e.level == Level::WARN); + assert_eq!(ev.message, "price-alert: TRIGGERED"); } } ``` -Run with `cargo test -p stop-loss`. Both tests should pass on a -plain host - no wasm toolchain involved. +Run with `cargo test -p price-alert`. See `strategy.rs` for the full test module, including the `MockHost` chain programming (`host.chain.respond_to`) and the throttle and error-path cases. + +Any behaviour expressible as "given this host state, do that" belongs here, not in the engine harness. See [testing-runtime-harness.md](testing-runtime-harness.md) for the guardrail between the two. -The takeaway: any time you can express a behaviour as "given this -host state, do that", the `MockHost` route is faster to iterate -than a full engine restart. +## Build and run -## 4. Build the `.wasm` artefact (5 minutes) +Build the artefact: ```sh -cargo build --target wasm32-wasip2 --release -p stop-loss -ls -lh target/wasm32-wasip2/release/stop_loss.wasm +cargo build --target wasm32-wasip2 --release -p price-alert +ls -lh target/wasm32-wasip2/release/price_alert.wasm ``` -Expected size: 250–350 KB. If it ballooned past ~500 KB, look at -`cargo tree -p stop-loss --target wasm32-wasip2` - usually a fresh -dependency pulled `reqwest` or `tokio` into the wasm graph. - -## 5. Wire `engine.toml` and run it (10 minutes) - -Add an RPC endpoint for Sepolia in `engine.toml`: +Block subscriptions ride `eth_subscribe`, so the chain needs a WebSocket endpoint. Point `engine.toml` at one: ```toml [chains.11155111] rpc_url = "wss://ethereum-sepolia-rpc.publicnode.com" ``` -WebSocket is required because the `[[subscription]]` is `kind = -"block"` and block subscriptions ride `eth_subscribe`. - -Run the engine pointed at your new module: +Run the engine over the module, supplying the chain config: ```sh cargo run -p nexum-cli -- \ - target/wasm32-wasip2/release/stop_loss.wasm \ - modules/examples/stop-loss/module.toml + target/wasm32-wasip2/release/price_alert.wasm \ + modules/examples/price-alert/module.toml \ + --engine-config engine.toml ``` -Expected output on first run (one log per: - -- `init`: `stop-loss: init ok` -- on each new block: either `stop-loss idle` (price above trigger) - or `stop-loss triggered, uid=0x...` then `already submitted` - on subsequent blocks. - -If the engine reports `unsupported` for any capability, double- -check that the module's `[capabilities].required` list matches the -imports the strategy actually uses. - -## 6. Where to go from here (10 minutes) - -- **Production hardening**: replace the synthetic `init` with the - per-module fuel + memory limits in `engine.toml::[engine.limits]` - (see [`docs/deployment.md`](./deployment.md)). -- **Real order assembly**: the `build_order_body` `todo!` in §3a - is the only piece this tutorial elided. Cross-reference - [`modules/twap-monitor/src/lib.rs::build_order_creation`] - - it's the canonical assembly path - (`cowprotocol::OrderCreation::from_signed_order_data` + - `serde_json::to_vec`). -- **Tests for the adapter layer**: the wit-bindgen ↔ `Host` - conversion functions are mechanical but worth a smoke test that - forces each enum variant through. See `shepherd-sdk-test`'s own - tests for the pattern. -- **Multi-chain operation**: change `[[subscription]].chain_id` and - the `engine.toml::[chains.]` entry. The strategy stays - unchanged because every host call already passes `chain_id` - through. - -## Time-budget check - -If a section ran much longer than the rough estimate above, please -file an issue tagged `docs/tutorial` with the section that dragged. -The target is **<4h cold from a fresh checkout to a successful run -in §5**, and we tighten the prose against feedback. - -## Reference index - -- SDK overview: [`docs/sdk.md`](./sdk.md) -- Deployment runbook: [`docs/deployment.md`](./deployment.md) +You should see the `init` line, then one `price-alert: ok` or `price-alert: TRIGGERED` line per new block. An `unsupported` fault means the module imports a capability its `[capabilities].required` list omits. + +## Where to go from here + +- Venue submission: a module that signs and posts orders to a venue depends on `videre-sdk` and a venue adapter crate (`cow-venue` for CoW), and uses `#[videre_sdk::keeper]` rather than `#[nexum_sdk::module]`. See [`modules/twap-monitor`](../modules/twap-monitor) and [docs/sdk.md](sdk.md). +- Local state: declare `local-store` and persist through `host.set`/`host.get`; see [`modules/examples/balance-tracker`](../modules/examples/balance-tracker). +- Outbound HTTP: [`modules/examples/http-probe`](../modules/examples/http-probe). +- Resource limits: [docs/deployment.md](deployment.md). + +## Reference + +- SDK overview: [docs/sdk.md](sdk.md) +- Deployment: [docs/deployment.md](deployment.md) - ADR-0001 (`engine.toml` vs `module.toml` split) -- ADR-0006 (TWAP / EthFlow as guest modules, no specialised - WIT interfaces) -- ADR-0007 (push protocol primitives to `cow-rs` first) -- Worked examples: [`price-alert`](../modules/examples/price-alert/), - [`balance-tracker`](../modules/examples/balance-tracker/), - [`twap-monitor`](../modules/twap-monitor/), - [`ethflow-watcher`](../modules/ethflow-watcher/) +- ADR-0006 (TWAP and EthFlow as guest modules, no specialised WIT interfaces) +- Worked examples: [`price-alert`](../modules/examples/price-alert/), [`balance-tracker`](../modules/examples/balance-tracker/), [`twap-monitor`](../modules/twap-monitor/), [`ethflow-watcher`](../modules/ethflow-watcher/) diff --git a/scripts/README.md b/scripts/README.md index 4d4fad15..25b2e858 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,10 +1,8 @@ -# scripts/ — E2E automation +# scripts/: E2E automation -Three-step automation for the E2E run on Sepolia. Wraps -the runbook (`docs/operations/e2e-testnet-runbook.md`) + the prep -punch list (`docs/operations/e2e-prep.md`) into shell -scripts so the operator only has to (a) fill in `.env` and -(b) decide when to stop. +Three-step automation for the E2E run on Sepolia. Wraps the runbook +(`docs/operations/e2e-testnet-runbook.md`) into shell scripts so the +operator only has to (a) fill in `.env` and (b) decide when to stop. ## One-time setup @@ -13,7 +11,7 @@ cp scripts/env-template scripts/.env $EDITOR scripts/.env # fill in RPC URLs + EOA private key ``` -`.env` is gitignored — secrets stay on disk, never enter chat, +`.env` is gitignored: secrets stay on disk, never enter chat, never get committed. Required external tools: @@ -66,12 +64,12 @@ Pre-flight: - Asserts EOA balance ≥ 0.02 ETH. Required actions: -1. **TWAP** — `cast send ComposableCoW.create((handler,salt,staticInput),true)` +1. **TWAP**: `cast send ComposableCoW.create((handler,salt,staticInput),true)` with calldata derived freshly per invocation by `scripts/_twap_calldata.py` (sets `t0 = now - 60` so part 0 is Ready immediately; hardcoding `t0 = 0` is the prior bug). Fires `ConditionalOrderCreated` → twap-monitor logs `watch:`. -2. **EthFlow** — calls `scripts/_ethflow_quote.py` to hit cow.fi +2. **EthFlow**: calls `scripts/_ethflow_quote.py` to hit cow.fi `/api/v1/quote`, encodes the returned `EthFlowOrder.Data`, then `cast send EthFlow.createOrder` with the right msg.value. Fires `OrderPlacement` → ethflow-watcher logs `submitted:`. @@ -82,12 +80,8 @@ Optional (gated on `RUN_OPTIONAL_PRESIGN=1` in `.env`): 5. `WETH9.approve(GPv2VaultRelayer, 0.005 ETH)`. Each tx hash appended to `scripts/.state` so the report generator -can link them. - -> stop-loss already produces `submitted:{uid}` on the very first -> block (verified in run-prep smoke — the CoW orderbook accepts -> PreSign orders upfront). The optional path is only needed if you -> want the order to actually **settle** on-chain. +can link them. The optional presign path is only needed if you want the +order to actually **settle** on-chain. ### `e2e-finish.sh` @@ -129,7 +123,7 @@ Operator: review + add anomalies (section 6) + sign off ## Re-running cleanly ```bash -scripts/e2e-finish.sh # safe even if it's the only command — graceful exit +scripts/e2e-finish.sh # safe even if it's the only command: graceful exit rm -rf data/e2e # wipe local-store rm scripts/.state # wipe run state scripts/e2e-run.sh # fresh start From 5743c6ec4d34dc98d38c4a655227fd4a145c6dd7 Mon Sep 17 00:00:00 2001 From: mfw78 Date: Sat, 25 Jul 2026 03:30:14 +0000 Subject: [PATCH 2/2] docs: unwrap paragraphs to one logical line for diff-friendliness Hard-wrapped prose churns diffs: a one-word edit reflows the whole paragraph. Join each paragraph onto a single logical line and let it soft-wrap. Content unchanged (word and heading counts preserved); code fences, tables, lists, blockquotes and headings untouched. --- docs/sdk.md | 103 ++++++-------------------------- docs/testing-runtime-harness.md | 74 +++++------------------ scripts/README.md | 44 +++++--------- 3 files changed, 48 insertions(+), 173 deletions(-) diff --git a/docs/sdk.md b/docs/sdk.md index a465dff3..5dc7fa39 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -1,17 +1,8 @@ # The module SDK: nexum-sdk + videre-sdk -`nexum-sdk` is the guest-side library every module consumes: typed -primitives, ABI helpers, an effect-trait seam for testing, the -`#[nexum_sdk::module]` attribute macro and per-module adapter macro, -and a `prelude` that keeps boilerplate out of module crates. -`videre-sdk` layers the venue surface on top: the typed venue client, -the intent-body codec, the `VenueAdapter` seam and the keeper run. -Modules that talk to a venue depend on both crates and import each -directly (nothing is re-exported between them). - -This page is the entry point. The full API reference is the rustdoc -site under `target/doc/nexum_sdk/` and `target/doc/videre_sdk/`, -generated by: +`nexum-sdk` is the guest-side library every module consumes: typed primitives, ABI helpers, an effect-trait seam for testing, the `#[nexum_sdk::module]` attribute macro and per-module adapter macro, and a `prelude` that keeps boilerplate out of module crates. `videre-sdk` layers the venue surface on top: the typed venue client, the intent-body codec, the `VenueAdapter` seam and the keeper run. Modules that talk to a venue depend on both crates and import each directly (nothing is re-exported between them). + +This page is the entry point. The full API reference is the rustdoc site under `target/doc/nexum_sdk/` and `target/doc/videre_sdk/`, generated by: ```sh RUSTDOCFLAGS="-D warnings -D missing-docs" cargo doc -p nexum-sdk -p videre-sdk -p nexum-module-macros -p videre-macros --no-deps --open @@ -19,27 +10,11 @@ RUSTDOCFLAGS="-D warnings -D missing-docs" cargo doc -p nexum-sdk -p videre-sdk ## Authoring a module -Modules are authored with the `#[nexum_sdk::module]` attribute -(re-exported from `nexum-module-macros`). Apply it to an inherent `impl` -block whose associated functions are the named event handlers - -`init`, `on_block`, `on_chain_logs`, `on_tick`, `on_message` - each -taking its wit-bindgen payload and returning `Result<(), Fault>`. -The macro generates the rest of the per-cdylib glue: the -`wit_bindgen::generate!` call, the `bind_host_via_wit_bindgen!()` -adapter, a `Guest` impl whose `on_event` dispatches to whichever -handlers are present (absent handlers no-op), and `export!`. See -[doc 05](05-sdk-design.md#the-nexum_sdkmodule-macro) for a worked -example and the `nexum-module-macros` rustdoc for the fine print. +Modules are authored with the `#[nexum_sdk::module]` attribute (re-exported from `nexum-module-macros`). Apply it to an inherent `impl` block whose associated functions are the named event handlers - `init`, `on_block`, `on_chain_logs`, `on_tick`, `on_message` - each taking its wit-bindgen payload and returning `Result<(), Fault>`. The macro generates the rest of the per-cdylib glue: the `wit_bindgen::generate!` call, the `bind_host_via_wit_bindgen!()` adapter, a `Guest` impl whose `on_event` dispatches to whichever handlers are present (absent handlers no-op), and `export!`. See [doc 05](05-sdk-design.md#the-nexum_sdkmodule-macro) for a worked example and the `nexum-module-macros` rustdoc for the fine print. ## Supported host capabilities -The SDK is host-neutral - it does not call wit-bindgen-generated -functions directly. Instead, it exposes traits that mirror the -on-the-wire host interfaces, and modules adapt their wit-bindgen -imports to the traits at the cdylib boundary (the -`bind_host_via_wit_bindgen!` macro generates that adapter). The traits -in [`nexum_sdk::host`][host-doc] and the venue seam in -[`videre_sdk::client`][client-doc] are: +The SDK is host-neutral - it does not call wit-bindgen-generated functions directly. Instead, it exposes traits that mirror the on-the-wire host interfaces, and modules adapt their wit-bindgen imports to the traits at the cdylib boundary (the `bind_host_via_wit_bindgen!` macro generates that adapter). The traits in [`nexum_sdk::host`][host-doc] and the venue seam in [`videre_sdk::client`][client-doc] are: | Trait | Mirrors | What it does | |---|---|---| @@ -49,86 +24,44 @@ in [`nexum_sdk::host`][host-doc] and the venue seam in | `Host` | supertrait | Bundles the three core traits; blanket impl | | `VenueTransport` (videre-sdk) | `videre:venue/client@0.1.0` | Quote, submit, observe, status and cancel against an installed venue adapter | -A module declaring `[capabilities].required = ["chain", "local-store", -"client", "logging"]` in its `module.toml` matches the host trait seam -one-for-one. +A module declaring `[capabilities].required = ["chain", "local-store", "client", "logging"]` in its `module.toml` matches the host trait seam one-for-one. -[host-doc]: ../target/doc/nexum_sdk/host/index.html -[client-doc]: ../target/doc/videre_sdk/client/index.html +[host-doc]: ../target/doc/nexum_sdk/host/index.html [client-doc]: ../target/doc/videre_sdk/client/index.html ## Modules - [`nexum_sdk::prelude`](../target/doc/nexum_sdk/prelude/index.html) - bulk re-exports covering the alloy primitives (`Address`, `B256`, - `Bytes`, `U256`, `keccak256`). `videre-sdk` has no prelude; it - re-exports its surface from the crate root. +`Bytes`, `U256`, `keccak256`). `videre-sdk` has no prelude; it re-exports its surface from the crate root. - [`client`](../target/doc/videre_sdk/client/index.html) - the typed - venue seam (in `videre-sdk`): a `Venue` marker drives `VenueClient`, - which encodes through `IntentBody` before the byte-level - `VenueTransport` seam. `keeper::retry_action` folds a `VenueFault` - into a `RetryAction`. +venue seam (in `videre-sdk`): a `Venue` marker drives `VenueClient`, which encodes through `IntentBody` before the byte-level `VenueTransport` seam. `keeper::retry_action` folds a `VenueFault` into a `RetryAction`. - CoW-specific pieces live in their own L3 crates rather than the SDK: - `cow_venue::assembly::gpv2_to_order_data` converts the on-chain - `GPv2OrderData` into the typed `OrderData` the orderbook signs - against, and `composable_cow::{Verdict, LegacyRevertAdapter}` give - typed dispatch over the five `IConditionalOrder` custom errors - (`OrderNotValid`, `PollTryNextBlock`, `PollTryAtBlock`, - `PollTryAtEpoch`, `PollNever`). +`cow_venue::assembly::gpv2_to_order_data` converts the on-chain `GPv2OrderData` into the typed `OrderData` the orderbook signs against, and `composable_cow::{Verdict, LegacyRevertAdapter}` give typed dispatch over the five `IConditionalOrder` custom errors (`OrderNotValid`, `PollTryNextBlock`, `PollTryAtBlock`, `PollTryAtEpoch`, `PollNever`). - [`chain`](../target/doc/nexum_sdk/chain/index.html) - `eth_call` - JSON plumbing (in `nexum-sdk`): +JSON plumbing (in `nexum-sdk`): - `chain::eth_call_params(to, data)` - build the `[{to, data}, "latest"]` params array. - `chain::parse_eth_call_result(json)` - parse the `"0x..."` hex response into bytes. - `chain::chainlink::read_latest_answer` - Chainlink AggregatorV3 reader over the two helpers above. - (The CoW-specific `LegacyRevertAdapter` - the `chain-error` rpc - revert bytes to a typed `Verdict` - lives in `composable-cow`.) +(The CoW-specific `LegacyRevertAdapter` - the `chain-error` rpc revert bytes to a typed `Verdict` - lives in `composable-cow`.) - [`host`](../target/doc/nexum_sdk/host/index.html) - host trait - seam plus the SDK's host-neutral `Fault` vocabulary (same cases - as wit-bindgen's, bridged via one-liner converters per module), - in `nexum-sdk`. `config` and `address` parsing helpers sit - alongside it. +seam plus the SDK's host-neutral `Fault` vocabulary (same cases as wit-bindgen's, bridged via one-liner converters per module), in `nexum-sdk`. `config` and `address` parsing helpers sit alongside it. - [`http`](../target/doc/nexum_sdk/http/index.html) - outbound - HTTP over wasi:http, in `nexum-sdk`. - `http::fetch` performs one synchronous - request from a `wasm32-wasip2` guest; requests and responses are - the standard `http` crate's `Request` / `Response` types, and the - SDK's `Fetch` trait seam, `FetchError` taxonomy, and per-phase - `FetchOptions` timeouts compile on every target for host-free - strategy tests. The module must declare the `http` capability - and list the hosts it may contact in `[capabilities.http].allow` - in its `module.toml`; an off-list host surfaces as the matchable - `FetchError::Denied`, distinct from timeouts and transport - failures. See `modules/examples/http-probe` for a complete - module. +HTTP over wasi:http, in `nexum-sdk`. `http::fetch` performs one synchronous request from a `wasm32-wasip2` guest; requests and responses are the standard `http` crate's `Request` / `Response` types, and the SDK's `Fetch` trait seam, `FetchError` taxonomy, and per-phase `FetchOptions` timeouts compile on every target for host-free strategy tests. The module must declare the `http` capability and list the hosts it may contact in `[capabilities.http].allow` in its `module.toml`; an off-list host surfaces as the matchable `FetchError::Denied`, distinct from timeouts and transport failures. See `modules/examples/http-probe` for a complete module. ## Companions: nexum-sdk-test and videre-test -Add `nexum-sdk-test` as a dev-dep on the module crate to write strategy -tests against in-memory mocks; its `MockHost` covers the chain, local -store and logging seams. `videre-test` is the venue-side kit: codec -vectors and header goldens for conformance runs, plus transport mocks -such as `MockFetch`. See the crate docs -([nexum-sdk-test](../crates/nexum-sdk-test/src/lib.rs), -[videre-test](../crates/videre-test/src/lib.rs)) for the usage -pattern. +Add `nexum-sdk-test` as a dev-dep on the module crate to write strategy tests against in-memory mocks; its `MockHost` covers the chain, local store and logging seams. `videre-test` is the venue-side kit: codec vectors and header goldens for conformance runs, plus transport mocks such as `MockFetch`. See the crate docs ([nexum-sdk-test](../crates/nexum-sdk-test/src/lib.rs), [videre-test](../crates/videre-test/src/lib.rs)) for the usage pattern. ## Versioning -The SDK crates are currently `0.1.0` and live at `crates/nexum-sdk/` -and `crates/videre-sdk/` in the shepherd monorepo. They are not yet -published to crates.io; modules depend on them via workspace paths. - -The `cowprotocol` crate is published to crates.io; the workspace -declares `cowprotocol = "0.2.0"` in `[workspace.dependencies]`, with a -temporary `[patch.crates-io]` git override to `nullislabs/cow-rs` -pending a release that ships the hash-only `OrderCreationAppData` -constructor (see the comment above the patch block in the root -`Cargo.toml`). Module Cargo.toml files that inherit from the workspace -pick it up automatically. +The SDK crates are currently `0.1.0` and live at `crates/nexum-sdk/` and `crates/videre-sdk/` in the shepherd monorepo. They are not yet published to crates.io; modules depend on them via workspace paths. + +The `cowprotocol` crate is published to crates.io; the workspace declares `cowprotocol = "0.2.0"` in `[workspace.dependencies]`, with a temporary `[patch.crates-io]` git override to `nullislabs/cow-rs` pending a release that ships the hash-only `OrderCreationAppData` constructor (see the comment above the patch block in the root `Cargo.toml`). Module Cargo.toml files that inherit from the workspace pick it up automatically. diff --git a/docs/testing-runtime-harness.md b/docs/testing-runtime-harness.md index 73e59913..d825d07a 100644 --- a/docs/testing-runtime-harness.md +++ b/docs/testing-runtime-harness.md @@ -1,54 +1,28 @@ # Testing the Runtime: the engine-side `test-utils` harness -Two separate mock surfaces exist, for testing two separate things: module -logic against a guest-side mock, and engine correctness against a real -compiled component. Read this before writing a runtime test. +Two separate mock surfaces exist, for testing two separate things: module logic against a guest-side mock, and engine correctness against a real compiled component. Read this before writing a runtime test. ## The guardrail - **Module business logic is tested in plain Rust, no wasm.** A module's - decision logic lives in a host-generic `strategy.rs` - (`fn on_block(host: &H, ...)`), and its tests drive it against - `nexum-sdk-test::MockHost`; venue-facing logic adds `videre-test`'s - transport mocks. No wasmtime, no component boundary, no engine crate at - all. This is the dominant pattern across every shipped module - (twap-monitor, ethflow-watcher, price-alert, balance-tracker); see - [docs/sdk.md](sdk.md#companions-nexum-sdk-test-and-videre-test). - **New module-logic tests belong here.** +decision logic lives in a host-generic `strategy.rs` (`fn on_block(host: &H, ...)`), and its tests drive it against `nexum-sdk-test::MockHost`; venue-facing logic adds `videre-test`'s transport mocks. No wasmtime, no component boundary, no engine crate at all. This is the dominant pattern across every shipped module (twap-monitor, ethflow-watcher, price-alert, balance-tracker); see [docs/sdk.md](sdk.md#companions-nexum-sdk-test-and-videre-test). **New module-logic tests belong here.** - **The engine harness (this page) is reserved for engine, host, and - boundary correctness**: supervision (poison, restart, resource traps), - dispatch isolation across chains and modules, fault and log capture, the - WASI clock override a real guest observes, capability wiring, stream - reconnect. These need a real compiled `.wasm` component over async mock - backends and genuinely cannot be faked in-process. +boundary correctness**: supervision (poison, restart, resource traps), dispatch isolation across chains and modules, fault and log capture, the WASI clock override a real guest observes, capability wiring, stream reconnect. These need a real compiled `.wasm` component over async mock backends and genuinely cannot be faked in-process. - **Do not test module business logic through the wasm harness.** If a - test only needs "given input X the strategy does Y", it belongs in - `strategy.rs` against `MockHost`, not in a booted fixture here. A - harness test that could be rewritten as a plain-Rust `MockHost` test - without losing coverage is in the wrong place. +test only needs "given input X the strategy does Y", it belongs in `strategy.rs` against `MockHost`, not in a booted fixture here. A harness test that could be rewritten as a plain-Rust `MockHost` test without losing coverage is in the wrong place. ## What `test-utils` provides -`crates/nexum-runtime`'s `test_utils` module (gated behind the -`test-utils` cargo feature) ships two layers: +`crates/nexum-runtime`'s `test_utils` module (gated behind the `test-utils` cargo feature) ships two layers: - **The bare mock backends** - `MockChainProvider`, `MockStateStore`, and - `MockTypes` implement the engine's component-seam traits with no - network and no disk. `mock_components` / `mock_components_from` bundle - them into a `Components` ready for `Supervisor::boot`. Use these when a - test needs to drive the supervisor directly - multi-module scenarios, - custom extensions, or checking host-interface wiring the harness below - doesn't expose. +`MockTypes` implement the engine's component-seam traits with no network and no disk. `mock_components` / `mock_components_from` bundle them into a `Components` ready for `Supervisor::boot`. Use these when a test needs to drive the supervisor directly - multi-module scenarios, custom extensions, or checking host-interface wiring the harness below doesn't expose. - **`TestRuntime` / `TestRuntimeBuilder`** - a higher-level harness over - the same mocks: launch *one* module through the real public - `RuntimeBuilder` path, inject events, and read back logs and store - writes, with no supervisor ceremony. This is what most engine-level - tests want. +the same mocks: launch *one* module through the real public `RuntimeBuilder` path, inject events, and read back logs and store writes, with no supervisor ceremony. This is what most engine-level tests want. ### Feature gate and the self dev-dependency -`test_utils` only compiles under the `test-utils` feature (it pulls -`tempfile`, needed for manifest staging): +`test_utils` only compiles under the `test-utils` feature (it pulls `tempfile`, needed for manifest staging): ```toml # crates/nexum-runtime/Cargo.toml @@ -62,15 +36,11 @@ test-utils = ["dep:tempfile"] nexum-runtime = { path = ".", features = ["test-utils"] } ``` -A crate outside `nexum-runtime` that wants the harness - an extension -crate testing its own backend against a real supervisor, for instance - -depends on `nexum-runtime` with `features = ["test-utils"]` directly; no -self-dependency trick needed there. +A crate outside `nexum-runtime` that wants the harness - an extension crate testing its own backend against a real supervisor, for instance - depends on `nexum-runtime` with `features = ["test-utils"]` directly; no self-dependency trick needed there. ## Worked example: `TestRuntime` -Build the module fixture once (`cargo build --target wasm32-wasip2 ---release -p example`, or `just build-module`), then: +Build the module fixture once (`cargo build --target wasm32-wasip2 --release -p example`, or `just build-module`), then: ```rust use alloy_rpc_types_eth::Header; @@ -119,28 +89,16 @@ chain_id = 1 Beyond the happy path above: - **Chain-log injection** mirrors blocks: `rt.push_chain_log(log)` where - `log: alloy_rpc_types_eth::Log`. +`log: alloy_rpc_types_eth::Log`. - **Chain-request programming**, for a module that calls `chain::request` - (e.g. an `eth_call` oracle read): `rt.chain().on_method(ChainMethod::EthCall, - r#""0x...""#)` before `launch`, or `on_request` for a full - `(method, params)` match. `ChainMethod` is - `nexum_runtime::host::component::ChainMethod`. +(e.g. an `eth_call` oracle read): `rt.chain().on_method(ChainMethod::EthCall, r#""0x...""#)` before `launch`, or `on_request` for a full `(method, params)` match. `ChainMethod` is `nexum_runtime::host::component::ChainMethod`. - **Error and stream-end injection**, to exercise reconnect and fault - paths: `rt.chain().push_block_err(err)` delivers a transport error to - the open block stream (`err: nexum_runtime::host::provider_pool::ProviderError`); - `rt.chain().close_block_stream()` simulates the upstream ending the - subscription, so the test can assert the event loop's reconnect logic - re-opens it. +paths: `rt.chain().push_block_err(err)` delivers a transport error to the open block stream (`err: nexum_runtime::host::provider_pool::ProviderError`); `rt.chain().close_block_stream()` simulates the upstream ending the subscription, so the test can assert the event loop's reconnect logic re-opens it. - **Store assertions**: `rt.store()` exposes the same `MockStateStore` - the module wrote through `local-store` - read back what a dispatched - event persisted. +the module wrote through `local-store` - read back what a dispatched event persisted. - **Guest time**: `rt.clock()` is the `ManualClock` installed as the - module's WASI clock override; advance it to test time-dependent logic - without a real sleep. +module's WASI clock override; advance it to test time-dependent logic without a real sleep. ## If you landed here looking for module tests -You want `nexum-sdk-test::MockHost` instead (plus `videre-test`'s -transport mocks for venue-facing logic): no wasm build, no engine crate, -runs in milliseconds. See [docs/sdk.md](sdk.md) and any shipped module's -`strategy.rs` test module for the pattern. +You want `nexum-sdk-test::MockHost` instead (plus `videre-test`'s transport mocks for venue-facing logic): no wasm build, no engine crate, runs in milliseconds. See [docs/sdk.md](sdk.md) and any shipped module's `strategy.rs` test module for the pattern. diff --git a/scripts/README.md b/scripts/README.md index 25b2e858..261158b5 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,8 +1,6 @@ # scripts/: E2E automation -Three-step automation for the E2E run on Sepolia. Wraps the runbook -(`docs/operations/e2e-testnet-runbook.md`) into shell scripts so the -operator only has to (a) fill in `.env` and (b) decide when to stop. +Three-step automation for the E2E run on Sepolia. Wraps the runbook (`docs/operations/e2e-testnet-runbook.md`) into shell scripts so the operator only has to (a) fill in `.env` and (b) decide when to stop. ## One-time setup @@ -11,13 +9,12 @@ cp scripts/env-template scripts/.env $EDITOR scripts/.env # fill in RPC URLs + EOA private key ``` -`.env` is gitignored: secrets stay on disk, never enter chat, -never get committed. +`.env` is gitignored: secrets stay on disk, never enter chat, never get committed. Required external tools: - `cargo` + the `wasm32-wasip2` target (already there if you've - built the workspace before). +built the workspace before). - `cast` from foundry (`curl -L https://foundry.paradigm.xyz | bash && foundryup`). - `jq`, `curl`, `python3` with `pip3 install eth-utils eth-abi pycryptodome`. @@ -39,75 +36,62 @@ Three artefacts land in `docs/operations/e2e-reports/`: | `metrics-end-.txt` | `/metrics` snapshot at SIGINT. | | `e2e-report-.md` | Auto-filled E2E report. Operator reviews + signs off + commits. | -The first three are gitignored; the report is committed manually -once you've reviewed it. +The first three are gitignored; the report is committed manually once you've reviewed it. ## Script details ### `e2e-run.sh` - Renders `engine.e2e.toml` → `engine.e2e.local.toml` - (gitignored via `*.local.toml`) with `RPC_URL_SEPOLIA` - substituted in. Embedded URL key never reaches git. +(gitignored via `*.local.toml`) with `RPC_URL_SEPOLIA` substituted in. Embedded URL key never reaches git. - Cleans `data/e2e/` for a fresh local-store. - Builds 5 modules + engine in `--release`. - Launches via `nohup`; engine survives the parent shell exiting. - Waits ≤ 60 s for `supervisor ready modules=5 chains=1`. - Persists `ENGINE_PID`, `LOG_FILE`, `METRICS_START`, `START_TS`, - `START_ISO` into `scripts/.state` (gitignored). +`START_ISO` into `scripts/.state` (gitignored). ### `e2e-onchain.sh` Pre-flight: - Derives the EOA address from `OPERATOR_PRIVATE_KEY` and asserts - it matches the pinned `0x7bF140727D27ea64b607E042f1225680B40ECa6A`. +it matches the pinned `0x7bF140727D27ea64b607E042f1225680B40ECa6A`. - Asserts EOA balance ≥ 0.02 ETH. Required actions: 1. **TWAP**: `cast send ComposableCoW.create((handler,salt,staticInput),true)` - with calldata derived freshly per invocation by - `scripts/_twap_calldata.py` (sets `t0 = now - 60` so part 0 is - Ready immediately; hardcoding `t0 = 0` is the prior bug). Fires - `ConditionalOrderCreated` → twap-monitor logs `watch:`. +with calldata derived freshly per invocation by `scripts/_twap_calldata.py` (sets `t0 = now - 60` so part 0 is Ready immediately; hardcoding `t0 = 0` is the prior bug). Fires `ConditionalOrderCreated` → twap-monitor logs `watch:`. 2. **EthFlow**: calls `scripts/_ethflow_quote.py` to hit cow.fi - `/api/v1/quote`, encodes the returned `EthFlowOrder.Data`, - then `cast send EthFlow.createOrder` with the right msg.value. - Fires `OrderPlacement` → ethflow-watcher logs `submitted:`. +`/api/v1/quote`, encodes the returned `EthFlowOrder.Data`, then `cast send EthFlow.createOrder` with the right msg.value. Fires `OrderPlacement` → ethflow-watcher logs `submitted:`. Optional (gated on `RUN_OPTIONAL_PRESIGN=1` in `.env`): 3. `WETH9.deposit()` payable 0.01 ETH. 4. `GPv2Settlement.setPreSignature(uid, true)` with the pinned UID. 5. `WETH9.approve(GPv2VaultRelayer, 0.005 ETH)`. -Each tx hash appended to `scripts/.state` so the report generator -can link them. The optional presign path is only needed if you want the -order to actually **settle** on-chain. +Each tx hash appended to `scripts/.state` so the report generator can link them. The optional presign path is only needed if you want the order to actually **settle** on-chain. ### `e2e-finish.sh` - Captures `metrics-end-.txt`. - Sends `SIGINT` to the engine PID. - Waits ≤ 30 s for `graceful shutdown complete` in the log - (graceful-shutdown path). +(graceful-shutdown path). - Escalates to `SIGKILL` if the engine is still alive after 30 s. - Invokes `e2e-report-gen.sh` to write the filled-in report. ### `e2e-report-gen.sh` -Reads `LOG_FILE`, `METRICS_START`, `METRICS_END`, `START_ISO`, -`END_ISO`, and the `TX_*` hashes from `scripts/.state`; computes: +Reads `LOG_FILE`, `METRICS_START`, `METRICS_END`, `START_ISO`, `END_ISO`, and the `TX_*` hashes from `scripts/.state`; computes: - Chain coverage (first/last block from `block_number` log fields). - Per-module first terminal marker timestamp + sample line. - Delta of every `shepherd_*` Prometheus counter / histogram. - ERROR + trapped + poisoned tallies. - Per-row acceptance checklist (auto-checks block delta ≥ 1500, - marker per module, zero traps, zero poisons, zero ERRORs, - TWAP+EthFlow tx hashes present). +marker per module, zero traps, zero poisons, zero ERRORs, TWAP+EthFlow tx hashes present). -Writes `e2e-report-.md` in `docs/operations/e2e-reports/`. -Operator: review + add anomalies (section 6) + sign off -(section 8) + commit with `git add -f`. +Writes `e2e-report-.md` in `docs/operations/e2e-reports/`. Operator: review + add anomalies (section 6) + sign off (section 8) + commit with `git add -f`. ## Troubleshooting