fix(supervisor): mark module alive=false when init returns Err (COW-1070) - #34
Closed
brunota20 wants to merge 3 commits into
Closed
fix(supervisor): mark module alive=false when init returns Err (COW-1070)#34brunota20 wants to merge 3 commits into
brunota20 wants to merge 3 commits into
Conversation
…070)
Pre-fix behaviour: `Supervisor::load` pushed every module into
`self.modules` with `alive = true`, even when `Guest::init` returned
`Err(HostError)`. The supervisor logged `WARN init failed` but the
dispatcher still routed every block / log to the dead module, where
the M3 example strategies short-circuited via
`SETTINGS.get().is_none() -> return Ok(())`. Safe but wasteful, and
the `supervisor up count=N` log was misleading (counted the dead
module as up).
Surfaced live on Sepolia by scenario 1.4 of
`docs/operations/m3-edge-case-validation.md`: set
`[config] threshold = "not-a-number"` in price-alert, observe init
return InvalidInput, then watch the dispatcher hammer the dead
module every block for 14s.
## Fix
`Supervisor::load` now captures the init result into
`init_succeeded: bool` and sets `LoadedModule.alive = init_succeeded`.
The boot log changes from `supervisor up count=N` to
`supervisor up loaded=N alive=M` so the discrepancy is loud.
## Regression test
`supervisor::tests::init_failure_marks_module_dead_and_excludes_from_dispatch`:
- Synthesises a manifest matching real price-alert shape but with
`threshold = "not-a-number"`.
- Boots the supervisor; asserts `module_count() == 1` (loaded) and
`alive_count() == 0` (dead).
- Dispatches a synthetic Sepolia block; asserts `dispatched == 0`
(the only "subscribed" module is dead, so the dispatch fast-path
skips it).
## Live validation on Sepolia (rerun of scenario 1.4 with fix)
Before fix:
```
INFO supervisor up count=3 <-- includes dead module
```
After fix:
```
WARN init failed - module loaded but marked dead; dispatcher will skip it
module=price-alert kind=HostErrorKind::InvalidInput
INFO supervisor up loaded=3 alive=2
```
## Docs update
`docs/operations/m3-edge-case-validation.md` scenario 1.4 verdict
updated from "✅ with minor observation" to
"✅; resolved in this PR series". The original observation block
is replaced with a note pointing at the regression test + the new
log line.
## Workspace state
- `cargo test --workspace` -> 151 host tests + 6 doctests passing
(was 150 + 6; +1 from the new regression test).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- 0 em-dashes in changed files.
Linear: COW-1070. Closes the only finding from PR #33.
## Considered alternatives
**Option B** (skip pushing the init-failed module into
`self.modules` entirely) would have been cleaner but requires
callers of `Supervisor::load_one` to handle the "module not added"
case. Option A (this PR - flip alive=false) preserves the existing
API surface; the dispatch fast-path already gates on `if !alive
{ continue; }` so the dispatched-event count drops to 0 without any
caller-side change.
**Option C** (visibility only - rename the boot log) was rejected;
it surfaces the discrepancy but does nothing about the per-block
no-op fuel cost on the dead module.
5 tasks
12 review threads addressed end-to-end. Net diff is -720 lines despite adding ~200 lines of new helpers + tests, because the WitBindgenHost adapter deduplication alone wipes ~400 lines. Per-thread: #1 (balance-tracker architecture): refactored to match the M3 host-trait+adapter split the other 4 modules use. Created `strategy.rs` with `on_block(&impl Host, ...)`, moved check_one / fetch_balance / parse_balance_hex / parse_settings into it, converted parse_config to use SDK config helpers + typed HostError instead of String. Added 3 MockHost-driven tests covering first-seen-above-threshold, below-threshold-persist, and error-does-not-abort-loop. #2 + #3 (WitBindgenHost dedup): new `shepherd_sdk::bind_host_via_wit_bindgen!()` declarative macro. Single source of truth in `crates/shepherd-sdk/src/wit_bindgen_macro.rs`; the 4 trait impls + convert_err / sdk_err_into_wit / convert_level collapse to one macro invocation per module. Migrated all 5 modules (twap-monitor, ethflow-watcher, price-alert, stop-loss, balance-tracker). Each module's lib.rs lost ~80 lines. #4 (scale_decimal + config_get dup): new `shepherd_sdk::config` with `get_required`, `get_optional`, `scale_decimal`, and a typed `ConfigError` enum (host-neutral). price-alert + stop-loss consume the SDK helpers; their local duplicates were deleted. Module-level decimal-parsing tests removed (covered by 7 SDK tests + 4 proptest cases now). #5 (Chainlink dup): new `shepherd_sdk::chain::chainlink` with `read_latest_answer(host, chain_id, oracle, domain) -> Option<I256>`. Encapsulates the eth_call → parse → ABI decode flow + Warn logging. price-alert + stop-loss now call the helper; their local AggregatorV3 sol! definitions + read_oracle / on_block oracle plumbing was deleted. SDK ships with 3 StubHost tests covering happy path, host error, and garbage-hex. #6 (WIT world capability elision): added new "Capability enforcement vs. the WIT world" section to ADR-0009 documenting that price-alert + balance-tracker compile against the shepherd:cow/shepherd supertype but their manifests omit cow-api, and that boot success depends on wasm-tools' unused- import elision. Flagged as load-bearing; M5 macro hardening path documented. #7 (poll-time revert classification inert): filed COW-1082 for the host-side fix (forward structured eth_call error data into HostError.data; analogous to COW-1075 for orderbook). #8 (classify_api_error retry-default unbounded): filed COW-1083 for the rate-limit / max-retry follow-up on the backoff: marker. #9 (RetryAction::Backoff dead variant): no code change; replied to thread clarifying it is reserved API surface waiting on a richer upstream retry_hint shape (open question for mfw78). #10 (no proptest anywhere): added `proptest` to shepherd-sdk dev-dependencies. New `crates/shepherd-sdk/src/proptests.rs` with 6 properties covering eth_call_params/parse_eth_call_result round-trip, parse_eth_call_result rejection on unquoted input, config::scale_decimal round-trip + sign-preservation, U256 LE byte round-trip, and no-panic guards for decode_revert_hex + gpv2_to_order_data marker dispatch. #11 (ethflow chain capability least-privilege): moved `chain` from required to optional in `modules/ethflow-watcher/module.toml`, mirroring the M2 mirror fix already applied. #12 (ADR-0009 test-count census): dropped the "145 host tests (twap 20, ethflow 12, ...)" breakdown; kept the qualitative claim. CI is now the authoritative count. Drive-by: alloy-sol-types moved from regular to dev-dependencies in price-alert and stop-loss now that the Chainlink ABI helper is inside shepherd-sdk and the modules only use sol! in their test helpers. Validation: - cargo test --workspace: every crate green; 5 modules + SDK + sdk-test + engine all pass. 8 host tests gained on balance-tracker; 6 proptest props gained on shepherd-sdk; 3 Chainlink helper tests gained. - cargo clippy --workspace --all-targets -- -D warnings: clean. - cargo fmt --check: clean. - cargo build --target wasm32-wasip2 --release for all 5 modules: clean. - Zero em-dashes in source code added. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
…ance) Filtered subset of the compliance applied in PRs #66/#67 of bleu/nullis-shepherd, restricted to files that exist on the M3 epic head. M4/M5-only files (shepherd-backtest, baseline-latency tools, etc.) are skipped, and compliance hunks that depended on M4-introduced types/functions (reconnect tasks, JoinSet plumbing in event_loop, the M4-shape `ProviderError`/Rpc variant, the supervisor restart loop, env-var substitution in engine.toml) are skipped too - they only make sense once the underlying M4 code lands. Brings the M3 epic in line with the repo-wide rust rubric in the cases that do transfer cleanly: - crates/nexum-engine/src/manifest/error.rs: swap manual Display/Error impls for `thiserror::Error` derives, mark `ParseError` `#[non_exhaustive]`, carry source via `#[from]`. - crates/nexum-engine/src/manifest/load.rs: drop the `.map_err(ParseError::Io/Toml)` call sites that the `#[from]` impls now cover, swap `eprintln!` lines for structured `tracing::{info,warn}`. - crates/nexum-engine/src/manifest/mod.rs: doc tidy-up. - crates/nexum-engine/src/host/mod.rs: tighten submodule visibility from `pub` to `pub(crate)` (no out-of-crate users). - crates/nexum-engine/src/host/error.rs + host/impls/cow_api.rs: drop the bespoke `hex_encode` helper in favour of `alloy_primitives::hex::encode_prefixed`, already a dep on M3 and used elsewhere in `shepherd-sdk`. - crates/nexum-engine/src/engine_config.rs: introduce a trimmed `EngineConfigError` (Io + Toml only - the M5 `Substitute` variant covers an env-var-substitution path that does not exist on M3) and return it from `load_or_default` instead of `anyhow::Result`. `main.rs`'s `?` still works thanks to `From<EngineConfigError> for anyhow::Error`. - crates/shepherd-sdk/src/cow/error.rs: mark `RetryAction` `#[non_exhaustive]`. - modules/{twap-monitor,examples/stop-loss,ethflow-watcher}/src/strategy.rs: add a default arm to each `match RetryAction { ... }` that now needs one, treating unknown future variants conservatively (retry on next block / leave watch in place) instead of silently dropping the watch on an SDK bump. cargo fmt + cargo clippy --all-targets -D warnings + cargo test --workspace --all-features all green on the worktree. AI Assistance: Claude Code (Opus 4.7) prepared and applied this filtered patch under direct human review. The instruction \`STOP and report\` on any non-trivial deviation was followed for each hunk; the skipped hunks (M4-coupled) are documented in the propagation summary that accompanies this branch update. A human (Bruno) is accountable for the result.
Author
|
Work landed via |
This was referenced Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When `Supervisor::load` boots a module and `Guest::init` returns `Err(HostError)`, the supervisor now marks the module `alive = false` (instead of `alive = true`). The dispatcher's fast-path `if !alive { continue; }` already excludes dead modules, so the init-failed module no longer receives per-block dispatches.
Boot log changes from `supervisor up count=N` to `supervisor up loaded=N alive=M` so the discrepancy is visible to the operator.
Why
Surfaced live on Sepolia by scenario 1.4 of `docs/operations/m3-edge-case-validation.md` (PR #33). Pre-fix: init-failed module stayed `alive = true`, received every block dispatch, and consumed fuel + RPC subscription cost on a no-op. Safe because all M3 example strategies guard with `SETTINGS.get().is_none() -> return Ok(())`, but future modules without that guard could trap.
Changes
Live validation on Sepolia (re-run of scenario 1.4)
Before fix:
```
INFO supervisor up count=3 <-- includes dead module
```
After fix:
```
WARN init failed - module loaded but marked dead; dispatcher will skip it
module=price-alert kind=HostErrorKind::InvalidInput
INFO supervisor up loaded=3 alive=2
```
Subsequent block dispatches reach only the 2 alive modules. Confirmed over 14s of block flow.
Considered alternatives
Chosen Option A (flip alive=false) because the dispatch fast-path already gates on alive; no caller-side change needed.
Breaking changes
The boot log key changed from `count` to `loaded`+`alive`. Anyone scraping engine logs by exact key match would need to update, but that is operator territory and the new shape is strictly more informative.
Testing
AI assistance disclosure
AI Assistance: this fix + description was produced by a Claude Code agent (Claude Opus 4.7 1M context). The agent surfaced the behaviour in scenario 1.4 of PR #33, filed COW-1070, implemented the fix, added the regression test, re-validated against live Sepolia, and updated the runbook. A human (Bruno) reviewed and is accountable for the result.
Linear: COW-1070. Closes the only follow-up from PR #33.
Stacks on #33 (M3 edge-case validation) -> #32 (M3 runbook + event_loop fix) -> #31 (M2 runbook) -> #30 (COW-1068) -> #29 (COW-1067) -> #28 (COW-1069) -> #27 (COW-1066) -> #26 (COW-1063 QA cleanup).