docs(m3): testnet edge-case validation - 5 scenarios run, all pass - #33
Closed
brunota20 wants to merge 1 commit into
Closed
docs(m3): testnet edge-case validation - 5 scenarios run, all pass#33brunota20 wants to merge 1 commit into
brunota20 wants to merge 1 commit into
Conversation
…pass Closes the gap "M3 happy path is validated on testnet but error paths are not". Five mutations of `engine.m3.toml` / `module.toml::[config]` run against the live `just run-m3` boot; each captured observed output + verdict. ## Scenarios run | # | Mutation | Observed | Verdict | |---|---|---|---| | 1.1 | engine.m3.toml: rpc_url = "wss://nonexistent.example.com" | `Error: connect chain 11155111: IO error: failed to lookup address information` + clean exit | ✅ structured + fail-fast | | 1.2 | price-alert: oracle_address = 0x...01 (EOA, no code) | `WARN price-alert: latestRoundData decode failed: ABI decoding failed: buffer overrun while deserializing` + module alive | ✅ graceful + clear error | | 1.3 | stop-loss: required = ["logging"] (dropped chain/local-store/cow-api) | `Error: load module ... capability violation in stop_loss.wasm component imports cow-api but it is not listed in [capabilities].required` | ✅ security boundary enforced | | 1.4 | price-alert: threshold = "not-a-number" | `WARN init failed module=price-alert kind=HostErrorKind::InvalidInput "threshold: non-digit character in 'not-a-number'"` + other modules unaffected | ✅ with 1 minor observation (see below) | | 1.5 | boot 1 (rm -rf data/m3) -> boot 2 (no rm) | both boots clean, redb file preserved | ✅ cross-restart persistence | ## Surfaced finding Scenario 1.4 caught a minor supervisor behaviour: init-failed modules stay `alive=true` and continue to receive dispatches. Safe in practice because all M3 example modules guard with `SETTINGS.get().is_none() -> return Ok(())`, but wastes fuel + RPC requests per block on a no-op. Filed as a follow-up issue recommending `Supervisor::load` set `alive=false` (or skip the push into `self.modules`) when `Guest::init` returns `Err(HostError)`. ## Validates - Engine error reporting: 5 distinct error paths each surface a typed error with clear domain + message. No silent failures, no panics, no infinite retry loops. - M3 SDK contract: BLEU-814 (32-byte namespace), COW-1025 (capability enforcement), BLEU-851 / -852 / -854 / -855 (typed Settings parsing via HostError) all verified on live Sepolia, not just MockHost. - Operator UX: every misconfiguration scenario produces output an operator can act on without reading source. ## Reproduce Each mutation is one line. `git checkout` to restore between runs. The full diff per scenario is inline in the doc. ## Not in scope (M4 territory) - Fuel exhaustion (COW-1036) - Module trap during on_event + supervisor restart (COW-1033 / COW-1032) - WS reconnect with backoff (current is bail + external restart; flagged in event_loop.rs as "0.3 fix") - State-dump CLI for redb inspection (M4 nice-to-have) ## Follow-up issue Filed separately: "Supervisor::load should mark module alive=false when init returns Err(HostError)". Linear MCP was unavailable at commit time; issue to be filed manually in COW project under M3 milestone.
6 tasks
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…070)
Pre-fix behaviour: `Supervisor::load` pushed every module into
`self.modules` with `alive = true`, even when `Guest::init` returned
`Err(HostError)`. The supervisor logged `WARN init failed` but the
dispatcher still routed every block / log to the dead module, where
the M3 example strategies short-circuited via
`SETTINGS.get().is_none() -> return Ok(())`. Safe but wasteful, and
the `supervisor up count=N` log was misleading (counted the dead
module as up).
Surfaced live on Sepolia by scenario 1.4 of
`docs/operations/m3-edge-case-validation.md`: set
`[config] threshold = "not-a-number"` in price-alert, observe init
return InvalidInput, then watch the dispatcher hammer the dead
module every block for 14s.
## Fix
`Supervisor::load` now captures the init result into
`init_succeeded: bool` and sets `LoadedModule.alive = init_succeeded`.
The boot log changes from `supervisor up count=N` to
`supervisor up loaded=N alive=M` so the discrepancy is loud.
## Regression test
`supervisor::tests::init_failure_marks_module_dead_and_excludes_from_dispatch`:
- Synthesises a manifest matching real price-alert shape but with
`threshold = "not-a-number"`.
- Boots the supervisor; asserts `module_count() == 1` (loaded) and
`alive_count() == 0` (dead).
- Dispatches a synthetic Sepolia block; asserts `dispatched == 0`
(the only "subscribed" module is dead, so the dispatch fast-path
skips it).
## Live validation on Sepolia (rerun of scenario 1.4 with fix)
Before fix:
```
INFO supervisor up count=3 <-- includes dead module
```
After fix:
```
WARN init failed - module loaded but marked dead; dispatcher will skip it
module=price-alert kind=HostErrorKind::InvalidInput
INFO supervisor up loaded=3 alive=2
```
## Docs update
`docs/operations/m3-edge-case-validation.md` scenario 1.4 verdict
updated from "✅ with minor observation" to
"✅; resolved in this PR series". The original observation block
is replaced with a note pointing at the regression test + the new
log line.
## Workspace state
- `cargo test --workspace` -> 151 host tests + 6 doctests passing
(was 150 + 6; +1 from the new regression test).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- 0 em-dashes in changed files.
Linear: COW-1070. Closes the only finding from PR #33.
## Considered alternatives
**Option B** (skip pushing the init-failed module into
`self.modules` entirely) would have been cleaner but requires
callers of `Supervisor::load_one` to handle the "module not added"
case. Option A (this PR - flip alive=false) preserves the existing
API surface; the dispatch fast-path already gates on `if !alive
{ continue; }` so the dispatched-event count drops to 0 without any
caller-side change.
**Option C** (visibility only - rename the boot log) was rejected;
it surfaces the discrepancy but does nothing about the per-block
no-op fuel cost on the dead module.
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…070)
Pre-fix behaviour: `Supervisor::load` pushed every module into
`self.modules` with `alive = true`, even when `Guest::init` returned
`Err(HostError)`. The supervisor logged `WARN init failed` but the
dispatcher still routed every block / log to the dead module, where
the M3 example strategies short-circuited via
`SETTINGS.get().is_none() -> return Ok(())`. Safe but wasteful, and
the `supervisor up count=N` log was misleading (counted the dead
module as up).
Surfaced live on Sepolia by scenario 1.4 of
`docs/operations/m3-edge-case-validation.md`: set
`[config] threshold = "not-a-number"` in price-alert, observe init
return InvalidInput, then watch the dispatcher hammer the dead
module every block for 14s.
## Fix
`Supervisor::load` now captures the init result into
`init_succeeded: bool` and sets `LoadedModule.alive = init_succeeded`.
The boot log changes from `supervisor up count=N` to
`supervisor up loaded=N alive=M` so the discrepancy is loud.
## Regression test
`supervisor::tests::init_failure_marks_module_dead_and_excludes_from_dispatch`:
- Synthesises a manifest matching real price-alert shape but with
`threshold = "not-a-number"`.
- Boots the supervisor; asserts `module_count() == 1` (loaded) and
`alive_count() == 0` (dead).
- Dispatches a synthetic Sepolia block; asserts `dispatched == 0`
(the only "subscribed" module is dead, so the dispatch fast-path
skips it).
## Live validation on Sepolia (rerun of scenario 1.4 with fix)
Before fix:
```
INFO supervisor up count=3 <-- includes dead module
```
After fix:
```
WARN init failed - module loaded but marked dead; dispatcher will skip it
module=price-alert kind=HostErrorKind::InvalidInput
INFO supervisor up loaded=3 alive=2
```
## Docs update
`docs/operations/m3-edge-case-validation.md` scenario 1.4 verdict
updated from "✅ with minor observation" to
"✅; resolved in this PR series". The original observation block
is replaced with a note pointing at the regression test + the new
log line.
## Workspace state
- `cargo test --workspace` -> 151 host tests + 6 doctests passing
(was 150 + 6; +1 from the new regression test).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- 0 em-dashes in changed files.
Linear: COW-1070. Closes the only finding from PR #33.
## Considered alternatives
**Option B** (skip pushing the init-failed module into
`self.modules` entirely) would have been cleaner but requires
callers of `Supervisor::load_one` to handle the "module not added"
case. Option A (this PR - flip alive=false) preserves the existing
API surface; the dispatch fast-path already gates on `if !alive
{ continue; }` so the dispatched-event count drops to 0 without any
caller-side change.
**Option C** (visibility only - rename the boot log) was rejected;
it surfaces the discrepancy but does nothing about the per-block
no-op fuel cost on the dead module.
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…070)
Pre-fix behaviour: `Supervisor::load` pushed every module into
`self.modules` with `alive = true`, even when `Guest::init` returned
`Err(HostError)`. The supervisor logged `WARN init failed` but the
dispatcher still routed every block / log to the dead module, where
the M3 example strategies short-circuited via
`SETTINGS.get().is_none() -> return Ok(())`. Safe but wasteful, and
the `supervisor up count=N` log was misleading (counted the dead
module as up).
Surfaced live on Sepolia by scenario 1.4 of
`docs/operations/m3-edge-case-validation.md`: set
`[config] threshold = "not-a-number"` in price-alert, observe init
return InvalidInput, then watch the dispatcher hammer the dead
module every block for 14s.
## Fix
`Supervisor::load` now captures the init result into
`init_succeeded: bool` and sets `LoadedModule.alive = init_succeeded`.
The boot log changes from `supervisor up count=N` to
`supervisor up loaded=N alive=M` so the discrepancy is loud.
## Regression test
`supervisor::tests::init_failure_marks_module_dead_and_excludes_from_dispatch`:
- Synthesises a manifest matching real price-alert shape but with
`threshold = "not-a-number"`.
- Boots the supervisor; asserts `module_count() == 1` (loaded) and
`alive_count() == 0` (dead).
- Dispatches a synthetic Sepolia block; asserts `dispatched == 0`
(the only "subscribed" module is dead, so the dispatch fast-path
skips it).
## Live validation on Sepolia (rerun of scenario 1.4 with fix)
Before fix:
```
INFO supervisor up count=3 <-- includes dead module
```
After fix:
```
WARN init failed - module loaded but marked dead; dispatcher will skip it
module=price-alert kind=HostErrorKind::InvalidInput
INFO supervisor up loaded=3 alive=2
```
## Docs update
`docs/operations/m3-edge-case-validation.md` scenario 1.4 verdict
updated from "✅ with minor observation" to
"✅; resolved in this PR series". The original observation block
is replaced with a note pointing at the regression test + the new
log line.
## Workspace state
- `cargo test --workspace` -> 151 host tests + 6 doctests passing
(was 150 + 6; +1 from the new regression test).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- 0 em-dashes in changed files.
Linear: COW-1070. Closes the only finding from PR #33.
## Considered alternatives
**Option B** (skip pushing the init-failed module into
`self.modules` entirely) would have been cleaner but requires
callers of `Supervisor::load_one` to handle the "module not added"
case. Option A (this PR - flip alive=false) preserves the existing
API surface; the dispatch fast-path already gates on `if !alive
{ continue; }` so the dispatched-event count drops to 0 without any
caller-side change.
**Option C** (visibility only - rename the boot log) was rejected;
it surfaces the discrepancy but does nothing about the per-block
no-op fuel cost on the dead module.
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…070)
Pre-fix behaviour: `Supervisor::load` pushed every module into
`self.modules` with `alive = true`, even when `Guest::init` returned
`Err(HostError)`. The supervisor logged `WARN init failed` but the
dispatcher still routed every block / log to the dead module, where
the M3 example strategies short-circuited via
`SETTINGS.get().is_none() -> return Ok(())`. Safe but wasteful, and
the `supervisor up count=N` log was misleading (counted the dead
module as up).
Surfaced live on Sepolia by scenario 1.4 of
`docs/operations/m3-edge-case-validation.md`: set
`[config] threshold = "not-a-number"` in price-alert, observe init
return InvalidInput, then watch the dispatcher hammer the dead
module every block for 14s.
## Fix
`Supervisor::load` now captures the init result into
`init_succeeded: bool` and sets `LoadedModule.alive = init_succeeded`.
The boot log changes from `supervisor up count=N` to
`supervisor up loaded=N alive=M` so the discrepancy is loud.
## Regression test
`supervisor::tests::init_failure_marks_module_dead_and_excludes_from_dispatch`:
- Synthesises a manifest matching real price-alert shape but with
`threshold = "not-a-number"`.
- Boots the supervisor; asserts `module_count() == 1` (loaded) and
`alive_count() == 0` (dead).
- Dispatches a synthetic Sepolia block; asserts `dispatched == 0`
(the only "subscribed" module is dead, so the dispatch fast-path
skips it).
## Live validation on Sepolia (rerun of scenario 1.4 with fix)
Before fix:
```
INFO supervisor up count=3 <-- includes dead module
```
After fix:
```
WARN init failed - module loaded but marked dead; dispatcher will skip it
module=price-alert kind=HostErrorKind::InvalidInput
INFO supervisor up loaded=3 alive=2
```
## Docs update
`docs/operations/m3-edge-case-validation.md` scenario 1.4 verdict
updated from "✅ with minor observation" to
"✅; resolved in this PR series". The original observation block
is replaced with a note pointing at the regression test + the new
log line.
## Workspace state
- `cargo test --workspace` -> 151 host tests + 6 doctests passing
(was 150 + 6; +1 from the new regression test).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- 0 em-dashes in changed files.
Linear: COW-1070. Closes the only finding from PR #33.
## Considered alternatives
**Option B** (skip pushing the init-failed module into
`self.modules` entirely) would have been cleaner but requires
callers of `Supervisor::load_one` to handle the "module not added"
case. Option A (this PR - flip alive=false) preserves the existing
API surface; the dispatch fast-path already gates on `if !alive
{ continue; }` so the dispatched-event count drops to 0 without any
caller-side change.
**Option C** (visibility only - rename the boot log) was rejected;
it surfaces the discrepancy but does nothing about the per-block
no-op fuel cost on the dead module.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Documents 5 edge-case scenarios run against live Sepolia via `just run-m3`. Each scenario mutates one config field, captures observed output, and assigns a verdict. All five pass; one minor supervisor behaviour surfaced as a follow-up issue.
Why
The M3 testnet runbook (#32) validated the happy path: boot + 3 strategy paths exercised + retry classification. It did not validate error paths. This PR closes that gap with 5 mutations of `engine.m3.toml` and module `[config]` that exercise the SDK + engine error surface end-to-end on a real chain.
Scenarios
Surfaced finding
Scenario 1.4: init-failed modules stay `alive=true` and continue to receive dispatches. Safe in practice because all M3 example modules guard with `SETTINGS.get().is_none() -> return Ok(())`, but wastes fuel + RPC requests per block on a no-op.
Recommended follow-up issue (Linear MCP was unavailable at commit time, please file manually):
Validates
Breaking changes
None. Doc-only.
Testing
How to reproduce
Each scenario is a one-line config mutation + `just run-m3`. Full diffs inline in the report. Restore between runs:
```bash
git checkout modules/examples/price-alert/module.toml \
modules/examples/stop-loss/module.toml \
engine.m3.toml
```
Not in scope
AI assistance disclosure
AI Assistance: this change + description was produced by a Claude Code agent (Claude Opus 4.7 1M context). The agent ran each of the 5 scenarios against live Sepolia and captured the observed output verbatim. A human (Bruno) reviewed and is accountable for the result.
Stacks on #32 (M3 runbook + event_loop fix) -> #31 (M2 runbook) -> #30 (COW-1068) -> ... -> #26 (COW-1063 QA cleanup).