feat(event-loop+supervisor): graceful shutdown + last-block persistence (COW-1072) - #42
Closed
brunota20 wants to merge 1 commit into
Closed
feat(event-loop+supervisor): graceful shutdown + last-block persistence (COW-1072)#42brunota20 wants to merge 1 commit into
brunota20 wants to merge 1 commit into
Conversation
…ce (COW-1072)
Two coupled changes that make operator-driven shutdowns observable
and recoverable:
## 1. Event loop: dispatch outside `select!`
`run()` previously had its `call_on_event().await` inside the
`tokio::select!`. A shutdown signal arriving mid-dispatch
cancelled the in-flight wasmtime call, leaving the wasm store in
an indeterminate state.
The refactor splits the loop into two phases:
- **Phase 1**: a small `tokio::select!` picks the next event OR
observes shutdown OR reports an upstream-task panic. Each branch
resolves into a `NextEvent` value; the select drops without
cancelling anything *outside* itself.
- **Phase 2**: `match next` dispatches the event to the supervisor
via a fully-awaited call, OR exits cleanly on the shutdown
variant.
The shutdown signal is now only observed *between* dispatches.
In-flight wasmtime calls always finish naturally.
## 2. Per-module last-dispatched-block persistence
Every successful `dispatch_block` writes a host-side marker to the
module's own local-store namespace:
```
namespace = module.name
key = "last_dispatched_block:{chain_id}"
value = block.number.to_le_bytes()
```
The marker survives engine restarts (it lives in the redb file
under `state_dir`). Operators can confirm at-which-block an engine
last ran without trawling the logs; modules that care about block-
gap detection can read it back on their next `init`.
Write failures are best-effort (a `WARN` log; the dispatch is not
considered failed).
## 3. Graceful shutdown log
The event loop now emits a structured exit line:
```
INFO graceful shutdown complete
dispatched_blocks=N dispatched_logs=M uptime_secs=K
```
Visible live on Sepolia after a `kill -TERM`:
```
INFO shutdown signal received signal=SIGTERM
INFO graceful shutdown complete dispatched_blocks=1 dispatched_logs=0 uptime_secs=13
```
## Out of scope
- 30s drain timeout via `tokio::time::timeout`. The current
dispatch path always terminates (fuel cap caps wall time to <1s
in practice); a 30s drain timer is dead code today. Worth
adding once a module ever needs longer-running host calls (HTTP
capability, etc).
- `engine.toml::[engine.shutdown]` config knobs. The internal
default is "wait as long as the in-flight dispatch takes";
configurable in 0.3.
- Module-side `shutdown` hook. Modules just see the dispatch
complete normally; the supervisor exits without invoking
anything new.
## Tests
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(unchanged shape; the dispatch refactor is transparent to the
existing test suite).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: ran the engine, observed the graceful
shutdown log + last_dispatched_block markers in
`data/m3/local-store.redb`.
Linear: COW-1072. Eighth M4 issue landed; stacks on #41 (COW-1032).
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
The supervisor's dispatch path is per-chain by construction (`dispatch_block(block)` filters modules by `block.chain_id` matching their `[[subscription]]` table), and the COW-1071 WS reconnect tasks own one per-chain backoff timer each. Multi-chain isolation is therefore structural, not derived. This PR locks the guarantee into the test suite with two new integration tests + a supervisor.rs docstring stating the invariant explicitly. ## New tests `multi_chain_dispatch_isolates_modules_by_chain`: - Boot two `example` modules with different `[[subscription]]` chain_ids (1 + 100). - Dispatch a block on chain 1 -> only module-a receives it (dispatched=1, alive_count=2 unchanged). - Dispatch a block on chain 100 -> only module-b receives it. - Validates: subscription filter is per-chain; a block on one chain does not even enter modules subscribed to a different chain. `multi_chain_poisoned_module_does_not_affect_other_chains`: - Boot fuel-bomb (always-traps) on chain 1 + example (healthy) on chain 100, with `PoisonPolicy::new(2, 60s)`. - Trap bomb #1 on chain 1 -> bomb dies, poisoned=0, example untouched. - Dispatch on chain 100 -> example receives (1/1). - Wait 1.1 s (bomb backoff window), trap bomb #2 -> poisoned=1. - Dispatch on chain 100 again -> example STILL receives. - Validates: a permanently-poisoned module on one chain does not consume restart slots, fuel, or scheduling attention from modules on any other chain. Total wall-clock ~1.2 s for the second test (one backoff window). ## supervisor.rs docstring The module-level comment now articulates the multi-chain isolation invariant explicitly so a future reader of the dispatch path knows the property is load-bearing. ## What this proves Supervisor side (dispatch fast-path): - Per-module `alive`, `failure_count`, `next_attempt`, `poisoned` are independent of which chain triggered the event. - Subscription filter excludes mismatched modules before any dispatch / restart logic runs. Upstream side (already proven by COW-1071's architecture): - `open_block_streams` spawns one task per chain; tasks share no state. A chain-A WS drop changes only chain-A's task state. - `open_log_streams` is per-(module, chain) -> even tighter isolation than block streams. ## Out of scope - A unit test that "fakes a WS drop" on chain A while chain B keeps yielding. Requires mocking `ProviderPool::subscribe_blocks` which today goes through real alloy / tokio infrastructure. The COW-1064 (E2E 4-6h testnet) and COW-1031 (7-day soak) will exercise this path against live RPCs. - Per-chain configurable backoff / health-window. Today the reconnect policy is workspace-wide; per-chain tuning is a 0.3 follow-up. ## Workspace impact - `cargo test --workspace` -> 163 host tests + 6 doctests passing (was 161 + 6; +2 from the new integration tests). - `cargo clippy --all-targets --workspace -- -D warnings` clean. - `cargo fmt --all --check` clean. Linear: COW-1073. Ninth M4 issue landed; stacks on #42 (COW-1072).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
The supervisor's dispatch path is per-chain by construction (`dispatch_block(block)` filters modules by `block.chain_id` matching their `[[subscription]]` table), and the COW-1071 WS reconnect tasks own one per-chain backoff timer each. Multi-chain isolation is therefore structural, not derived. This PR locks the guarantee into the test suite with two new integration tests + a supervisor.rs docstring stating the invariant explicitly. ## New tests `multi_chain_dispatch_isolates_modules_by_chain`: - Boot two `example` modules with different `[[subscription]]` chain_ids (1 + 100). - Dispatch a block on chain 1 -> only module-a receives it (dispatched=1, alive_count=2 unchanged). - Dispatch a block on chain 100 -> only module-b receives it. - Validates: subscription filter is per-chain; a block on one chain does not even enter modules subscribed to a different chain. `multi_chain_poisoned_module_does_not_affect_other_chains`: - Boot fuel-bomb (always-traps) on chain 1 + example (healthy) on chain 100, with `PoisonPolicy::new(2, 60s)`. - Trap bomb #1 on chain 1 -> bomb dies, poisoned=0, example untouched. - Dispatch on chain 100 -> example receives (1/1). - Wait 1.1 s (bomb backoff window), trap bomb #2 -> poisoned=1. - Dispatch on chain 100 again -> example STILL receives. - Validates: a permanently-poisoned module on one chain does not consume restart slots, fuel, or scheduling attention from modules on any other chain. Total wall-clock ~1.2 s for the second test (one backoff window). ## supervisor.rs docstring The module-level comment now articulates the multi-chain isolation invariant explicitly so a future reader of the dispatch path knows the property is load-bearing. ## What this proves Supervisor side (dispatch fast-path): - Per-module `alive`, `failure_count`, `next_attempt`, `poisoned` are independent of which chain triggered the event. - Subscription filter excludes mismatched modules before any dispatch / restart logic runs. Upstream side (already proven by COW-1071's architecture): - `open_block_streams` spawns one task per chain; tasks share no state. A chain-A WS drop changes only chain-A's task state. - `open_log_streams` is per-(module, chain) -> even tighter isolation than block streams. ## Out of scope - A unit test that "fakes a WS drop" on chain A while chain B keeps yielding. Requires mocking `ProviderPool::subscribe_blocks` which today goes through real alloy / tokio infrastructure. The COW-1064 (E2E 4-6h testnet) and COW-1031 (7-day soak) will exercise this path against live RPCs. - Per-chain configurable backoff / health-window. Today the reconnect policy is workspace-wide; per-chain tuning is a 0.3 follow-up. ## Workspace impact - `cargo test --workspace` -> 163 host tests + 6 doctests passing (was 161 + 6; +2 from the new integration tests). - `cargo clippy --all-targets --workspace -- -D warnings` clean. - `cargo fmt --all --check` clean. Linear: COW-1073. Ninth M4 issue landed; stacks on #42 (COW-1072).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
The supervisor's dispatch path is per-chain by construction (`dispatch_block(block)` filters modules by `block.chain_id` matching their `[[subscription]]` table), and the COW-1071 WS reconnect tasks own one per-chain backoff timer each. Multi-chain isolation is therefore structural, not derived. This PR locks the guarantee into the test suite with two new integration tests + a supervisor.rs docstring stating the invariant explicitly. ## New tests `multi_chain_dispatch_isolates_modules_by_chain`: - Boot two `example` modules with different `[[subscription]]` chain_ids (1 + 100). - Dispatch a block on chain 1 -> only module-a receives it (dispatched=1, alive_count=2 unchanged). - Dispatch a block on chain 100 -> only module-b receives it. - Validates: subscription filter is per-chain; a block on one chain does not even enter modules subscribed to a different chain. `multi_chain_poisoned_module_does_not_affect_other_chains`: - Boot fuel-bomb (always-traps) on chain 1 + example (healthy) on chain 100, with `PoisonPolicy::new(2, 60s)`. - Trap bomb #1 on chain 1 -> bomb dies, poisoned=0, example untouched. - Dispatch on chain 100 -> example receives (1/1). - Wait 1.1 s (bomb backoff window), trap bomb #2 -> poisoned=1. - Dispatch on chain 100 again -> example STILL receives. - Validates: a permanently-poisoned module on one chain does not consume restart slots, fuel, or scheduling attention from modules on any other chain. Total wall-clock ~1.2 s for the second test (one backoff window). ## supervisor.rs docstring The module-level comment now articulates the multi-chain isolation invariant explicitly so a future reader of the dispatch path knows the property is load-bearing. ## What this proves Supervisor side (dispatch fast-path): - Per-module `alive`, `failure_count`, `next_attempt`, `poisoned` are independent of which chain triggered the event. - Subscription filter excludes mismatched modules before any dispatch / restart logic runs. Upstream side (already proven by COW-1071's architecture): - `open_block_streams` spawns one task per chain; tasks share no state. A chain-A WS drop changes only chain-A's task state. - `open_log_streams` is per-(module, chain) -> even tighter isolation than block streams. ## Out of scope - A unit test that "fakes a WS drop" on chain A while chain B keeps yielding. Requires mocking `ProviderPool::subscribe_blocks` which today goes through real alloy / tokio infrastructure. The COW-1064 (E2E 4-6h testnet) and COW-1031 (7-day soak) will exercise this path against live RPCs. - Per-chain configurable backoff / health-window. Today the reconnect policy is workspace-wide; per-chain tuning is a 0.3 follow-up. ## Workspace impact - `cargo test --workspace` -> 163 host tests + 6 doctests passing (was 161 + 6; +2 from the new integration tests). - `cargo clippy --all-targets --workspace -- -D warnings` clean. - `cargo fmt --all --check` clean. Linear: COW-1073. Ninth M4 issue landed; stacks on #42 (COW-1072).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
The supervisor's dispatch path is per-chain by construction (`dispatch_block(block)` filters modules by `block.chain_id` matching their `[[subscription]]` table), and the COW-1071 WS reconnect tasks own one per-chain backoff timer each. Multi-chain isolation is therefore structural, not derived. This PR locks the guarantee into the test suite with two new integration tests + a supervisor.rs docstring stating the invariant explicitly. ## New tests `multi_chain_dispatch_isolates_modules_by_chain`: - Boot two `example` modules with different `[[subscription]]` chain_ids (1 + 100). - Dispatch a block on chain 1 -> only module-a receives it (dispatched=1, alive_count=2 unchanged). - Dispatch a block on chain 100 -> only module-b receives it. - Validates: subscription filter is per-chain; a block on one chain does not even enter modules subscribed to a different chain. `multi_chain_poisoned_module_does_not_affect_other_chains`: - Boot fuel-bomb (always-traps) on chain 1 + example (healthy) on chain 100, with `PoisonPolicy::new(2, 60s)`. - Trap bomb #1 on chain 1 -> bomb dies, poisoned=0, example untouched. - Dispatch on chain 100 -> example receives (1/1). - Wait 1.1 s (bomb backoff window), trap bomb #2 -> poisoned=1. - Dispatch on chain 100 again -> example STILL receives. - Validates: a permanently-poisoned module on one chain does not consume restart slots, fuel, or scheduling attention from modules on any other chain. Total wall-clock ~1.2 s for the second test (one backoff window). ## supervisor.rs docstring The module-level comment now articulates the multi-chain isolation invariant explicitly so a future reader of the dispatch path knows the property is load-bearing. ## What this proves Supervisor side (dispatch fast-path): - Per-module `alive`, `failure_count`, `next_attempt`, `poisoned` are independent of which chain triggered the event. - Subscription filter excludes mismatched modules before any dispatch / restart logic runs. Upstream side (already proven by COW-1071's architecture): - `open_block_streams` spawns one task per chain; tasks share no state. A chain-A WS drop changes only chain-A's task state. - `open_log_streams` is per-(module, chain) -> even tighter isolation than block streams. ## Out of scope - A unit test that "fakes a WS drop" on chain A while chain B keeps yielding. Requires mocking `ProviderPool::subscribe_blocks` which today goes through real alloy / tokio infrastructure. The COW-1064 (E2E 4-6h testnet) and COW-1031 (7-day soak) will exercise this path against live RPCs. - Per-chain configurable backoff / health-window. Today the reconnect policy is workspace-wide; per-chain tuning is a 0.3 follow-up. ## Workspace impact - `cargo test --workspace` -> 163 host tests + 6 doctests passing (was 161 + 6; +2 from the new integration tests). - `cargo clippy --all-targets --workspace -- -D warnings` clean. - `cargo fmt --all --check` clean. Linear: COW-1073. Ninth M4 issue landed; stacks on #42 (COW-1072).
lgahdl
added a commit
that referenced
this pull request
Jul 10, 2026
…abs#303) Closes #42. A stuck alloy `raw_request` call with no deadline would hang the event loop indefinitely. This wraps every `chain::request` dispatch in `tokio::time::timeout`, bounded by a new per-chain `request_timeout_secs` field in `ChainConfig` (default 30 s; `0` is rejected at boot). A lapse surfaces to the guest as the typed `timeout` fault, so a module can tell a slow node apart from a revert or an unreachable endpoint. `eth_subscribe` streams and the canonical log poller are unaffected: the timeout only guards the one-shot RPC calls modules issue via `chain::request`. Co-authored-by: Luiz Gustavo Abou Hatem de Liz <luizgustavoahsc@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Two coupled changes that make operator-driven shutdowns observable + recoverable:
tokio::select!, so a shutdown signal arriving mid-`call_on_event` no longer cancels the in-flight wasmtime call. The shutdown is observed between dispatches.Eighth M4 issue landed.
Live Sepolia validation
`data/m3/local-store.redb` (3.6 MB) holds the per-module markers post-shutdown.
Design — event loop refactor
Before: `call_on_event().await` was inside the `tokio::select!`, so a shutdown signal arriving during a wasmtime call cancelled it.
After: the select picks the next event variant (block / log / shutdown / panic-bail) into a local enum. The dispatch itself happens in a separate `match` outside the select. Shutdown is only observed between iterations of the outer loop.
Out of scope
Tests
AI assistance disclosure
AI Assistance: this change + description was produced by a Claude Code agent (Claude Opus 4.7 1M context). The agent designed the dispatch-outside-select refactor, wired the per-module marker write, validated the exit log live on Sepolia, and authored this PR description. A human (Bruno) reviewed and is accountable for the result.
Linear: COW-1072. Stacks on #41 (COW-1032 poison-pill).