feat(supervisor): exponential-backoff restart with component reinstantiation (COW-1033) - #39
Closed
brunota20 wants to merge 1 commit into
Closed
Conversation
…tiation (COW-1033)
When a module traps in `on_event` (OutOfFuel, MemoryOutOfBounds,
unhandled host error), the supervisor now:
1. Marks the module `alive = false` and increments `failure_count`.
2. Schedules a `next_attempt` instant via the new
`runtime::restart_policy::backoff_for` (1s → 2s → 4s → ... cap
5 min). All dispatches before that instant skip the module.
3. On the first dispatch past the backoff window, the supervisor
tears down the trapped wasmtime Store + component instance and
re-instantiates from the cached `Component`. The instance state
resets but host-side persistent state (local-store) survives
so a module's progress counters live across restarts.
4. On a successful `on_event` after recovery, `failure_count` resets
to 0 + `next_attempt = None`.
## Why the reinstantiation is required
A wasmtime trap leaves the component instance poisoned: subsequent
`call_on_event` returns "wasm trap: cannot enter component instance".
Just refueling the Store does not recover. The supervisor caches
the `Component`, `init_config`, and `http_allowlist` on
`LoadedModule` at boot so a restart only needs a fresh Store +
re-instantiation - the compiled component bytes are reused.
## New types / files
- `crates/nexum-engine/src/runtime/restart_policy.rs`: `backoff_for(failure_count) -> Duration` with the 1s → 5min schedule. 4 unit tests covering the steady-state, first-failure, doubling, and cap arms.
- `Supervisor` gains four cached backends (`engine`, `cow_pool`, `provider_pool`, `local_store`) so `reinstantiate_one(idx)` can rebuild the wasi Linker + HostState + Store + bindings on demand.
- `LoadedModule` gains `component: Component`, `init_config: Config`, `http_allowlist: Vec<String>` (all cloned at boot), plus `failure_count: u32` and `next_attempt: Option<Instant>` for the schedule.
## Dispatch path changes
`dispatch_block` and `dispatch_log` now restructure into two
phases:
1. **Phase 1 (restart sweep)**: walk modules, collect indices of
dead-but-due modules, call `reinstantiate_one` on each. Failed
restarts bump the backoff again. Successful restarts flip
`alive = true` so phase 2 dispatches the next event to them.
2. **Phase 2 (steady-state dispatch)**: unchanged from before -
walk modules, dispatch where subscribed + alive. Trap path
sets `next_attempt` + bumps `failure_count`; success path
resets both.
The structured logs from COW-1035 gain `failure_count` + `backoff_ms`
on trap + `restart attempt` info lines on each restart. The
`shepherd_module_restarts_total{module}` Prometheus counter from
COW-1034 increments on every restart attempt.
## New fixture + integration test
`modules/fixtures/flaky-bomb/` (test-only): traps via OutOfFuel on
the first N events (N from `[config].fail_first_n`) and recovers
afterwards. Uses local-store for the attempt counter because the
wasm instance state resets on each reinstantiation; the counter
persists in the host-side store so the module deterministically
recovers after the configured N.
`supervisor::tests::restart_flaky_module_recovers_after_backoff`
(new): boots flaky-bomb with fail_first_n=1, dispatches, observes:
- Dispatch 1: trap. alive=false, failure_count=1, next_attempt=+1s.
- Immediate redispatch: skipped (still in backoff).
- Sleep 1.1s.
- Dispatch 3: restart fires, fresh instance attempts again. With
attempt=2 > N=1, returns Ok. alive=true, failure_count=0,
next_attempt=None.
- Dispatch 4: steady-state, dispatches normally.
Test wall-clock ~1.4s.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing.
+4 from `restart_policy` unit tests + 1 from the new integration
test (was 154 + 6).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing resource-limit tests (COW-1036) still pass against
the new dispatch shape: their assertions are against state
*immediately* after the trap (before backoff elapses), so the
restart machinery is transparent.
- The `init_failure_marks_module_dead_and_excludes_from_dispatch`
test (COW-1070) still passes: init-failed modules carry
`next_attempt = None` so the restart sweep never picks them up.
## Out of scope
- Persistence of `failure_count` / `next_attempt` across full
engine restarts. The schedule resets on every boot; cross-engine
persistence is a 0.3 follow-up.
- WS reconnect-with-backoff for upstream RPC drops - that is
COW-1071, a separate axis.
- Operator-tunable backoff via `engine.toml::[engine.restart]`.
The current constants are workspace literals in
`runtime::restart_policy`; configurable in 0.3.
- Module-side `on_restart` hook. Modules just see a fresh `init`
call after a restart, same as boot.
Linear: COW-1033. Fifth M4 issue landed; stacks on #38 (COW-1034).
5 tasks
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…OW-1071)
Replaces the previous "bail on WS drop" semantic (flagged as the
"0.3 fix" in the source) with per-stream reconnect-aware tasks. Each
chain's block subscription and each (module, chain) log subscription
gets a dedicated task that:
1. Opens the subscription via `ProviderPool`.
2. Pumps items to an mpsc channel until the underlying stream
yields `None` (WS drop) or `Err` (transport-level error).
3. Logs the drop + sleeps for `restart_policy::backoff_for(attempt)`
(1s -> 2s -> 4s -> ... cap 5 min, reusing the COW-1033 policy).
4. Reopens. The first event after a reopen emits an `INFO ...
reopened` line + increments `shepherd_stream_reconnects_total`.
5. Resets `attempt = 0` once the stream has been healthy for the
`HEALTHY_WINDOW` (60 s of uninterrupted events) so a flaky-but-
then-stable connection reverts to fast retries on the next drop.
The event loop reads the channel as a regular `Stream` (wrapped
with `futures::stream::unfold` to avoid pulling in `tokio-stream`
just for `ReceiverStream`). A bare `None` from the merged stream
now indicates the reconnect task itself exited (panic or channel
closed); that path still bails the engine as before, but the log
message updated to reflect the new semantic.
## Key behavioural change
Public Sepolia (`wss://ethereum-sepolia-rpc.publicnode.com`) drops
WS connections after ~20 min of sustained load. Pre-fix: engine
bailed within seconds of the first drop, an operator restart
re-opened the subscription but the engine had missed every event in
between. Post-fix: the reconnect task waits 1s and reopens; only
events that arrived during the 1s gap are missed. Multi-minute drops
get progressively longer waits, capped at 5 min.
## New metric (consumed via COW-1034)
`shepherd_stream_reconnects_total{kind, chain_id, module}` counter,
incremented on every successful reopen. Operators write SLO alerts
against this for "stream churn" (e.g. > 5 reconnects per 10 min on
the same chain).
## Channel buffer + back-pressure
Buffer is 64 events per task. Real-time dispatch usually drains in
~12 s (Sepolia block time) so the buffer is overkill for normal
operation; it absorbs a brief dispatch-side stall (e.g. a stop-loss
cow-api submit that takes 2 s) without dropping events at the WS
boundary.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing
(unchanged shape - all existing tests still pass, including the
`run_does_not_bail_when_both_stream_kinds_are_empty` regression
guard which verifies the empty-stream path).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia happy path: `just run-m3` boots, all 3 modules
dispatch normally, `subscription open` log line emitted, no
reconnect activity in 60 s window (network was stable). Clean
SIGTERM shutdown.
## Out of scope
- WS endpoint failover (swap Alchemy <-> publicnode on failure).
Operator concern; track via `[engine.chains.<id>]` schema if
demand arises.
- Backfill of events missed during the drop window. Live-stream
semantic only; backfill is an indexer concern outside the M4
engine scope.
- Operator-tunable backoff / healthy-window via `engine.toml`. The
current constants are workspace literals; configurable in 0.3.
- Per-chain isolation across reconnects (COW-1073). The current
patch already gives partial isolation: each chain's task drops
+ reconnects independently and one task's failure does not
starve the others. COW-1073 covers the supervisor-side
multi-chain coordination.
Linear: COW-1071. Sixth M4 issue landed; stacks on #39 (COW-1033).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…OW-1071)
Replaces the previous "bail on WS drop" semantic (flagged as the
"0.3 fix" in the source) with per-stream reconnect-aware tasks. Each
chain's block subscription and each (module, chain) log subscription
gets a dedicated task that:
1. Opens the subscription via `ProviderPool`.
2. Pumps items to an mpsc channel until the underlying stream
yields `None` (WS drop) or `Err` (transport-level error).
3. Logs the drop + sleeps for `restart_policy::backoff_for(attempt)`
(1s -> 2s -> 4s -> ... cap 5 min, reusing the COW-1033 policy).
4. Reopens. The first event after a reopen emits an `INFO ...
reopened` line + increments `shepherd_stream_reconnects_total`.
5. Resets `attempt = 0` once the stream has been healthy for the
`HEALTHY_WINDOW` (60 s of uninterrupted events) so a flaky-but-
then-stable connection reverts to fast retries on the next drop.
The event loop reads the channel as a regular `Stream` (wrapped
with `futures::stream::unfold` to avoid pulling in `tokio-stream`
just for `ReceiverStream`). A bare `None` from the merged stream
now indicates the reconnect task itself exited (panic or channel
closed); that path still bails the engine as before, but the log
message updated to reflect the new semantic.
## Key behavioural change
Public Sepolia (`wss://ethereum-sepolia-rpc.publicnode.com`) drops
WS connections after ~20 min of sustained load. Pre-fix: engine
bailed within seconds of the first drop, an operator restart
re-opened the subscription but the engine had missed every event in
between. Post-fix: the reconnect task waits 1s and reopens; only
events that arrived during the 1s gap are missed. Multi-minute drops
get progressively longer waits, capped at 5 min.
## New metric (consumed via COW-1034)
`shepherd_stream_reconnects_total{kind, chain_id, module}` counter,
incremented on every successful reopen. Operators write SLO alerts
against this for "stream churn" (e.g. > 5 reconnects per 10 min on
the same chain).
## Channel buffer + back-pressure
Buffer is 64 events per task. Real-time dispatch usually drains in
~12 s (Sepolia block time) so the buffer is overkill for normal
operation; it absorbs a brief dispatch-side stall (e.g. a stop-loss
cow-api submit that takes 2 s) without dropping events at the WS
boundary.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing
(unchanged shape - all existing tests still pass, including the
`run_does_not_bail_when_both_stream_kinds_are_empty` regression
guard which verifies the empty-stream path).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia happy path: `just run-m3` boots, all 3 modules
dispatch normally, `subscription open` log line emitted, no
reconnect activity in 60 s window (network was stable). Clean
SIGTERM shutdown.
## Out of scope
- WS endpoint failover (swap Alchemy <-> publicnode on failure).
Operator concern; track via `[engine.chains.<id>]` schema if
demand arises.
- Backfill of events missed during the drop window. Live-stream
semantic only; backfill is an indexer concern outside the M4
engine scope.
- Operator-tunable backoff / healthy-window via `engine.toml`. The
current constants are workspace literals; configurable in 0.3.
- Per-chain isolation across reconnects (COW-1073). The current
patch already gives partial isolation: each chain's task drops
+ reconnects independently and one task's failure does not
starve the others. COW-1073 covers the supervisor-side
multi-chain coordination.
Linear: COW-1071. Sixth M4 issue landed; stacks on #39 (COW-1033).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…OW-1071)
Replaces the previous "bail on WS drop" semantic (flagged as the
"0.3 fix" in the source) with per-stream reconnect-aware tasks. Each
chain's block subscription and each (module, chain) log subscription
gets a dedicated task that:
1. Opens the subscription via `ProviderPool`.
2. Pumps items to an mpsc channel until the underlying stream
yields `None` (WS drop) or `Err` (transport-level error).
3. Logs the drop + sleeps for `restart_policy::backoff_for(attempt)`
(1s -> 2s -> 4s -> ... cap 5 min, reusing the COW-1033 policy).
4. Reopens. The first event after a reopen emits an `INFO ...
reopened` line + increments `shepherd_stream_reconnects_total`.
5. Resets `attempt = 0` once the stream has been healthy for the
`HEALTHY_WINDOW` (60 s of uninterrupted events) so a flaky-but-
then-stable connection reverts to fast retries on the next drop.
The event loop reads the channel as a regular `Stream` (wrapped
with `futures::stream::unfold` to avoid pulling in `tokio-stream`
just for `ReceiverStream`). A bare `None` from the merged stream
now indicates the reconnect task itself exited (panic or channel
closed); that path still bails the engine as before, but the log
message updated to reflect the new semantic.
## Key behavioural change
Public Sepolia (`wss://ethereum-sepolia-rpc.publicnode.com`) drops
WS connections after ~20 min of sustained load. Pre-fix: engine
bailed within seconds of the first drop, an operator restart
re-opened the subscription but the engine had missed every event in
between. Post-fix: the reconnect task waits 1s and reopens; only
events that arrived during the 1s gap are missed. Multi-minute drops
get progressively longer waits, capped at 5 min.
## New metric (consumed via COW-1034)
`shepherd_stream_reconnects_total{kind, chain_id, module}` counter,
incremented on every successful reopen. Operators write SLO alerts
against this for "stream churn" (e.g. > 5 reconnects per 10 min on
the same chain).
## Channel buffer + back-pressure
Buffer is 64 events per task. Real-time dispatch usually drains in
~12 s (Sepolia block time) so the buffer is overkill for normal
operation; it absorbs a brief dispatch-side stall (e.g. a stop-loss
cow-api submit that takes 2 s) without dropping events at the WS
boundary.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing
(unchanged shape - all existing tests still pass, including the
`run_does_not_bail_when_both_stream_kinds_are_empty` regression
guard which verifies the empty-stream path).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia happy path: `just run-m3` boots, all 3 modules
dispatch normally, `subscription open` log line emitted, no
reconnect activity in 60 s window (network was stable). Clean
SIGTERM shutdown.
## Out of scope
- WS endpoint failover (swap Alchemy <-> publicnode on failure).
Operator concern; track via `[engine.chains.<id>]` schema if
demand arises.
- Backfill of events missed during the drop window. Live-stream
semantic only; backfill is an indexer concern outside the M4
engine scope.
- Operator-tunable backoff / healthy-window via `engine.toml`. The
current constants are workspace literals; configurable in 0.3.
- Per-chain isolation across reconnects (COW-1073). The current
patch already gives partial isolation: each chain's task drops
+ reconnects independently and one task's failure does not
starve the others. COW-1073 covers the supervisor-side
multi-chain coordination.
Linear: COW-1071. Sixth M4 issue landed; stacks on #39 (COW-1033).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…OW-1071)
Replaces the previous "bail on WS drop" semantic (flagged as the
"0.3 fix" in the source) with per-stream reconnect-aware tasks. Each
chain's block subscription and each (module, chain) log subscription
gets a dedicated task that:
1. Opens the subscription via `ProviderPool`.
2. Pumps items to an mpsc channel until the underlying stream
yields `None` (WS drop) or `Err` (transport-level error).
3. Logs the drop + sleeps for `restart_policy::backoff_for(attempt)`
(1s -> 2s -> 4s -> ... cap 5 min, reusing the COW-1033 policy).
4. Reopens. The first event after a reopen emits an `INFO ...
reopened` line + increments `shepherd_stream_reconnects_total`.
5. Resets `attempt = 0` once the stream has been healthy for the
`HEALTHY_WINDOW` (60 s of uninterrupted events) so a flaky-but-
then-stable connection reverts to fast retries on the next drop.
The event loop reads the channel as a regular `Stream` (wrapped
with `futures::stream::unfold` to avoid pulling in `tokio-stream`
just for `ReceiverStream`). A bare `None` from the merged stream
now indicates the reconnect task itself exited (panic or channel
closed); that path still bails the engine as before, but the log
message updated to reflect the new semantic.
## Key behavioural change
Public Sepolia (`wss://ethereum-sepolia-rpc.publicnode.com`) drops
WS connections after ~20 min of sustained load. Pre-fix: engine
bailed within seconds of the first drop, an operator restart
re-opened the subscription but the engine had missed every event in
between. Post-fix: the reconnect task waits 1s and reopens; only
events that arrived during the 1s gap are missed. Multi-minute drops
get progressively longer waits, capped at 5 min.
## New metric (consumed via COW-1034)
`shepherd_stream_reconnects_total{kind, chain_id, module}` counter,
incremented on every successful reopen. Operators write SLO alerts
against this for "stream churn" (e.g. > 5 reconnects per 10 min on
the same chain).
## Channel buffer + back-pressure
Buffer is 64 events per task. Real-time dispatch usually drains in
~12 s (Sepolia block time) so the buffer is overkill for normal
operation; it absorbs a brief dispatch-side stall (e.g. a stop-loss
cow-api submit that takes 2 s) without dropping events at the WS
boundary.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing
(unchanged shape - all existing tests still pass, including the
`run_does_not_bail_when_both_stream_kinds_are_empty` regression
guard which verifies the empty-stream path).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia happy path: `just run-m3` boots, all 3 modules
dispatch normally, `subscription open` log line emitted, no
reconnect activity in 60 s window (network was stable). Clean
SIGTERM shutdown.
## Out of scope
- WS endpoint failover (swap Alchemy <-> publicnode on failure).
Operator concern; track via `[engine.chains.<id>]` schema if
demand arises.
- Backfill of events missed during the drop window. Live-stream
semantic only; backfill is an indexer concern outside the M4
engine scope.
- Operator-tunable backoff / healthy-window via `engine.toml`. The
current constants are workspace literals; configurable in 0.3.
- Per-chain isolation across reconnects (COW-1073). The current
patch already gives partial isolation: each chain's task drops
+ reconnects independently and one task's failure does not
starve the others. COW-1073 covers the supervisor-side
multi-chain coordination.
Linear: COW-1071. Sixth M4 issue landed; stacks on #39 (COW-1033).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When a module traps in `on_event` (OutOfFuel, MemoryOutOfBounds, unhandled host error), the supervisor now schedules an exponential-backoff restart instead of permanently quarantining the module. After the backoff window the supervisor re-instantiates the component (fresh wasmtime Store + bindings + re-call `init`) and dispatches again. A successful event resets the failure counter.
Fifth M4 issue landed.
Why reinstantiation, not just refuel
A wasmtime trap leaves the component instance poisoned: subsequent `call_on_event` returns `wasm trap: cannot enter component instance`. Just `set_fuel` on the Store does not recover - we have to tear it down. The supervisor caches the `Component`, `init_config`, and `http_allowlist` on `LoadedModule` at boot so a restart only needs a fresh Store + re-instantiation - the compiled component bytes are reused.
Backoff schedule
Changes
New integration test
`supervisor::tests::restart_flaky_module_recovers_after_backoff` (real-time, ~1.4s wall clock):
Breaking changes
`Supervisor` struct gained 4 new private fields. Callers that constructed `Supervisor { modules: ... }` directly (only the 2 unit tests in `supervisor::tests`) updated to use the new `Supervisor::empty_for_test(engine, store)` helper.
Tests
Out of scope
AI assistance disclosure
AI Assistance: this change + description was produced by a Claude Code agent (Claude Opus 4.7 1M context). The agent diagnosed the wasmtime "cannot enter component instance" semantic, designed the reinstantiation path, implemented the policy + test fixture, and authored this PR description. A human (Bruno) reviewed and is accountable for the result.
Linear: COW-1033. Stacks on #38 (COW-1034 Prometheus metrics).