feat(supervisor): poison-pill detection + quarantine (COW-1032) - #41
Closed
brunota20 wants to merge 1 commit into
Closed
feat(supervisor): poison-pill detection + quarantine (COW-1032)#41brunota20 wants to merge 1 commit into
brunota20 wants to merge 1 commit into
Conversation
Escalates the COW-1033 restart policy: when a module traps more
than `PoisonPolicy.max_failures` times within a sliding
`PoisonPolicy.window`, the supervisor marks it **poisoned**:
- Dispatch path skips poisoned modules forever (no further restart
attempts, no fuel + RPC cost on no-ops).
- A WARN log emits the module name + last error class with a hint
to remove it from `engine.toml::[[modules]]` + restart.
- `shepherd_module_poisoned{module}` gauge flips to 1.
Production thresholds: 5 traps inside 10 minutes -> quarantine.
Aggressive enough to catch a deterministically broken module
without burning every restart slot from the COW-1033 backoff
schedule; lenient enough that a one-off RPC blip during a real
cow-api submit does not get a module quarantined.
Recovery requires an operator action: remove the entry from
`engine.toml::[[modules]]` + restart the engine. There is no
automatic recovery on the production schedule; the assumption is
that 5 traps inside 10 min is a structural failure, not a
transient that would self-heal.
## New file
`crates/nexum-engine/src/runtime/poison_policy.rs`:
- `POISON_MAX_FAILURES = 5`, `POISON_WINDOW = 600 s` consts.
- `PoisonPolicy { max_failures, window }` struct with `Default`
pointing at production + `::new(...)` for tests.
- `should_poison(policy, recent_failures) -> bool` helper.
- 2 unit tests covering the threshold edge cases.
## supervisor.rs changes
- `Supervisor` gains `poison_policy: PoisonPolicy` (defaults to
production; tests override via `with_poison_policy`).
- `LoadedModule` gains `failure_timestamps: VecDeque<Instant>` +
`poisoned: bool`.
- New free-function `record_failure_and_maybe_poison` is called
from every trap arm in `dispatch_block` + `dispatch_log`. It
prunes old entries beyond the window, pushes the current
timestamp, and flips `poisoned = true` if the window holds
>= `policy.max_failures` entries.
- Restart sweep + dispatch fast-path both check `poisoned` first,
excluding quarantined modules from any further work.
- New `poisoned_count()` accessor for metrics + tests.
## New integration test
`poison_pill_quarantines_module_after_threshold` (real-time,
~3.5 s wall clock):
1. Boot fuel-bomb (always-trapping fixture from COW-1036) with a
tight policy: `PoisonPolicy::new(3, Duration::from_secs(60))`.
2. Dispatch 1 -> trap. failure_count=1, next_attempt=+1s, poisoned=0.
3. Sleep 1.1s, dispatch 2 -> trap. failure_count=2, poisoned=0.
4. Sleep 2.1s, dispatch 3 -> trap. failure_count=3. **3 failures
inside the 60-s window crosses the threshold -> poisoned=1.**
5. Dispatch 4 (no wait) -> returns 0, no restart attempt, no
dispatch entered. The module is silently excluded.
## Workspace impact
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(was 159 + 6; +2 from `poison_policy` units + 1 from the
integration test).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing tests pass against the new dispatch shape: the
`restart_flaky_module_recovers_after_backoff` test (COW-1033)
uses fail_first_n=1 with the default production policy, so the
module recovers well before the 5-trap threshold.
- `resource_limit_dead_bomb_does_not_starve_healthy_module`
(COW-1036) dispatches the bomb twice; both with the default
policy, well under 5 traps -> no quarantine.
## Out of scope
- Operator-tunable thresholds via `engine.toml::[engine.poison]`.
The current constants live in `runtime::poison_policy`;
configurable in 0.3.
- Auto-recovery via slow decay (e.g. "after 1 h of being
poisoned, try one more time"). The spec is explicit: poisoned
modules need operator action.
- Per-module poison policies. One workspace-wide threshold today.
Linear: COW-1032. Seventh M4 issue landed; stacks on #40 (COW-1071).
4 tasks
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…ce (COW-1072)
Two coupled changes that make operator-driven shutdowns observable
and recoverable:
## 1. Event loop: dispatch outside `select!`
`run()` previously had its `call_on_event().await` inside the
`tokio::select!`. A shutdown signal arriving mid-dispatch
cancelled the in-flight wasmtime call, leaving the wasm store in
an indeterminate state.
The refactor splits the loop into two phases:
- **Phase 1**: a small `tokio::select!` picks the next event OR
observes shutdown OR reports an upstream-task panic. Each branch
resolves into a `NextEvent` value; the select drops without
cancelling anything *outside* itself.
- **Phase 2**: `match next` dispatches the event to the supervisor
via a fully-awaited call, OR exits cleanly on the shutdown
variant.
The shutdown signal is now only observed *between* dispatches.
In-flight wasmtime calls always finish naturally.
## 2. Per-module last-dispatched-block persistence
Every successful `dispatch_block` writes a host-side marker to the
module's own local-store namespace:
```
namespace = module.name
key = "last_dispatched_block:{chain_id}"
value = block.number.to_le_bytes()
```
The marker survives engine restarts (it lives in the redb file
under `state_dir`). Operators can confirm at-which-block an engine
last ran without trawling the logs; modules that care about block-
gap detection can read it back on their next `init`.
Write failures are best-effort (a `WARN` log; the dispatch is not
considered failed).
## 3. Graceful shutdown log
The event loop now emits a structured exit line:
```
INFO graceful shutdown complete
dispatched_blocks=N dispatched_logs=M uptime_secs=K
```
Visible live on Sepolia after a `kill -TERM`:
```
INFO shutdown signal received signal=SIGTERM
INFO graceful shutdown complete dispatched_blocks=1 dispatched_logs=0 uptime_secs=13
```
## Out of scope
- 30s drain timeout via `tokio::time::timeout`. The current
dispatch path always terminates (fuel cap caps wall time to <1s
in practice); a 30s drain timer is dead code today. Worth
adding once a module ever needs longer-running host calls (HTTP
capability, etc).
- `engine.toml::[engine.shutdown]` config knobs. The internal
default is "wait as long as the in-flight dispatch takes";
configurable in 0.3.
- Module-side `shutdown` hook. Modules just see the dispatch
complete normally; the supervisor exits without invoking
anything new.
## Tests
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(unchanged shape; the dispatch refactor is transparent to the
existing test suite).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: ran the engine, observed the graceful
shutdown log + last_dispatched_block markers in
`data/m3/local-store.redb`.
Linear: COW-1072. Eighth M4 issue landed; stacks on #41 (COW-1032).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…ce (COW-1072)
Two coupled changes that make operator-driven shutdowns observable
and recoverable:
## 1. Event loop: dispatch outside `select!`
`run()` previously had its `call_on_event().await` inside the
`tokio::select!`. A shutdown signal arriving mid-dispatch
cancelled the in-flight wasmtime call, leaving the wasm store in
an indeterminate state.
The refactor splits the loop into two phases:
- **Phase 1**: a small `tokio::select!` picks the next event OR
observes shutdown OR reports an upstream-task panic. Each branch
resolves into a `NextEvent` value; the select drops without
cancelling anything *outside* itself.
- **Phase 2**: `match next` dispatches the event to the supervisor
via a fully-awaited call, OR exits cleanly on the shutdown
variant.
The shutdown signal is now only observed *between* dispatches.
In-flight wasmtime calls always finish naturally.
## 2. Per-module last-dispatched-block persistence
Every successful `dispatch_block` writes a host-side marker to the
module's own local-store namespace:
```
namespace = module.name
key = "last_dispatched_block:{chain_id}"
value = block.number.to_le_bytes()
```
The marker survives engine restarts (it lives in the redb file
under `state_dir`). Operators can confirm at-which-block an engine
last ran without trawling the logs; modules that care about block-
gap detection can read it back on their next `init`.
Write failures are best-effort (a `WARN` log; the dispatch is not
considered failed).
## 3. Graceful shutdown log
The event loop now emits a structured exit line:
```
INFO graceful shutdown complete
dispatched_blocks=N dispatched_logs=M uptime_secs=K
```
Visible live on Sepolia after a `kill -TERM`:
```
INFO shutdown signal received signal=SIGTERM
INFO graceful shutdown complete dispatched_blocks=1 dispatched_logs=0 uptime_secs=13
```
## Out of scope
- 30s drain timeout via `tokio::time::timeout`. The current
dispatch path always terminates (fuel cap caps wall time to <1s
in practice); a 30s drain timer is dead code today. Worth
adding once a module ever needs longer-running host calls (HTTP
capability, etc).
- `engine.toml::[engine.shutdown]` config knobs. The internal
default is "wait as long as the in-flight dispatch takes";
configurable in 0.3.
- Module-side `shutdown` hook. Modules just see the dispatch
complete normally; the supervisor exits without invoking
anything new.
## Tests
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(unchanged shape; the dispatch refactor is transparent to the
existing test suite).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: ran the engine, observed the graceful
shutdown log + last_dispatched_block markers in
`data/m3/local-store.redb`.
Linear: COW-1072. Eighth M4 issue landed; stacks on #41 (COW-1032).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…ce (COW-1072)
Two coupled changes that make operator-driven shutdowns observable
and recoverable:
## 1. Event loop: dispatch outside `select!`
`run()` previously had its `call_on_event().await` inside the
`tokio::select!`. A shutdown signal arriving mid-dispatch
cancelled the in-flight wasmtime call, leaving the wasm store in
an indeterminate state.
The refactor splits the loop into two phases:
- **Phase 1**: a small `tokio::select!` picks the next event OR
observes shutdown OR reports an upstream-task panic. Each branch
resolves into a `NextEvent` value; the select drops without
cancelling anything *outside* itself.
- **Phase 2**: `match next` dispatches the event to the supervisor
via a fully-awaited call, OR exits cleanly on the shutdown
variant.
The shutdown signal is now only observed *between* dispatches.
In-flight wasmtime calls always finish naturally.
## 2. Per-module last-dispatched-block persistence
Every successful `dispatch_block` writes a host-side marker to the
module's own local-store namespace:
```
namespace = module.name
key = "last_dispatched_block:{chain_id}"
value = block.number.to_le_bytes()
```
The marker survives engine restarts (it lives in the redb file
under `state_dir`). Operators can confirm at-which-block an engine
last ran without trawling the logs; modules that care about block-
gap detection can read it back on their next `init`.
Write failures are best-effort (a `WARN` log; the dispatch is not
considered failed).
## 3. Graceful shutdown log
The event loop now emits a structured exit line:
```
INFO graceful shutdown complete
dispatched_blocks=N dispatched_logs=M uptime_secs=K
```
Visible live on Sepolia after a `kill -TERM`:
```
INFO shutdown signal received signal=SIGTERM
INFO graceful shutdown complete dispatched_blocks=1 dispatched_logs=0 uptime_secs=13
```
## Out of scope
- 30s drain timeout via `tokio::time::timeout`. The current
dispatch path always terminates (fuel cap caps wall time to <1s
in practice); a 30s drain timer is dead code today. Worth
adding once a module ever needs longer-running host calls (HTTP
capability, etc).
- `engine.toml::[engine.shutdown]` config knobs. The internal
default is "wait as long as the in-flight dispatch takes";
configurable in 0.3.
- Module-side `shutdown` hook. Modules just see the dispatch
complete normally; the supervisor exits without invoking
anything new.
## Tests
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(unchanged shape; the dispatch refactor is transparent to the
existing test suite).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: ran the engine, observed the graceful
shutdown log + last_dispatched_block markers in
`data/m3/local-store.redb`.
Linear: COW-1072. Eighth M4 issue landed; stacks on #41 (COW-1032).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…ce (COW-1072)
Two coupled changes that make operator-driven shutdowns observable
and recoverable:
## 1. Event loop: dispatch outside `select!`
`run()` previously had its `call_on_event().await` inside the
`tokio::select!`. A shutdown signal arriving mid-dispatch
cancelled the in-flight wasmtime call, leaving the wasm store in
an indeterminate state.
The refactor splits the loop into two phases:
- **Phase 1**: a small `tokio::select!` picks the next event OR
observes shutdown OR reports an upstream-task panic. Each branch
resolves into a `NextEvent` value; the select drops without
cancelling anything *outside* itself.
- **Phase 2**: `match next` dispatches the event to the supervisor
via a fully-awaited call, OR exits cleanly on the shutdown
variant.
The shutdown signal is now only observed *between* dispatches.
In-flight wasmtime calls always finish naturally.
## 2. Per-module last-dispatched-block persistence
Every successful `dispatch_block` writes a host-side marker to the
module's own local-store namespace:
```
namespace = module.name
key = "last_dispatched_block:{chain_id}"
value = block.number.to_le_bytes()
```
The marker survives engine restarts (it lives in the redb file
under `state_dir`). Operators can confirm at-which-block an engine
last ran without trawling the logs; modules that care about block-
gap detection can read it back on their next `init`.
Write failures are best-effort (a `WARN` log; the dispatch is not
considered failed).
## 3. Graceful shutdown log
The event loop now emits a structured exit line:
```
INFO graceful shutdown complete
dispatched_blocks=N dispatched_logs=M uptime_secs=K
```
Visible live on Sepolia after a `kill -TERM`:
```
INFO shutdown signal received signal=SIGTERM
INFO graceful shutdown complete dispatched_blocks=1 dispatched_logs=0 uptime_secs=13
```
## Out of scope
- 30s drain timeout via `tokio::time::timeout`. The current
dispatch path always terminates (fuel cap caps wall time to <1s
in practice); a 30s drain timer is dead code today. Worth
adding once a module ever needs longer-running host calls (HTTP
capability, etc).
- `engine.toml::[engine.shutdown]` config knobs. The internal
default is "wait as long as the in-flight dispatch takes";
configurable in 0.3.
- Module-side `shutdown` hook. Modules just see the dispatch
complete normally; the supervisor exits without invoking
anything new.
## Tests
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(unchanged shape; the dispatch refactor is transparent to the
existing test suite).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: ran the engine, observed the graceful
shutdown log + last_dispatched_block markers in
`data/m3/local-store.redb`.
Linear: COW-1072. Eighth M4 issue landed; stacks on #41 (COW-1032).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Escalates the COW-1033 restart policy: when a module traps more than `PoisonPolicy.max_failures` times within a sliding `PoisonPolicy.window`, the supervisor marks it poisoned and stops all dispatch / restart attempts. Recovery is operator-driven only.
Seventh M4 issue landed.
Production thresholds
5 traps inside 10 minutes -> quarantine. Aggressive enough to catch a deterministically broken module without burning every restart slot from the COW-1033 backoff schedule; lenient enough that a one-off RPC blip during a real cow-api submit does not get a module quarantined.
Changes
New metric
`shepherd_module_poisoned{module}` gauge: flips to 1 on the first quarantine. Surfaces in Grafana so SRE can page on "module went terminal".
New integration test (real-time, ~3.5 s wall clock)
`poison_pill_quarantines_module_after_threshold` boots fuel-bomb (the always-trapping COW-1036 fixture) with a tight `PoisonPolicy::new(3, 60s)`:
Out of scope
Tests
AI assistance disclosure
AI Assistance: this change + description was produced by a Claude Code agent (Claude Opus 4.7 1M context). The agent designed the policy + sliding-window check, implemented the helper + dispatch integration, validated against the existing test suite, and authored this PR description. A human (Bruno) reviewed and is accountable for the result.
Linear: COW-1032. Stacks on #40 (COW-1071 WS reconnect).