test(resource-limits): evil fixtures + 3 trap-isolation tests (COW-1036) - #36
Closed
brunota20 wants to merge 1 commit into
Closed
Conversation
…1036)
Locks the M1 fuel + memory wiring (BLEU-818) against regression with
two evil-by-design wasm fixtures and three supervisor integration
tests that exercise the full trap path: dispatch -> wasmtime trap ->
supervisor catches -> module marked dead -> engine continues.
## New fixtures
`modules/fixtures/fuel-bomb/` (66 KB wasm)
on_event runs an unbounded `wrapping_add` loop with
`std::hint::black_box` so the optimiser cannot elide it.
wasmtime exhausts the per-event DEFAULT_FUEL_PER_EVENT (1B) and
traps with OutOfFuel.
`modules/fixtures/memory-bomb/` (67 KB wasm)
on_event allocates 128 MiB which exceeds the per-store
DEFAULT_MEMORY_LIMIT (64 MiB). wasmtime rejects the memory.grow
and traps the module.
Both fixtures live under `modules/fixtures/` so they are obviously
test-only - the M2 / M3 testnet configs never reference them. Both
declare only the `logging` capability + a single block subscription.
## New supervisor integration tests
`resource_limit_fuel_bomb_traps_and_marks_module_dead`
Boots fuel-bomb alone, dispatches a block, asserts:
- dispatched == 0 (trap, not delivery)
- alive_count() == 0 (module marked dead)
- second dispatch returns 0 (dead module excluded)
-> proves the fuel limit fires + the supervisor catches the
trap without panicking.
`resource_limit_memory_bomb_traps_and_marks_module_dead`
Same shape for the 64 MiB cap. The wasm32 allocator surfaces
"memory allocation of 134217728 bytes failed" (the trap firing).
`resource_limit_dead_bomb_does_not_starve_healthy_module`
Strongest isolation test: loads fuel-bomb + the M1 example
module side-by-side, dispatches a block, asserts:
- dispatched == 1 (example survived + accepted the dispatch
even though the bomb trapped on the same block)
- alive_count() == 1 (only example alive)
- second dispatch == 1 (dead bomb skipped, example continues)
-> proves a rogue module cannot starve the supervisor or
starve sibling modules.
## Validation
- `cargo test -p nexum-engine resource_limit` -> 3 passed.
- `cargo test --workspace` -> 154 host tests + 6 doctests passing
(was 151 + 6; +3 from the new tests).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release -p {fuel-bomb,memory-bomb}`
clean.
- 0 em-dashes in new files.
## Out of scope
- Fuel + memory limits made configurable per-module via `engine.toml`
(today they are workspace constants in `runtime/limits.rs`).
Already noted in the source comments as "configurable in 0.3";
acknowledged not addressed here.
- Adversarial fuzz of the resource-limit defaults under sustained
load. That is COW-1065 (security review) territory.
- CI integration of the fixtures into the build matrix (PR #27).
Not needed - `cargo test --workspace` already builds them in the
test profile, and the `module_wasm_or_skip` guard means CI does
not need a separate fixture-build job.
Linear: COW-1036. Second M4 issue landed; stacks on #35 (COW-1029).
5 tasks
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Closes the COW-1035 structured-logging audit. The engine now ships
JSON-formatted logs by default, with consistent field shapes across
every dispatch, host call, and order submission. A single `jq` /
Loki / Grafana stream over the engine output reconstructs the full
timeline of any module event.
## Changes
### Formatter
- New `--pretty-logs` CLI flag (parsed in `cli.rs`). When set, the
engine uses the historical 0.1 human-readable formatter; otherwise
emits JSON with flattened event fields + no current-span noise.
- `main.rs` `tracing_subscriber::fmt` builder picks the formatter from
the flag. `EnvFilter` (`RUST_LOG` / `engine.toml::[engine].log_level`)
applies to both.
- `tracing-subscriber` feature set gains `json`.
### Dispatch logs (supervisor.rs)
Every `dispatch_block` and `dispatch_log` invocation now emits a
structured log line:
- `dispatch ok` (DEBUG) on success with `module`, `chain_id`,
`event_kind` ("block" / "log"), `block_number`, `latency_ms`.
- Existing host-error WARN + trap ERROR paths gain the same fields
for cross-correlation.
Live Sepolia validation:
```json
{"timestamp":"2026-06-18T13:56:48.587125Z","level":"DEBUG","message":"dispatch ok",
"module":"price-alert","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":134,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:48.857911Z","level":"DEBUG","message":"dispatch ok",
"module":"balance-tracker","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":270,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:50.193531Z","level":"DEBUG","message":"dispatch ok",
"module":"stop-loss","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":1335,"target":"nexum_engine::supervisor"}
```
The latency-per-module distribution is now observable per-block:
price-alert ~135 ms (1 eth_call), balance-tracker ~270 ms (2
eth_getBalance), stop-loss ~1.3 s (oracle read + OrderCreation +
cow-api submit + retry classify).
### Host-call logs (pre-existing, kept as-is)
The M1 host backends already emit structured DEBUG with `chain_id`,
`method`, `bytes`, `latency_ms` on every `chain::request` /
`cow-api::submit-order` / `cow-api::request`. The audit confirmed
they already satisfy the COW-1035 contract; no changes needed.
### Runbook ergonomics
`just run-m2` and `just run-m3` pass `--pretty-logs` so the
runbook output samples (M2 + M3 runbooks) keep matching what the
operator sees locally. Production deploys (`cargo run -p
nexum-engine -- --engine-config engine.toml` directly, e.g. from
a Docker entrypoint) get JSON by default.
## Validation
- `cargo test --workspace` -> 154 host tests + 6 doctests passing.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: JSON output captured; per-module dispatch
latencies plausible against expected work per module.
- Pretty-logs flag verified to opt back into the 0.1 formatter.
## What this still doesn't do
- Per-order timeline aggregation (would need a `uid` field on the
stop-loss / twap-monitor submit logs). Currently the order UID
is logged by each module's `host.log` call (module-side) but
not joined with the supervisor dispatch line. Acceptable today;
the JSON shape supports adding the field later without breaking
consumers.
- Trace IDs / spans across the dispatch -> host-call boundary.
Worth a follow-up once Prometheus (COW-1034) lands and we know
the metric labels we want to correlate against.
Linear: COW-1035. Third M4 issue landed; stacks on #36 (COW-1036).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Closes the COW-1035 structured-logging audit. The engine now ships
JSON-formatted logs by default, with consistent field shapes across
every dispatch, host call, and order submission. A single `jq` /
Loki / Grafana stream over the engine output reconstructs the full
timeline of any module event.
## Changes
### Formatter
- New `--pretty-logs` CLI flag (parsed in `cli.rs`). When set, the
engine uses the historical 0.1 human-readable formatter; otherwise
emits JSON with flattened event fields + no current-span noise.
- `main.rs` `tracing_subscriber::fmt` builder picks the formatter from
the flag. `EnvFilter` (`RUST_LOG` / `engine.toml::[engine].log_level`)
applies to both.
- `tracing-subscriber` feature set gains `json`.
### Dispatch logs (supervisor.rs)
Every `dispatch_block` and `dispatch_log` invocation now emits a
structured log line:
- `dispatch ok` (DEBUG) on success with `module`, `chain_id`,
`event_kind` ("block" / "log"), `block_number`, `latency_ms`.
- Existing host-error WARN + trap ERROR paths gain the same fields
for cross-correlation.
Live Sepolia validation:
```json
{"timestamp":"2026-06-18T13:56:48.587125Z","level":"DEBUG","message":"dispatch ok",
"module":"price-alert","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":134,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:48.857911Z","level":"DEBUG","message":"dispatch ok",
"module":"balance-tracker","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":270,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:50.193531Z","level":"DEBUG","message":"dispatch ok",
"module":"stop-loss","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":1335,"target":"nexum_engine::supervisor"}
```
The latency-per-module distribution is now observable per-block:
price-alert ~135 ms (1 eth_call), balance-tracker ~270 ms (2
eth_getBalance), stop-loss ~1.3 s (oracle read + OrderCreation +
cow-api submit + retry classify).
### Host-call logs (pre-existing, kept as-is)
The M1 host backends already emit structured DEBUG with `chain_id`,
`method`, `bytes`, `latency_ms` on every `chain::request` /
`cow-api::submit-order` / `cow-api::request`. The audit confirmed
they already satisfy the COW-1035 contract; no changes needed.
### Runbook ergonomics
`just run-m2` and `just run-m3` pass `--pretty-logs` so the
runbook output samples (M2 + M3 runbooks) keep matching what the
operator sees locally. Production deploys (`cargo run -p
nexum-engine -- --engine-config engine.toml` directly, e.g. from
a Docker entrypoint) get JSON by default.
## Validation
- `cargo test --workspace` -> 154 host tests + 6 doctests passing.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: JSON output captured; per-module dispatch
latencies plausible against expected work per module.
- Pretty-logs flag verified to opt back into the 0.1 formatter.
## What this still doesn't do
- Per-order timeline aggregation (would need a `uid` field on the
stop-loss / twap-monitor submit logs). Currently the order UID
is logged by each module's `host.log` call (module-side) but
not joined with the supervisor dispatch line. Acceptable today;
the JSON shape supports adding the field later without breaking
consumers.
- Trace IDs / spans across the dispatch -> host-call boundary.
Worth a follow-up once Prometheus (COW-1034) lands and we know
the metric labels we want to correlate against.
Linear: COW-1035. Third M4 issue landed; stacks on #36 (COW-1036).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Closes the COW-1035 structured-logging audit. The engine now ships
JSON-formatted logs by default, with consistent field shapes across
every dispatch, host call, and order submission. A single `jq` /
Loki / Grafana stream over the engine output reconstructs the full
timeline of any module event.
## Changes
### Formatter
- New `--pretty-logs` CLI flag (parsed in `cli.rs`). When set, the
engine uses the historical 0.1 human-readable formatter; otherwise
emits JSON with flattened event fields + no current-span noise.
- `main.rs` `tracing_subscriber::fmt` builder picks the formatter from
the flag. `EnvFilter` (`RUST_LOG` / `engine.toml::[engine].log_level`)
applies to both.
- `tracing-subscriber` feature set gains `json`.
### Dispatch logs (supervisor.rs)
Every `dispatch_block` and `dispatch_log` invocation now emits a
structured log line:
- `dispatch ok` (DEBUG) on success with `module`, `chain_id`,
`event_kind` ("block" / "log"), `block_number`, `latency_ms`.
- Existing host-error WARN + trap ERROR paths gain the same fields
for cross-correlation.
Live Sepolia validation:
```json
{"timestamp":"2026-06-18T13:56:48.587125Z","level":"DEBUG","message":"dispatch ok",
"module":"price-alert","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":134,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:48.857911Z","level":"DEBUG","message":"dispatch ok",
"module":"balance-tracker","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":270,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:50.193531Z","level":"DEBUG","message":"dispatch ok",
"module":"stop-loss","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":1335,"target":"nexum_engine::supervisor"}
```
The latency-per-module distribution is now observable per-block:
price-alert ~135 ms (1 eth_call), balance-tracker ~270 ms (2
eth_getBalance), stop-loss ~1.3 s (oracle read + OrderCreation +
cow-api submit + retry classify).
### Host-call logs (pre-existing, kept as-is)
The M1 host backends already emit structured DEBUG with `chain_id`,
`method`, `bytes`, `latency_ms` on every `chain::request` /
`cow-api::submit-order` / `cow-api::request`. The audit confirmed
they already satisfy the COW-1035 contract; no changes needed.
### Runbook ergonomics
`just run-m2` and `just run-m3` pass `--pretty-logs` so the
runbook output samples (M2 + M3 runbooks) keep matching what the
operator sees locally. Production deploys (`cargo run -p
nexum-engine -- --engine-config engine.toml` directly, e.g. from
a Docker entrypoint) get JSON by default.
## Validation
- `cargo test --workspace` -> 154 host tests + 6 doctests passing.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: JSON output captured; per-module dispatch
latencies plausible against expected work per module.
- Pretty-logs flag verified to opt back into the 0.1 formatter.
## What this still doesn't do
- Per-order timeline aggregation (would need a `uid` field on the
stop-loss / twap-monitor submit logs). Currently the order UID
is logged by each module's `host.log` call (module-side) but
not joined with the supervisor dispatch line. Acceptable today;
the JSON shape supports adding the field later without breaking
consumers.
- Trace IDs / spans across the dispatch -> host-call boundary.
Worth a follow-up once Prometheus (COW-1034) lands and we know
the metric labels we want to correlate against.
Linear: COW-1035. Third M4 issue landed; stacks on #36 (COW-1036).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Closes the COW-1035 structured-logging audit. The engine now ships
JSON-formatted logs by default, with consistent field shapes across
every dispatch, host call, and order submission. A single `jq` /
Loki / Grafana stream over the engine output reconstructs the full
timeline of any module event.
- New `--pretty-logs` CLI flag (parsed in `cli.rs`). When set, the
engine uses the historical 0.1 human-readable formatter; otherwise
emits JSON with flattened event fields + no current-span noise.
- `main.rs` `tracing_subscriber::fmt` builder picks the formatter from
the flag. `EnvFilter` (`RUST_LOG` / `engine.toml::[engine].log_level`)
applies to both.
- `tracing-subscriber` feature set gains `json`.
Every `dispatch_block` and `dispatch_log` invocation now emits a
structured log line:
- `dispatch ok` (DEBUG) on success with `module`, `chain_id`,
`event_kind` ("block" / "log"), `block_number`, `latency_ms`.
- Existing host-error WARN + trap ERROR paths gain the same fields
for cross-correlation.
Live Sepolia validation:
```json
{"timestamp":"2026-06-18T13:56:48.587125Z","level":"DEBUG","message":"dispatch ok",
"module":"price-alert","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":134,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:48.857911Z","level":"DEBUG","message":"dispatch ok",
"module":"balance-tracker","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":270,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:50.193531Z","level":"DEBUG","message":"dispatch ok",
"module":"stop-loss","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":1335,"target":"nexum_engine::supervisor"}
```
The latency-per-module distribution is now observable per-block:
price-alert ~135 ms (1 eth_call), balance-tracker ~270 ms (2
eth_getBalance), stop-loss ~1.3 s (oracle read + OrderCreation +
cow-api submit + retry classify).
The M1 host backends already emit structured DEBUG with `chain_id`,
`method`, `bytes`, `latency_ms` on every `chain::request` /
`cow-api::submit-order` / `cow-api::request`. The audit confirmed
they already satisfy the COW-1035 contract; no changes needed.
`just run-m2` and `just run-m3` pass `--pretty-logs` so the
runbook output samples (M2 + M3 runbooks) keep matching what the
operator sees locally. Production deploys (`cargo run -p
nexum-engine -- --engine-config engine.toml` directly, e.g. from
a Docker entrypoint) get JSON by default.
- `cargo test --workspace` -> 154 host tests + 6 doctests passing.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: JSON output captured; per-module dispatch
latencies plausible against expected work per module.
- Pretty-logs flag verified to opt back into the 0.1 formatter.
- Per-order timeline aggregation (would need a `uid` field on the
stop-loss / twap-monitor submit logs). Currently the order UID
is logged by each module's `host.log` call (module-side) but
not joined with the supervisor dispatch line. Acceptable today;
the JSON shape supports adding the field later without breaking
consumers.
- Trace IDs / spans across the dispatch -> host-call boundary.
Worth a follow-up once Prometheus (COW-1034) lands and we know
the metric labels we want to correlate against.
Linear: COW-1035. Third M4 issue landed; stacks on #36 (COW-1036).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Locks the M1 fuel + memory wiring (BLEU-818) against regression with two evil-by-design wasm fixtures and three supervisor integration tests that exercise the full trap path: dispatch -> wasmtime trap -> supervisor catches -> module marked dead -> engine continues.
Second M4 issue landed.
New fixtures
Both fixtures live under `modules/fixtures/` so they are obviously test-only - the M2 / M3 testnet configs never reference them. Both declare only `logging` + a single block subscription.
New supervisor integration tests
The third test is the load-bearing invariant for production: a rogue module cannot starve the supervisor or sibling modules.
Changes
Breaking changes
None. Test-only additions + 2 new workspace members.
Testing
Out of scope
AI assistance disclosure
AI Assistance: this change + description was produced by a Claude Code agent (Claude Opus 4.7 1M context). The agent designed the evil fixtures, wrote the integration tests, validated the trap path locally, and authored this PR description. A human (Bruno) reviewed and is accountable for the result.
Linear: COW-1036. Second M4 issue landed; stacks on #35 (COW-1029 non_exhaustive).