feat(metrics): Prometheus /metrics endpoint + 4 recording sites (COW-1034) - #38
Closed
brunota20 wants to merge 1 commit into
Closed
Conversation
…1034)
Wires `metrics` + `metrics-exporter-prometheus` into the engine.
When `engine.toml::[engine.metrics].enabled = true` the engine binds
a Prometheus HTTP exporter on the configured `bind_addr` (default
`127.0.0.1:9100`) and serves `/metrics`. When disabled, the recorder
is still installed so call sites stay live but no port binds. The
same binary now ships into CI / tests (recorder no-ops) and into
production (full exporter) by flipping one config flag.
## Recording sites instrumented
| Metric | Type | Labels | Site |
|---|---|---|---|
| `shepherd_event_latency_seconds` | summary | module, event_kind | supervisor::dispatch_{block,log} on the OK path |
| `shepherd_module_errors_total` | counter | module, error_kind | supervisor::dispatch_{block,log} on host-error WARN + trap ERROR paths (error_kind = `{:?}` of HostErrorKind, or `"trap"`) |
| `shepherd_chain_request_total` | counter | chain_id, method, outcome | host::impls::chain after every `chain::request` |
| `shepherd_cow_api_submit_total` | counter | chain_id, outcome | host::impls::cow_api after every `cow-api::submit-order` |
Sites match the structured logging audit (COW-1035) so each metric
event has a sibling log line with the same labels for cross-
correlation.
## Live Sepolia scrape
```
# TYPE shepherd_cow_api_submit_total counter
shepherd_cow_api_submit_total{chain_id="11155111",outcome="err"} 2
# TYPE shepherd_chain_request_total counter
shepherd_chain_request_total{chain_id="11155111",method="eth_getBalance",outcome="ok"} 4
shepherd_chain_request_total{chain_id="11155111",method="eth_call",outcome="ok"} 4
# TYPE shepherd_event_latency_seconds summary
shepherd_event_latency_seconds{module="price-alert", event_kind="block",quantile="0.5"} 0.1394
shepherd_event_latency_seconds{module="stop-loss", event_kind="block",quantile="0.5"} 0.9091
shepherd_event_latency_seconds{module="balance-tracker",event_kind="block",quantile="0.5"} 0.2823
```
Quantiles (p50/p90/p95/p99) appear automatically via the default
reservoir sampling. Production SRE can write SLO alerts against
these without any further wiring.
## Config schema
```toml
[engine.metrics]
enabled = false # default
bind_addr = "127.0.0.1:9100"
```
Disabled by default so M3 runbook smoke runs do not bind a port
unintentionally. Operators flip `enabled = true` for production.
## Deferred from issue scope
| Metric | Why deferred |
|---|---|
| `shepherd_module_uptime_seconds` | Needs a per-module start-time tracker on LoadedModule. Easy to add but cluttered the supervisor.rs diff. Worth a follow-up. |
| `shepherd_fuel_consumed` | Requires reading `store.get_fuel()` after each dispatch to compute consumed = DEFAULT - remaining. Mechanical addition, not in this PR to keep scope tight. |
| `shepherd_memory_peak_bytes` | Requires inspecting the wasmtime component's exported Memory instance. Harder; defer until the memory-bomb fixture surfaces a useful baseline. |
| `shepherd_module_restarts_total` | Depends on COW-1033 (supervisor auto-restart) which is not built yet. Wired into restart sites when that lands. |
| `shepherd_module_poisoned` | Depends on COW-1032 (poison pill) for the same reason. |
The 4 metrics shipped here cover the load-bearing observability use
cases (dispatch latency SLO + error rate by kind + per-RPC method
volume + per-orderbook submit outcome). The 5 deferred ones lock in
on top of M4 work that isn't built yet.
## Validation
- `cargo test --workspace` -> 154 host tests + 6 doctests passing.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: enabled the exporter on port 9101, scraped
/metrics via `curl`, confirmed all 4 metrics flowing with correct
labels.
- Recorder no-op path verified: `[engine.metrics].enabled = false`
(the default) does not bind a port; call sites stay silent but
do not panic.
Linear: COW-1034. Fourth M4 issue landed; stacks on #37 (COW-1035).
5 tasks
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…tiation (COW-1033)
When a module traps in `on_event` (OutOfFuel, MemoryOutOfBounds,
unhandled host error), the supervisor now:
1. Marks the module `alive = false` and increments `failure_count`.
2. Schedules a `next_attempt` instant via the new
`runtime::restart_policy::backoff_for` (1s → 2s → 4s → ... cap
5 min). All dispatches before that instant skip the module.
3. On the first dispatch past the backoff window, the supervisor
tears down the trapped wasmtime Store + component instance and
re-instantiates from the cached `Component`. The instance state
resets but host-side persistent state (local-store) survives
so a module's progress counters live across restarts.
4. On a successful `on_event` after recovery, `failure_count` resets
to 0 + `next_attempt = None`.
## Why the reinstantiation is required
A wasmtime trap leaves the component instance poisoned: subsequent
`call_on_event` returns "wasm trap: cannot enter component instance".
Just refueling the Store does not recover. The supervisor caches
the `Component`, `init_config`, and `http_allowlist` on
`LoadedModule` at boot so a restart only needs a fresh Store +
re-instantiation - the compiled component bytes are reused.
## New types / files
- `crates/nexum-engine/src/runtime/restart_policy.rs`: `backoff_for(failure_count) -> Duration` with the 1s → 5min schedule. 4 unit tests covering the steady-state, first-failure, doubling, and cap arms.
- `Supervisor` gains four cached backends (`engine`, `cow_pool`, `provider_pool`, `local_store`) so `reinstantiate_one(idx)` can rebuild the wasi Linker + HostState + Store + bindings on demand.
- `LoadedModule` gains `component: Component`, `init_config: Config`, `http_allowlist: Vec<String>` (all cloned at boot), plus `failure_count: u32` and `next_attempt: Option<Instant>` for the schedule.
## Dispatch path changes
`dispatch_block` and `dispatch_log` now restructure into two
phases:
1. **Phase 1 (restart sweep)**: walk modules, collect indices of
dead-but-due modules, call `reinstantiate_one` on each. Failed
restarts bump the backoff again. Successful restarts flip
`alive = true` so phase 2 dispatches the next event to them.
2. **Phase 2 (steady-state dispatch)**: unchanged from before -
walk modules, dispatch where subscribed + alive. Trap path
sets `next_attempt` + bumps `failure_count`; success path
resets both.
The structured logs from COW-1035 gain `failure_count` + `backoff_ms`
on trap + `restart attempt` info lines on each restart. The
`shepherd_module_restarts_total{module}` Prometheus counter from
COW-1034 increments on every restart attempt.
## New fixture + integration test
`modules/fixtures/flaky-bomb/` (test-only): traps via OutOfFuel on
the first N events (N from `[config].fail_first_n`) and recovers
afterwards. Uses local-store for the attempt counter because the
wasm instance state resets on each reinstantiation; the counter
persists in the host-side store so the module deterministically
recovers after the configured N.
`supervisor::tests::restart_flaky_module_recovers_after_backoff`
(new): boots flaky-bomb with fail_first_n=1, dispatches, observes:
- Dispatch 1: trap. alive=false, failure_count=1, next_attempt=+1s.
- Immediate redispatch: skipped (still in backoff).
- Sleep 1.1s.
- Dispatch 3: restart fires, fresh instance attempts again. With
attempt=2 > N=1, returns Ok. alive=true, failure_count=0,
next_attempt=None.
- Dispatch 4: steady-state, dispatches normally.
Test wall-clock ~1.4s.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing.
+4 from `restart_policy` unit tests + 1 from the new integration
test (was 154 + 6).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing resource-limit tests (COW-1036) still pass against
the new dispatch shape: their assertions are against state
*immediately* after the trap (before backoff elapses), so the
restart machinery is transparent.
- The `init_failure_marks_module_dead_and_excludes_from_dispatch`
test (COW-1070) still passes: init-failed modules carry
`next_attempt = None` so the restart sweep never picks them up.
## Out of scope
- Persistence of `failure_count` / `next_attempt` across full
engine restarts. The schedule resets on every boot; cross-engine
persistence is a 0.3 follow-up.
- WS reconnect-with-backoff for upstream RPC drops - that is
COW-1071, a separate axis.
- Operator-tunable backoff via `engine.toml::[engine.restart]`.
The current constants are workspace literals in
`runtime::restart_policy`; configurable in 0.3.
- Module-side `on_restart` hook. Modules just see a fresh `init`
call after a restart, same as boot.
Linear: COW-1033. Fifth M4 issue landed; stacks on #38 (COW-1034).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…tiation (COW-1033)
When a module traps in `on_event` (OutOfFuel, MemoryOutOfBounds,
unhandled host error), the supervisor now:
1. Marks the module `alive = false` and increments `failure_count`.
2. Schedules a `next_attempt` instant via the new
`runtime::restart_policy::backoff_for` (1s → 2s → 4s → ... cap
5 min). All dispatches before that instant skip the module.
3. On the first dispatch past the backoff window, the supervisor
tears down the trapped wasmtime Store + component instance and
re-instantiates from the cached `Component`. The instance state
resets but host-side persistent state (local-store) survives
so a module's progress counters live across restarts.
4. On a successful `on_event` after recovery, `failure_count` resets
to 0 + `next_attempt = None`.
## Why the reinstantiation is required
A wasmtime trap leaves the component instance poisoned: subsequent
`call_on_event` returns "wasm trap: cannot enter component instance".
Just refueling the Store does not recover. The supervisor caches
the `Component`, `init_config`, and `http_allowlist` on
`LoadedModule` at boot so a restart only needs a fresh Store +
re-instantiation - the compiled component bytes are reused.
## New types / files
- `crates/nexum-engine/src/runtime/restart_policy.rs`: `backoff_for(failure_count) -> Duration` with the 1s → 5min schedule. 4 unit tests covering the steady-state, first-failure, doubling, and cap arms.
- `Supervisor` gains four cached backends (`engine`, `cow_pool`, `provider_pool`, `local_store`) so `reinstantiate_one(idx)` can rebuild the wasi Linker + HostState + Store + bindings on demand.
- `LoadedModule` gains `component: Component`, `init_config: Config`, `http_allowlist: Vec<String>` (all cloned at boot), plus `failure_count: u32` and `next_attempt: Option<Instant>` for the schedule.
## Dispatch path changes
`dispatch_block` and `dispatch_log` now restructure into two
phases:
1. **Phase 1 (restart sweep)**: walk modules, collect indices of
dead-but-due modules, call `reinstantiate_one` on each. Failed
restarts bump the backoff again. Successful restarts flip
`alive = true` so phase 2 dispatches the next event to them.
2. **Phase 2 (steady-state dispatch)**: unchanged from before -
walk modules, dispatch where subscribed + alive. Trap path
sets `next_attempt` + bumps `failure_count`; success path
resets both.
The structured logs from COW-1035 gain `failure_count` + `backoff_ms`
on trap + `restart attempt` info lines on each restart. The
`shepherd_module_restarts_total{module}` Prometheus counter from
COW-1034 increments on every restart attempt.
## New fixture + integration test
`modules/fixtures/flaky-bomb/` (test-only): traps via OutOfFuel on
the first N events (N from `[config].fail_first_n`) and recovers
afterwards. Uses local-store for the attempt counter because the
wasm instance state resets on each reinstantiation; the counter
persists in the host-side store so the module deterministically
recovers after the configured N.
`supervisor::tests::restart_flaky_module_recovers_after_backoff`
(new): boots flaky-bomb with fail_first_n=1, dispatches, observes:
- Dispatch 1: trap. alive=false, failure_count=1, next_attempt=+1s.
- Immediate redispatch: skipped (still in backoff).
- Sleep 1.1s.
- Dispatch 3: restart fires, fresh instance attempts again. With
attempt=2 > N=1, returns Ok. alive=true, failure_count=0,
next_attempt=None.
- Dispatch 4: steady-state, dispatches normally.
Test wall-clock ~1.4s.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing.
+4 from `restart_policy` unit tests + 1 from the new integration
test (was 154 + 6).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing resource-limit tests (COW-1036) still pass against
the new dispatch shape: their assertions are against state
*immediately* after the trap (before backoff elapses), so the
restart machinery is transparent.
- The `init_failure_marks_module_dead_and_excludes_from_dispatch`
test (COW-1070) still passes: init-failed modules carry
`next_attempt = None` so the restart sweep never picks them up.
## Out of scope
- Persistence of `failure_count` / `next_attempt` across full
engine restarts. The schedule resets on every boot; cross-engine
persistence is a 0.3 follow-up.
- WS reconnect-with-backoff for upstream RPC drops - that is
COW-1071, a separate axis.
- Operator-tunable backoff via `engine.toml::[engine.restart]`.
The current constants are workspace literals in
`runtime::restart_policy`; configurable in 0.3.
- Module-side `on_restart` hook. Modules just see a fresh `init`
call after a restart, same as boot.
Linear: COW-1033. Fifth M4 issue landed; stacks on #38 (COW-1034).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…tiation (COW-1033)
When a module traps in `on_event` (OutOfFuel, MemoryOutOfBounds,
unhandled host error), the supervisor now:
1. Marks the module `alive = false` and increments `failure_count`.
2. Schedules a `next_attempt` instant via the new
`runtime::restart_policy::backoff_for` (1s → 2s → 4s → ... cap
5 min). All dispatches before that instant skip the module.
3. On the first dispatch past the backoff window, the supervisor
tears down the trapped wasmtime Store + component instance and
re-instantiates from the cached `Component`. The instance state
resets but host-side persistent state (local-store) survives
so a module's progress counters live across restarts.
4. On a successful `on_event` after recovery, `failure_count` resets
to 0 + `next_attempt = None`.
## Why the reinstantiation is required
A wasmtime trap leaves the component instance poisoned: subsequent
`call_on_event` returns "wasm trap: cannot enter component instance".
Just refueling the Store does not recover. The supervisor caches
the `Component`, `init_config`, and `http_allowlist` on
`LoadedModule` at boot so a restart only needs a fresh Store +
re-instantiation - the compiled component bytes are reused.
## New types / files
- `crates/nexum-engine/src/runtime/restart_policy.rs`: `backoff_for(failure_count) -> Duration` with the 1s → 5min schedule. 4 unit tests covering the steady-state, first-failure, doubling, and cap arms.
- `Supervisor` gains four cached backends (`engine`, `cow_pool`, `provider_pool`, `local_store`) so `reinstantiate_one(idx)` can rebuild the wasi Linker + HostState + Store + bindings on demand.
- `LoadedModule` gains `component: Component`, `init_config: Config`, `http_allowlist: Vec<String>` (all cloned at boot), plus `failure_count: u32` and `next_attempt: Option<Instant>` for the schedule.
## Dispatch path changes
`dispatch_block` and `dispatch_log` now restructure into two
phases:
1. **Phase 1 (restart sweep)**: walk modules, collect indices of
dead-but-due modules, call `reinstantiate_one` on each. Failed
restarts bump the backoff again. Successful restarts flip
`alive = true` so phase 2 dispatches the next event to them.
2. **Phase 2 (steady-state dispatch)**: unchanged from before -
walk modules, dispatch where subscribed + alive. Trap path
sets `next_attempt` + bumps `failure_count`; success path
resets both.
The structured logs from COW-1035 gain `failure_count` + `backoff_ms`
on trap + `restart attempt` info lines on each restart. The
`shepherd_module_restarts_total{module}` Prometheus counter from
COW-1034 increments on every restart attempt.
## New fixture + integration test
`modules/fixtures/flaky-bomb/` (test-only): traps via OutOfFuel on
the first N events (N from `[config].fail_first_n`) and recovers
afterwards. Uses local-store for the attempt counter because the
wasm instance state resets on each reinstantiation; the counter
persists in the host-side store so the module deterministically
recovers after the configured N.
`supervisor::tests::restart_flaky_module_recovers_after_backoff`
(new): boots flaky-bomb with fail_first_n=1, dispatches, observes:
- Dispatch 1: trap. alive=false, failure_count=1, next_attempt=+1s.
- Immediate redispatch: skipped (still in backoff).
- Sleep 1.1s.
- Dispatch 3: restart fires, fresh instance attempts again. With
attempt=2 > N=1, returns Ok. alive=true, failure_count=0,
next_attempt=None.
- Dispatch 4: steady-state, dispatches normally.
Test wall-clock ~1.4s.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing.
+4 from `restart_policy` unit tests + 1 from the new integration
test (was 154 + 6).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing resource-limit tests (COW-1036) still pass against
the new dispatch shape: their assertions are against state
*immediately* after the trap (before backoff elapses), so the
restart machinery is transparent.
- The `init_failure_marks_module_dead_and_excludes_from_dispatch`
test (COW-1070) still passes: init-failed modules carry
`next_attempt = None` so the restart sweep never picks them up.
## Out of scope
- Persistence of `failure_count` / `next_attempt` across full
engine restarts. The schedule resets on every boot; cross-engine
persistence is a 0.3 follow-up.
- WS reconnect-with-backoff for upstream RPC drops - that is
COW-1071, a separate axis.
- Operator-tunable backoff via `engine.toml::[engine.restart]`.
The current constants are workspace literals in
`runtime::restart_policy`; configurable in 0.3.
- Module-side `on_restart` hook. Modules just see a fresh `init`
call after a restart, same as boot.
Linear: COW-1033. Fifth M4 issue landed; stacks on #38 (COW-1034).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…tiation (COW-1033)
When a module traps in `on_event` (OutOfFuel, MemoryOutOfBounds,
unhandled host error), the supervisor now:
1. Marks the module `alive = false` and increments `failure_count`.
2. Schedules a `next_attempt` instant via the new
`runtime::restart_policy::backoff_for` (1s → 2s → 4s → ... cap
5 min). All dispatches before that instant skip the module.
3. On the first dispatch past the backoff window, the supervisor
tears down the trapped wasmtime Store + component instance and
re-instantiates from the cached `Component`. The instance state
resets but host-side persistent state (local-store) survives
so a module's progress counters live across restarts.
4. On a successful `on_event` after recovery, `failure_count` resets
to 0 + `next_attempt = None`.
## Why the reinstantiation is required
A wasmtime trap leaves the component instance poisoned: subsequent
`call_on_event` returns "wasm trap: cannot enter component instance".
Just refueling the Store does not recover. The supervisor caches
the `Component`, `init_config`, and `http_allowlist` on
`LoadedModule` at boot so a restart only needs a fresh Store +
re-instantiation - the compiled component bytes are reused.
## New types / files
- `crates/nexum-engine/src/runtime/restart_policy.rs`: `backoff_for(failure_count) -> Duration` with the 1s → 5min schedule. 4 unit tests covering the steady-state, first-failure, doubling, and cap arms.
- `Supervisor` gains four cached backends (`engine`, `cow_pool`, `provider_pool`, `local_store`) so `reinstantiate_one(idx)` can rebuild the wasi Linker + HostState + Store + bindings on demand.
- `LoadedModule` gains `component: Component`, `init_config: Config`, `http_allowlist: Vec<String>` (all cloned at boot), plus `failure_count: u32` and `next_attempt: Option<Instant>` for the schedule.
## Dispatch path changes
`dispatch_block` and `dispatch_log` now restructure into two
phases:
1. **Phase 1 (restart sweep)**: walk modules, collect indices of
dead-but-due modules, call `reinstantiate_one` on each. Failed
restarts bump the backoff again. Successful restarts flip
`alive = true` so phase 2 dispatches the next event to them.
2. **Phase 2 (steady-state dispatch)**: unchanged from before -
walk modules, dispatch where subscribed + alive. Trap path
sets `next_attempt` + bumps `failure_count`; success path
resets both.
The structured logs from COW-1035 gain `failure_count` + `backoff_ms`
on trap + `restart attempt` info lines on each restart. The
`shepherd_module_restarts_total{module}` Prometheus counter from
COW-1034 increments on every restart attempt.
## New fixture + integration test
`modules/fixtures/flaky-bomb/` (test-only): traps via OutOfFuel on
the first N events (N from `[config].fail_first_n`) and recovers
afterwards. Uses local-store for the attempt counter because the
wasm instance state resets on each reinstantiation; the counter
persists in the host-side store so the module deterministically
recovers after the configured N.
`supervisor::tests::restart_flaky_module_recovers_after_backoff`
(new): boots flaky-bomb with fail_first_n=1, dispatches, observes:
- Dispatch 1: trap. alive=false, failure_count=1, next_attempt=+1s.
- Immediate redispatch: skipped (still in backoff).
- Sleep 1.1s.
- Dispatch 3: restart fires, fresh instance attempts again. With
attempt=2 > N=1, returns Ok. alive=true, failure_count=0,
next_attempt=None.
- Dispatch 4: steady-state, dispatches normally.
Test wall-clock ~1.4s.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing.
+4 from `restart_policy` unit tests + 1 from the new integration
test (was 154 + 6).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing resource-limit tests (COW-1036) still pass against
the new dispatch shape: their assertions are against state
*immediately* after the trap (before backoff elapses), so the
restart machinery is transparent.
- The `init_failure_marks_module_dead_and_excludes_from_dispatch`
test (COW-1070) still passes: init-failed modules carry
`next_attempt = None` so the restart sweep never picks them up.
## Out of scope
- Persistence of `failure_count` / `next_attempt` across full
engine restarts. The schedule resets on every boot; cross-engine
persistence is a 0.3 follow-up.
- WS reconnect-with-backoff for upstream RPC drops - that is
COW-1071, a separate axis.
- Operator-tunable backoff via `engine.toml::[engine.restart]`.
The current constants are workspace literals in
`runtime::restart_policy`; configurable in 0.3.
- Module-side `on_restart` hook. Modules just see a fresh `init`
call after a restart, same as boot.
Linear: COW-1033. Fifth M4 issue landed; stacks on #38 (COW-1034).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Wires `metrics` + `metrics-exporter-prometheus` into the engine. When `engine.toml::[engine.metrics].enabled = true` the engine binds a Prometheus HTTP exporter on `bind_addr` (default `127.0.0.1:9100`) and serves `/metrics`. When disabled (the default), the recorder is still installed so call sites stay live but no port binds.
Fourth M4 issue landed.
Recording sites
Recording sites match the COW-1035 structured-logging audit so every metric event has a sibling log line with the same labels.
Live Sepolia /metrics scrape
```
TYPE shepherd_cow_api_submit_total counter
shepherd_cow_api_submit_total{chain_id="11155111",outcome="err"} 2
TYPE shepherd_chain_request_total counter
shepherd_chain_request_total{chain_id="11155111",method="eth_getBalance",outcome="ok"} 4
shepherd_chain_request_total{chain_id="11155111",method="eth_call",outcome="ok"} 4
TYPE shepherd_event_latency_seconds summary
shepherd_event_latency_seconds{module="price-alert", event_kind="block",quantile="0.5"} 0.1394
shepherd_event_latency_seconds{module="stop-loss", event_kind="block",quantile="0.5"} 0.9091
shepherd_event_latency_seconds{module="balance-tracker",event_kind="block",quantile="0.5"} 0.2823
```
p50/p90/p95/p99 quantiles flow automatically via default reservoir sampling — SRE can write SLO alerts without further wiring.
Config schema
```toml
[engine.metrics]
enabled = false # default
bind_addr = "127.0.0.1:9100"
```
Disabled by default so the M3 runbook smoke runs do not bind a port unintentionally. Production ops flip `enabled = true`.
Deferred from issue scope
The 4 metrics shipped here cover the load-bearing observability cases. The other 5 from the issue body need work that is not yet in M4:
Breaking changes
`engine.toml` parsing for the new `[engine.metrics]` section is backwards-compatible (the section is `default()` when absent).
Testing
AI assistance disclosure
AI Assistance: this change + description was produced by a Claude Code agent (Claude Opus 4.7 1M context). The agent designed the metric surface, instrumented the recording sites, ran the live Sepolia scrape, and authored this PR description. A human (Bruno) reviewed and is accountable for the result.
Linear: COW-1034. Fourth M4 issue landed; stacks on #37 (COW-1035 logging audit).