feat(ops): orderbook EthFlow indexer baseline tool (COW-1084) - #57
Closed
brunota20 wants to merge 1 commit into
Closed
feat(ops): orderbook EthFlow indexer baseline tool (COW-1084)#57brunota20 wants to merge 1 commit into
brunota20 wants to merge 1 commit into
Conversation
…W-1084) Adds `tools/baseline-latency/baseline_latency.py`, a per-chain script that pairs every on-chain `EthFlow.OrderPlacement` event in a trailing window with the orderbook's record for the same UID and reports `(creationDate - block.timestamp)`. Matching is rigorous: the script ABI-decodes the GPv2OrderData from each event, computes the EIP-712 order digest against the chain's GPv2Settlement domain, and looks up the resulting UID against the orderbook's bulk `/account/.../orders` fetch (single-UID fallback if missed). No temporal-FIFO approximation. ## Headline finding For EthFlow orders the orderbook indexer sets `creationDate := block.timestamp` (not the indexer's ingest time), so the historical delta is structurally 0s on every chain. This is intentional back-fill-style behaviour, not a measurement bug. **Implication**: EthFlow indexer latency cannot be derived from historical orderbook data — the meaningful relayer-latency baseline lives on the TWAP lane (where the orderbook records the indexer's `now()` per child order PUT). TWAP child-latency is a follow-up; it needs per-part UID derivation from each parent `ConditionalOrderCreated` static input. Sepolia ran clean: 256 events scanned, 200 UID-derived pairs, all 200 matched against the bulk fetch (`bulk_hit=200`). Median = p95 = 0.0s, exactly as the finding predicts. ## Mainnet/Gnosis/Arbitrum/Base = RPC-LIMITED Public-tier RPCs (drpc.org free, 1rpc.io, ankr w/o key, llamarpc, cloudflare-eth) all refuse / throttle `eth_getLogs` at any usable chunk size on the production chains. The script halves down to 50-block chunks and gives up after 3 consecutive failures, marking the cell `RPC-LIMITED` with a pointer to the `RPC_URL_*` env override. This is the same constraint the M5 soak (COW-1031) will face and independently confirms the paid-endpoint requirement for any serious log-scanning workload. ## Files - `tools/baseline-latency/baseline_latency.py` (~520 lines): argparse CLI, per-chain `Chain` dataclass, JSON-RPC helper with halving retry + `RpcLimited` sentinel, EIP-712 order digest + UID derivation, UID-keyed orderbook matching, markdown report renderer. - `tools/baseline-latency/data/*.json`: per-chain raw dump (events, pairs, deltas, diagnostics) for auditability. - `docs/operations/baselines/baseline-latency-2026-06-19.md`: the first run's report. ## Why this matters for the grant Pinning the orderbook's `creationDate` semantics matters because the COW-1079 and COW-1031 KPIs reference "watchtower latency" — the M4 report needs to be honest about which lane the latency lives on (TWAP relayer PUT, not EthFlow indexer ingest). The Sepolia data set also gives the M4 e2e harness ground-truth UID ↔ block pairings to cross-check against. AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end and validated against live Sepolia data by the operator.
This was referenced Jun 22, 2026
Author
This was referenced Jun 24, 2026
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
brunota20
added a commit
that referenced
this pull request
Jun 26, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
jean-neiverth
pushed a commit
that referenced
this pull request
Jun 29, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
jean-neiverth
pushed a commit
that referenced
this pull request
Jun 30, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
jean-neiverth
pushed a commit
that referenced
this pull request
Jun 30, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
jean-neiverth
pushed a commit
that referenced
this pull request
Jun 30, 2026
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tools/baseline-latency/baseline_latency.py— per-chain script that pairs every on-chainEthFlow.OrderPlacementevent with the orderbook's record for the same UID and reports the(creationDate - block.timestamp)delta. Matching is rigorous: ABI-decodes the GPv2OrderData from each event, derives the EIP-712 OrderUid against the chain's GPv2Settlement domain, looks up against the bulk/account/.../ordersfetch with/orders/{uid}fallback. No temporal-FIFO approximation.creationDate := block.timestampfor EthFlow orders, so the historical delta is structurally 0s on every chain. This is intentional back-fill behaviour, not a measurement bug — but it means EthFlow indexer latency cannot be derived from historical data. The meaningful relayer-latency baseline lives on the TWAP lane (where the orderbook records the indexer'snow()per child PUT). TWAP child-latency tracked as a follow-up.bulk_hit=200). Median = p95 = 0.0s exactly as the finding predicts. Mainnet/Gnosis/Arbitrum/Base hit public-tier RPC limits (drpc.orgfree,1rpc.io,ankrw/o key all refuseeth_getLogs); script halves to 50-block chunks then gives up after 3 consecutive failures, flagging the cellRPC-LIMITED. Independently confirms the paid-endpoint requirement COW-1031 has flagged.Why this matters for the grant
Pinning the orderbook's
creationDatesemantics matters because the COW-1079 + COW-1031 KPIs reference "watchtower latency" — the M4 report needs to be honest about which lane the latency lives on (TWAP relayer PUT, not EthFlow indexer ingest). The Sepolia data set also gives the M4 e2e harness ground-truth UID ↔ block pairings to cross-check against.Files
tools/baseline-latency/baseline_latency.py(~520 lines)tools/baseline-latency/data/*.json(per-chain raw dumps for auditability)docs/operations/baselines/baseline-latency-2026-06-19.md(first run report)Test plan
*.drpc.org(256 events → 200 pairs, allbulk_hit)--window-days 7 --max-events-per-chain 200) — completes in ~2 min total wall-clock with stuck chains short-circuitedStacked on
PR #53 (
feat/load-gen-calibration-cow-1080) → upstream M4 epic (TBD).AI-assistance disclosure
Authored with Claude (Opus 4.7); reviewed end-to-end and validated against live Sepolia + public RPCs by the operator before push.