Skip to content

[help wanted] Benchmark harness proving the 10x-vs-JS claim (criterion + Node baseline) #9

@abhicris

Description

@abhicris

Context

The repo tagline claims 10x faster than JS solvers. We need a reproducible benchmark harness to back that up — it becomes a marketing artifact and a regression tripwire. criterion is the canonical Rust benchmarking crate; we can compare the Rust Across decoder against the JS reference implementation (@across-protocol/sdk) on the same fixture set.

Scope

Medium-effort ticket.

  • Add criterion = "0.5" as a dev-dep.
  • Create benches/intent_decoding.rs that, for each decoder, benchmarks:
    • Parsing N=100 fixtures from tests/fixtures/.
    • Cold-path (first parse) vs warm-path (steady-state).
  • In the same PR, add benches/compare_js/ — a tiny Node script (npm i @across-protocol/sdk pinned) that decodes the same fixtures and emits ns/op.
  • benches/README.md documents: cargo bench for Rust, node benches/compare_js/index.js for JS. Include a generated table of the last-measured numbers.
  • CI job .github/workflows/bench.yml — runs cargo bench --no-run to ensure bench code at least compiles on every PR. Actual perf numbers run on-demand, not per PR.

Acceptance criteria

  • cargo bench runs successfully locally.
  • JS comparison script runs and produces a number in the same units (ns/op or μs/op).
  • benches/README.md shows a table with at least one data point per decoder and the 10x ratio (or an honest correction if it's not 10x).
  • CI job green.

Reference: resolver#1 is the existing Benchmark UniswapX/Across/CoW fill-latency issue — this ticket is complementary (decoder latency, not fill latency). PR #3 shows the Across decoder structure under test.

Estimated effort

M (1–2 days) — bulk of time is setting up the JS side reproducibly.

kcolbchain / Abhishek Krishna

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions