FRIX — Friction-Realistic Intraday eXecution benchmark

One honest question: once you model execution friction the way it actually behaves — maker/taker fees, queue position, partial fills, slippage, adverse selection — does any "smart" method (deep RL or an LLM agent) actually beat a simple classical execution schedule?

Answer, on real BTC/USDT and ETH/USDT order-book data: no. Under honest friction, PPO, SAC, and an LLM-augmented agent never beat a simple classical schedule. On BTC they land above Almgren–Chriss (within noise — all p > 0.2, but never cheaper); on ETH, deep RL is significantly worse (PPO, p = 0.035) than a passive baseline. Execution cost is fee-dominated at ~5 bps. There is no free lunch in intraday execution for an individual — and FRIX is built to prove that reproducibly, rather than assert the opposite by accident.

The null result is the contribution: an honest, reproducible measuring stick that resists fabricated edges, on real public limit-order-book data.

---

TL;DR — final result (BTC/USDT)

Task: buy 2.0 BTC over a 5-minute window (300 × 1 s steps), minimizing implementation shortfall (IS). Evaluated on 87 held-out test windows of real BTC/USDT LOB data. Lower IS = better.

Performance — all methods:

Method	IS (bps)	fees	fill %	maker %
TWAP	5.13	5.00	100.0	0.0
VWAP	5.13	5.00	100.0	0.0
POV	5.34	5.00	100.0	0.0
Almgren–Chriss	5.05	5.00	100.0	0.0
Submit-and-leave	5.10	5.01	83.7	1.1
LLM (Claude meta-controller)	5.22	5.04	97.9	5.1
PPO (5 seeds)	5.67 ± 0.36	4.15	93.7	82.1
SAC (5 seeds)	5.56 ± 0.12	3.40	97.7	90.2

Significance — does any "smart" method beat the best classical baseline (Almgren–Chriss, 5.05 bps)? Paired bootstrap, 10 000 resamples, fixed seed; negative ΔIS = better.

Method	ΔIS vs A–C (bps)	95% CI	p	verdict
LLM	+0.17	[−0.93, +1.41]	0.80	not significant
PPO	+0.61	[−0.29, +1.79]	0.22	not significant
SAC	+0.51	[−0.33, +1.56]	0.30	not significant

No method beats the baseline — every "smart" method lands above Almgren–Chriss (higher cost), and while none of the gaps is statistically significant (every 95% CI straddles zero, every p > 0.2), the point estimates all trend the wrong way. The simple classical schedule is the cheapest method on BTC, full stop.

Full numbers are committed in results.csv; regenerate the figures with python plots.py.

Second asset: ETH/USDT — the null generalizes

Re-running the whole pipeline on a second asset (ETH/USDT, identical protocol, post-fix engine):

Method	IS (bps)	fees	fill %	maker %
Submit-and-leave	4.80	5.00	99.9	1.1
TWAP	4.90	5.00	100.0	0.0
VWAP	4.90	5.00	100.0	0.0
POV	4.99	5.00	100.0	0.0
Almgren–Chriss	5.09	5.00	100.0	0.0
PPO (5 seeds)	5.87 ± 1.15	4.65	100.0	52.9
SAC (5 seeds)	5.95 ± 0.74	3.74	100.0	88.6

On ETH the cheapest baseline is the passive Submit-and-leave (4.80 bps), and the deep-RL agents are not just tied but worse — and the paired-bootstrap test (vs the best baseline on this asset) makes it concrete:

Method	ΔIS vs best baseline (bps)	95% CI	p	verdict
PPO	+1.07 (worse)	[+0.07, +2.24]	0.035	significantly worse
SAC	+1.15 (worse)	[−0.67, +3.18]	0.228	worse, not significant

PPO is significantly worse than the simple passive baseline (p = 0.035), and both agents carry large seed variance (one SAC seed lands at 7.03 bps). The null doesn't merely survive a second asset — DRL's fragility becomes visible: the simple classical schedule is cheaper and far more stable.

Across both assets, no method beats the best classical baseline (Almgren–Chriss on BTC, Submit-and-leave on ETH); on ETH, deep RL is significantly worse. The conclusion generalizes — the "smart" methods add complexity without benefit, and sometimes cost.

Note: the LLM arm is evaluated on BTC only. The significance test references the best classical baseline per asset (Almgren–Chriss on BTC, Submit-and-leave on ETH), selected data-driven by lowest mean IS.

---

Why this exists

A lot of "RL/LLM beats execution baselines" results quietly depend on an over-optimistic simulator: passive limit orders that always fill, no adverse selection, no realistic queue, costless re-quoting. Relax those assumptions and the apparent edge tends to evaporate. FRIX is a single honest protocol to measure that — the same friction model, the same task, the same metrics — for classical / DRL / LLM side by side.

The honesty story (what the benchmark caught)

This is the point of the project, so it's worth stating plainly. During development the DRL agents "discovered" a ~2 bps edge by always posting passive. It was fake — the benchmark surfaced four distinct simulator artifacts, each of which I then fixed:

1. resting orders each claiming the full step volume (everything filled at once, no realistic queue contention);
2. a passive-fill model with no adverse selection (orders filled cheaply without the hidden cost of getting filled right before the price moves against you);
3. an accounting bug letting executed quantity exceed the parent order (fill % > 100%);
4. forced completion double-counting leftover passive inventory (a strategy could execute up to ~2× its order).

Fixing all four erased the edge. A training-length ablation (20k → 100k timesteps) then showed a flat plateau (IS unchanged within noise), confirming the null is not an undertraining artifact. The benchmark's whole job is to catch exactly these illusions — and here it caught four.

---

Friction model

What's modeled in the simulated venue (frix/core.py):

• Maker / taker fees (taker 5 bps, maker 1 bps).
• Book-walking slippage — large marketable orders eat into deeper levels.
• Queue position & partial fills for resting limit orders, consuming real per-step trade volume.
• Adverse selection — a passive fill is marked against the mid h steps later, so getting filled "cheaply" right before an adverse move is correctly penalized.
• Forced completion — unfilled passive inventory is completed at market at the horizon.
• Execution latency (optional, latency_steps, default 0) — decision at t applied at t + δ, observation as-of t.

Deliberate scope notes (honest):

• Funding is not modeled: over a 5-minute execution window it is negligible (funding is an 8-hour carry cost, ~0 at this horizon). It matters for holding a position, not for executing one.
• Latency is modeled as an optional uniform execution delay (latency_steps, default 0): an order decided at step t is applied to the book at t + δ while the agent's observation stays as-of t. At δ = 0 vs 1 the ranking is unchanged — it erodes timing edges without favoring any method.
• Two assets (BTC/USDT, ETH/USDT) over a bounded date range — this is a measuring-stick prototype, not a market-wide study.

---

Methods compared

• Classical baselines (frix/strategies.py): TWAP, VWAP, POV, Almgren–Chriss, Submit-and-leave.
• Deep RL (frix/rl/): PPO and SAC (Stable-Baselines3) over a Gymnasium execution environment, market and passive-limit actions, trained on a temporally-split train set and evaluated out-of-sample across 5 seeds.
• LLM-augmented (frix/llm_strategy.py): a per-window meta-controller — at each window start it reads a compact, look-ahead-free market summary (spread, imbalance, realized vol, remaining qty, steps left) and returns execution parameters (participation rate, passive ratio, urgency); a deterministic executor then follows them. One API call per window. Reproducible by construction: temperature=0 + an on-disk SHA-256 prompt cache (re-runs make no API calls and are deterministic). Falls back to a TWAP-equivalent policy on any failure.

Evaluation protocol

• Real public LOB data, reconstructed to a 1 s grid.
• Temporally coherent train/test split (no look-ahead).
• Metric: implementation shortfall in bps, plus fees, fill %, maker %.
• Multi-seed DRL (5 seeds, mean ± std) and a training-length ablation.
• Paired bootstrap significance test vs the best classical baseline (10k resamples, fixed seed): mean ΔIS, 95% CI, two-sided p-value.

---

Reproduce

pip install -r requirements.txt

# 1) Download free public BTC/USDT LOB data into ./data
python download.py

# 2) End-to-end smoke test (single 5-min window — sanity, not the benchmark)
python run_phase0.py

# 3) Train the DRL agents (PPO + SAC, 5 seeds)
python -m frix.rl.train --data-dir data --n-seeds 5 --adverse-horizon-steps 5

# 4) The benchmark: classical + DRL + LLM over 87 held-out windows + significance test
export ANTHROPIC_API_KEY=...        # PowerShell: $env:ANTHROPIC_API_KEY="..."
python -m frix.rl.evaluate --data-dir data --n-seeds 5 --adverse-horizon-steps 5 --llm

The headline baseline/DRL/LLM tables come from step 4 (evaluate.py, 87 windows). run_phase0.py (step 2) is a single-window end-to-end check, not the benchmark.

The LLM arm is optional — omit --llm to run classical + DRL only. The first --llm run makes ~87 cached API calls; subsequent runs are free and deterministic.

To run a second asset without clobbering the first, use a per-asset model directory:

python -m frix.rl.train    --data-dir data_eth --model-dir models_eth --n-seeds 5
python -m frix.rl.evaluate --data-dir data_eth --model-dir models_eth --n-seeds 5

Repository layout

frix/
  core.py          # order book, friction model, simulated venue (slippage, queue, adverse selection)
  strategies.py    # classical baselines (TWAP / VWAP / POV / Almgren-Chriss / Submit-and-leave)
  engine.py        # execution engine + result metrics (IS, fees, fill%, maker%)
  data_real.py     # Binance snapshot/diff reconstruction -> 1s grid
  data_tardis.py   # Tardis CSV loaders
  llm_strategy.py  # LLM meta-controller (cached, deterministic)
  rl/              # Gymnasium env + PPO/SAC training, evaluation, significance test
tests/             # offline unit tests (friction, fills, metrics)
download.py        # fetch free public LOB data
run_phase0.py      # end-to-end classical baselines
plots.py           # regenerate the figures from results
results.csv        # committed final results
figures/           # committed result figures

---

Takeaway

Under an honest friction model, execution is a near-solved, fee-dominated problem: clever DRL and LLM agents converge to — but do not beat — a simple classical schedule, and a significance test confirms the gaps are noise. The value of FRIX is the discipline: a protocol that turns "my agent beats the market" claims into testable, reproducible, falsifiable numbers, and that caught its own fabricated edges before reporting them.

License

MIT — see LICENSE.