Trading Microstructure Engine + AI Coach

A tick-by-tick futures data engine (MNQ and similar) that computes market microstructure metrics — order-flow imbalance, realized volatility, volume imbalance — with its own backtesting layer, plus a Claude-powered agent that reads your trade journal and surfaces behavioral patterns and emotional biases correlated with the market regime at the time.

Disclaimer: this is not financial advice and it does not predict prices. It is a tool for analyzing your own trading behavior against microstructure context.

Architecture

data-feed → [Rust: microstructure engine] → metrics (order-flow imbalance, realized volatility)
                                                       ↓
                                          [Python: backtesting + aggregation]
                                                       ↓
                                            trade journal (SQLite/Postgres)
                                                       ↓
                              [Claude API: behavioral analysis agent]
                                                       ↓
                                       [Next.js: dashboard + reports]

Layer	Language	Why
engine/	Rust	Low-latency processing of high-frequency tick streams is exactly what Rust is for.
backtest/	Python	Statistical metrics, aggregation and the API that feeds the agent.
dashboard/	TypeScript (Next.js)	Metric charts and the trade journal view.
Agent	Claude API	Behavioral observations from the journal — not price predictions.

Status

v0.1.0 — the full vertical slice runs end to end, demo-able with one command. Roadmap:

• Phase 0 — Scaffolding: monorepo layout, linters, CI.
• Phase 1 — Vertical slice: sample tick CSV → order-flow imbalance → SQLite → API endpoint.
• Phase 2 — More metrics (realized volatility, volume, VWAP), tests against known data.
• Phase 3 — Trade journal model + Claude API behavioral agent.
• Phase 4 — Next.js dashboard.
• Phase 5 — v0.1.0 release with sample-data demo.

See CHANGELOG.md for what each release delivered.

Quick start

Requires Rust (stable), Python ≥ 3.11 and Node ≥ 22. One command builds the engine, computes the metrics over the bundled synthetic tape, seeds the trade journal and serves both the API and the dashboard until Ctrl-C:

make demo        # or: bash scripts/demo.sh

Open http://localhost:3000 — the four microstructure charts:

…and the trade journal, every entry joined to the market regime at its entry time, with the Claude behavioral-analysis panel (that one call needs ANTHROPIC_API_KEY; everything else works without it):

make help lists the rest of the shortcuts (setup, verify, build, test, lint, clean); the sections below run each layer by hand.

Running locally

Requires Rust (stable), Python ≥ 3.11 and Node ≥ 22 (for the dashboard). All bundled data is synthetic — never real trades or amounts.

# 1. (Optional) regenerate the sample tape — deterministic, seeded
python3 data/generate_ticks.py

# 2. Compute microstructure metrics per 1s window into metrics.db
cargo run --manifest-path engine/Cargo.toml --release -- \
  --input data/sample_mnq_ticks.csv --db metrics.db --window 1s

# 3. Serve the metrics
python3 -m venv .venv && .venv/bin/pip install -e "backtest[dev]"
METRICS_DB=metrics.db .venv/bin/uvicorn backtest.api:app

# 4. Query them — one table, three views
curl 'localhost:8000/metrics/ofi?limit=5'         # order-flow imbalance
curl 'localhost:8000/metrics/volatility?limit=5'  # realized volatility + trade count
curl 'localhost:8000/metrics/volume?limit=5'      # buy/sell/total volume + VWAP

Phase 3 adds a trade journal (stored alongside the metrics) and a Claude-powered behavioral agent that reads it. The agent needs an Anthropic API key; it produces behavioral observations correlated with the market regime — not trading advice or price predictions.

# 5. Generate a synthetic trade journal (deterministic; all synthetic)
python3 data/generate_journal.py                  # writes data/sample_journal.csv

# 6. Analyze behavior with Claude (defaults to claude-opus-4-8;
#    override with ANTHROPIC_MODEL). --load seeds the journal into the DB first.
export ANTHROPIC_API_KEY=sk-ant-...
python3 -m backtest.coach --load data/sample_journal.csv --db metrics.db

# 7. Or drive the journal over the API (shares the same METRICS_DB)
curl -X POST localhost:8000/journal -H 'content-type: application/json' \
  -d '{"symbol":"MNQ","side":"long","entered_at_ns":1760000030000000000,
       "entry_price":21400,"size":1,"notes":"followed my plan","emotion":"calm"}'
curl 'localhost:8000/journal?limit=5'                 # entries joined to the market regime
curl 'localhost:8000/journal?since_ns=...&until_ns=...' # bound by entry time (since incl., until excl.)
curl 'localhost:8000/journal/1'                       # one entry by id (404 if absent)
curl -X PATCH localhost:8000/journal/1 -H 'content-type: application/json' \
  -d '{"emotion":"disciplined"}'                       # partial update
curl -X DELETE 'localhost:8000/journal/1'             # 204 on success, 404 if absent
curl -X POST 'localhost:8000/journal/analyze'         # Claude behavioral analysis (503 without a key)

side must be long/short, size/entry_price/entered_at_ns must be positive, and exited_at_ns (if given) must be ≥ entered_at_ns — on PATCH that ordering is checked against the stored entry too, so updating just one timestamp can't leave the exit before the entry. Invalid bodies return 422. If Claude declines or returns no structured analysis, /journal/analyze returns 502.

Phase 4 puts a dashboard on top: metric charts (order-flow imbalance, realized volatility, volume, VWAP) and the journal + behavioral-analysis view.

# 8. Dashboard (Next.js) — proxies /api/backend/* to the API from step 3
cd dashboard && npm install && npm run dev
# open http://localhost:3000 — set BACKEND_URL if the API is not on localhost:8000

Run the checks the same way CI does (make lint and make test wrap them):

cd engine && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test
cd backtest && ruff check . && ruff format --check . && pytest
cd dashboard && npm run lint && npm run typecheck && npm test && npm run build

Verify the whole Phase 1–3 pipeline end to end (tape → engine → metrics → journal → regime join; no API key needed):

bash scripts/verify_pipeline.sh    # or: make verify

Design decisions

This project is built in deliberate collaboration with Claude. This section documents who decided what — what the AI proposed, what got corrected or rejected, and which calls were made by me — as the project progresses.

• Rust over Go for the engine (mine): the reference ecosystem for high-performance trading in this space is Rust (e.g. barter-rs), and the tick-processing hot path is where the language choice actually matters.
• Dashboard scaffold deferred to Phase 4 (Claude's proposal, accepted): running create-next-app during scaffolding would bury the early history under thousands of generated lines; a placeholder folder keeps Phase 0 reviewable.
• Order-flow imbalance first (mine): of the planned metrics it is the one that most depends on true tick data (aggressor side per trade) instead of OHLC candles, so it proves the pipeline earns its keep before anything else gets built.
• OFI defined as (buy − sell) / (buy + sell) per epoch-aligned window (Claude's proposal, accepted): the normalized form is bounded in [-1, 1], which makes windows comparable across volume regimes. Buckets accumulate in an ordered map so an out-of-order tape still produces sorted output.
• FastAPI serves the metric in Phase 1 (Claude's proposal, accepted): the endpoint belongs to the Python layer that will own backtesting and aggregation anyway; giving the Rust engine an HTTP server would couple the hot path to serving concerns.
• Synthetic tape with a drifting buy-pressure regime (Claude's proposal, corrected by me): a naive uniform-random tape makes OFI hover near zero everywhere, which hides bugs — sign errors produce equally plausible noise. The generator drifts its buy-pressure between 0.35 and 0.65 so imbalance windows show persistent, verifiable signal.
• One unified metrics table instead of a table per metric (Phase 2, Claude's proposal, accepted): every metric shares the same (bucket_start_ns, window_ns) key and is produced in a single windowing pass, so a wide row is the natural shape. The Python API projects the columns each endpoint needs (/metrics/ofi, /metrics/volatility, /metrics/volume) out of that one table; the Phase 1 /metrics/ofi contract is unchanged.
• Realized volatility is continuous across window boundaries (Phase 2, mine): a window's realized variance includes the log return from the previous window's last trade, so summing per-window variances recovers the whole tape's realized variance and a window with a single tick still carries the volatility of its move. The alternative — resetting returns at each window edge — silently discards the boundary moves and understates volatility.
• VWAP folded into the same pass (Phase 2, Claude's proposal, accepted): it needs only the running Σ price·size the engine already touches per tick, so it comes almost for free and gives the dashboard a price anchor alongside the volume figures.
• The trade journal is Python-owned and lives in the same database (Phase 3, mine): unlike metrics, the journal is written and read by the Python + agent layer, not the Rust hot path, so Python owns the journal_entries schema — a deliberate inversion of the Phase 1 "the engine owns the schema" rule. Keeping it in the same SQLite file lets each entry join to the microstructure regime at its entry time (GET /journal and the agent both use that join), which is the whole point of the feature. SQLite only, no ORM, to stay consistent with the rest of the stack; Postgres in the diagram stays a future option, not a Phase 3 dependency.
• Behavioral agent defaults to claude-opus-4-8 (Phase 3, mine): overridable via the ANTHROPIC_MODEL env var. It emits behavioral observations tied to the regime — by design not trading advice and not price predictions. The Anthropic client is dependency-injected, so the analysis is unit-tested against a mock and CI needs neither a key nor network access.
• Phase 3 polish: validation, CRUD, refusal handling, time filters (Phase 3, mine): journal entries are validated at the API boundary (side ∈ long/short, positive size/price/timestamps, exited_at_ns ≥ entered_at_ns) → 422 on bad input. The journal is a full resource (GET/PATCH/DELETE /journal/{id}) with since_ns/until_ns filters for scoping analysis to a session. Because messages.parse().parsed_output is Optional (it is None on a safety refusal or an empty structured parse), the agent raises AnalysisUnavailable and the endpoint maps it to a 502 rather than surfacing a bare null. scripts/verify_pipeline.sh exercises the whole tape → engine → journal → regime-join chain without an API key.
• Nanosecond timestamps deferred as a Phase 4 JSON concern (Phase 3, mine): the API returns every *_ns field as a JSON integer, and these epochs (~1.76e18) exceed JavaScript's safe-integer range (2^53 ≈ 9e15). Rather than change the Phase 1–2 metrics contract now, the Phase 4 dashboard will read them as strings/BigInt; documented in the backtest.api module docstring so it is not forgotten.
• Phase 0–3 hardening review before starting Phase 4 (mine): rather than build the dashboard on top of whatever was already merged, I had Claude review the finished Phases 0–3 for correctness and form. The baseline was green (Rust + Python suites, lint, end-to-end pipeline); the fixes it proposed and I accepted were:
  • Zero-size ticks rejected at parse (Claude's proposal, accepted): a window whose ticks all had size == 0 gave a zero denominator, making OFI and VWAP NaN and poisoning the SQLite write. A serde field validator now rejects size == 0 when the tape is parsed (with row context), so MetricsBucket::ofi's "denominator is never zero" is actually true. The synthetic generator never emits 0, so this only ever bit hand-crafted tapes — a latent hole, closed at the boundary rather than checked per metric.
  • PATCH time-ordering checked against the stored row (Claude's proposal, accepted): the API-boundary validator only caught exited_at_ns < entered_at_ns when both fields arrived together, so a partial update of just one timestamp could persist an exit before its entry. journal.update_entry now merges the update with the stored row before writing and raises InvalidEntryUpdate; the endpoint maps it to 422 — one status code for one domain rule (409 was considered and rejected for that reason) — keeping the pydantic both-fields check as a fast path.
  • Journal INSERT shared, created_at_ns stamped per row (Claude's proposal, accepted): insert_entry and import_csv now build the INSERT from one _INSERT_SQL constant, and the bulk import stamps each row at its own insertion time instead of sharing one batch timestamp, matching the single-insert path. Form only; behavior otherwise unchanged.
  • anthropic capped below the next major (mine): >=0.117,<1, because the coach rides recent SDK surface (messages.parse(output_format=...), parsed_output, adaptive thinking) that a 1.0 could break; fastapi/uvicorn stay lower-bounded so CI keeps exercising latest. A full uv.lock was considered and deferred (it would mean migrating CI off pip); the stale uv.lock entry in .gitattributes, for a tool the project doesn't use, was removed.
• Recharts for the dashboard charts (Phase 4, mine): chosen over TradingView's lightweight-charts (canvas, candlestick-first, imperative) and visx (lower-level than this needs). The bucketed metrics are at most a few thousand SVG points, and declarative React components keep the four charts maintainable. One rule shaped them: no dual axes — VWAP (a price) gets its own frame instead of riding a second y-scale on the volume chart, and the buy↔sell diverging pair was machine-validated for CVD separation and contrast against the dark surface.
• Same-origin proxy instead of CORS (Phase 4, Claude's proposal, accepted): the dashboard rewrites /api/backend/* to the backend (BACKEND_URL, default localhost:8000), so the browser never makes a cross-origin request and the FastAPI layer ships exactly as Phase 1–3 left it — no middleware, no origin allowlist to maintain.
• *_ns timestamps as BigInt end-to-end (Phase 4, Claude's proposal, accepted): response bodies are parsed with the ES2025 reviver source access (context.source) so *_ns integers materialize as exact BigInts, and request bodies serialize them back as unquoted integers via JSON.rawJSON — the pydantic models expect int, so quoting them as strings was not an option. Nanoseconds are truncated to milliseconds only for chart axes. This closes the Phase 3 note that deferred the > 2^53 JSON-integer problem to the dashboard; the parser fails loudly on runtimes without source access (Node ≥ 22, evergreen browsers) rather than silently rounding.
• Journal UI scope: read + create + analyze (Phase 4, mine): the dashboard lists entries with their regime join, logs new trades (client-side mirror of the exited_at_ns ≥ entered_at_ns rule, server 422 details surfaced verbatim) and triggers the behavioral agent behind an explicit button — one Claude call per click, with the 503 no-key and 502 no-analysis states told apart. PATCH/DELETE stay API-only until the journal view earns editing.
• One-command demo as plain bash, not docker-compose (Phase 5, Claude's proposal, accepted): scripts/demo.sh boots the stack in dependency order (tape → engine → metrics DB → journal seed → API :8000 → dashboard :3000) with readiness polls, a port pre-flight and a cleanup trap that tears both servers down on Ctrl-C — and drops next dev's generated-type scratch, which a mid-write kill can leave half-written where tsc would trip over it. It reuses the toolchain the project already requires; containerizing a three-language dev stack would add a dependency without demoing anything better. Choices inside it: the dashboard runs next dev (no build wait, identical charts); METRICS_DB is recreated on every run so re-runs stay deterministic instead of duplicating journal seeds; .venv/node_modules install only when missing, so the second boot takes seconds.
• Screenshots captured from the running demo, committed to the repo (Phase 5, Claude's proposal, accepted): the README opens with docs/images/dashboard-*.png, taken from this demo over the sample tape — a dashboard the README only describes is one the reader takes on faith. ~600 KB of PNG in-repo is the accepted cost.
• Root Makefile as an index, not a build system (Phase 5, Claude's proposal, accepted): the targets (setup/demo/verify/build/test/lint/clean) only wrap the long per-component commands already documented in this README; CI keeps invoking the underlying commands directly, so the Makefile cannot drift from what CI actually runs.