diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..230a92a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,49 @@ +# Changelog + +All notable changes to this project are documented in this file. + +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the +project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +## [0.1.0] — 2026-07-23 + +First tagged release: the complete vertical slice — synthetic tape → Rust engine → +SQLite metrics → FastAPI → trade journal + Claude behavioral agent → Next.js +dashboard — runnable end to end with one command (`make demo`). All bundled data is +synthetic; the agent produces behavioral observations, never trading advice or price +predictions. + +### Added + +- **Phase 0 — scaffolding**: monorepo layout (`engine/` Rust, `backtest/` Python, + `dashboard/` placeholder), rustfmt/clippy/ruff configuration, CI pipeline. +- **Phase 1 — vertical slice**: deterministic synthetic MNQ tick generator + (drifting buy-pressure regime); the engine parses the CSV tape and computes + normalized order-flow imbalance per epoch-aligned 1s window into SQLite; + FastAPI serves it at `GET /metrics/ofi`. +- **Phase 2 — more metrics**: realized volatility (continuous across window + boundaries), buy/sell/total volume and VWAP computed in the same windowing pass + into one unified `metrics` table; `GET /metrics/volatility` and + `GET /metrics/volume`. +- **Phase 3 — trade journal + coach**: `journal_entries` lives in the same SQLite + file so every entry joins to the microstructure regime at its entry time; full + journal CRUD over the API (422-validated, `since_ns`/`until_ns` filters); Claude + behavioral agent (`python -m backtest.coach`, `POST /journal/analyze`) with a + dependency-injected client so tests and CI need no API key. Hardening review on + top: engine rejects non-positive tick sizes at parse (a zero-size window made OFI + and VWAP `NaN`), `PATCH /journal/{id}` re-validates exit-after-entry against the + stored row, `anthropic` capped below the next major. +- **Phase 4 — dashboard**: Next.js 16 app with the four metric charts (OFI, + realized volatility, volume, VWAP; Recharts, no dual axes) and the journal + + behavioral-analysis views; `*_ns` nanosecond timestamps handled as exact `BigInt` + end to end; same-origin proxy to the API instead of CORS. +- **Phase 5 — release**: `scripts/demo.sh` boots the whole stack (engine → metrics + DB → journal seed → API → dashboard) on the sample data with one command; root + `Makefile` (`setup`/`demo`/`verify`/`build`/`test`/`lint`/`clean`); README quick + start with dashboard screenshots; this changelog; versions aligned at 0.1.0 + across engine, backtest and dashboard. + +[Unreleased]: https://github.com/1816x/Trading-Microstructure-Engine/compare/v0.1.0...HEAD +[0.1.0]: https://github.com/1816x/Trading-Microstructure-Engine/releases/tag/v0.1.0 diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..89feada --- /dev/null +++ b/Makefile @@ -0,0 +1,41 @@ +# Root shortcuts for the per-component commands documented in the README. +# Recipes mirror CI (.github/workflows/ci.yml) — no logic of their own. + +.PHONY: help setup demo verify build test lint clean + +help: ## list the available targets + @grep -E '^[a-z]+:.*## ' Makefile | awk -F':.*## ' '{printf " make %-7s %s\n", $$1, $$2}' + +setup: .venv ## install the Python venv (.venv) and dashboard node_modules + npm ci --prefix dashboard + +.venv: + python3 -m venv .venv + .venv/bin/pip install -e "backtest[dev]" + +demo: ## full-stack demo on the synthetic sample data (Ctrl-C stops it) + bash scripts/demo.sh + +verify: ## end-to-end pipeline smoke test (tape -> engine -> journal -> regime join) + bash scripts/verify_pipeline.sh + +build: ## release build of the engine + production build of the dashboard + cargo build --release --manifest-path engine/Cargo.toml + npm run build --prefix dashboard + +test: | .venv ## all three test suites, same as CI + cargo test --manifest-path engine/Cargo.toml + cd backtest && ../.venv/bin/pytest + npm test --prefix dashboard + +lint: | .venv ## all linters and format checks, same as CI + cargo fmt --check --manifest-path engine/Cargo.toml + cargo clippy --all-targets --manifest-path engine/Cargo.toml -- -D warnings + cd backtest && ../.venv/bin/ruff check . && ../.venv/bin/ruff format --check . + npm run lint --prefix dashboard + npm run typecheck --prefix dashboard + +clean: ## remove the generated DB, engine target/ and dashboard .next/ + rm -f metrics.db + cargo clean --manifest-path engine/Cargo.toml + rm -rf dashboard/.next diff --git a/README.md b/README.md index a204bc4..303452c 100644 --- a/README.md +++ b/README.md @@ -31,14 +31,40 @@ data-feed → [Rust: microstructure engine] → metrics (order-flow imbalance, r ## Status -Early scaffolding. Roadmap: +**v0.1.0** — the full vertical slice runs end to end, demo-able with one command. +Roadmap: - [x] Phase 0 — Scaffolding: monorepo layout, linters, CI. - [x] Phase 1 — Vertical slice: sample tick CSV → order-flow imbalance → SQLite → API endpoint. - [x] Phase 2 — More metrics (realized volatility, volume, VWAP), tests against known data. - [x] Phase 3 — Trade journal model + Claude API behavioral agent. - [x] Phase 4 — Next.js dashboard. -- [ ] Phase 5 — v0.1.0 release with sample-data demo. +- [x] Phase 5 — v0.1.0 release with sample-data demo. + +See [CHANGELOG.md](CHANGELOG.md) for what each release delivered. + +## Quick start + +Requires Rust (stable), Python ≥ 3.11 and Node ≥ 22. One command builds the engine, +computes the metrics over the bundled synthetic tape, seeds the trade journal and +serves both the API and the dashboard until Ctrl-C: + +```bash +make demo # or: bash scripts/demo.sh +``` + +Open http://localhost:3000 — the four microstructure charts: + +![Metrics view: order-flow imbalance, realized volatility, volume and VWAP charts over the sample tape](docs/images/dashboard-metrics.png) + +…and the trade journal, every entry joined to the market regime at its entry time, +with the Claude behavioral-analysis panel (that one call needs `ANTHROPIC_API_KEY`; +everything else works without it): + +![Journal view: trades joined to their regime, the log-a-trade form and the behavioral-analysis panel](docs/images/dashboard-journal.png) + +`make help` lists the rest of the shortcuts (`setup`, `verify`, `build`, `test`, +`lint`, `clean`); the sections below run each layer by hand. ## Running locally @@ -105,7 +131,7 @@ cd dashboard && npm install && npm run dev # open http://localhost:3000 — set BACKEND_URL if the API is not on localhost:8000 ``` -Run the checks the same way CI does: +Run the checks the same way CI does (`make lint` and `make test` wrap them): ```bash cd engine && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test @@ -117,7 +143,7 @@ Verify the whole Phase 1–3 pipeline end to end (tape → engine → metrics join; no API key needed): ```bash -bash scripts/verify_pipeline.sh +bash scripts/verify_pipeline.sh # or: make verify ``` ## Design decisions @@ -239,3 +265,23 @@ calls were made by me — as the project progresses. entered_at_ns` rule, server 422 details surfaced verbatim) and triggers the behavioral agent behind an explicit button — one Claude call per click, with the 503 no-key and 502 no-analysis states told apart. `PATCH`/`DELETE` stay API-only until the journal view earns editing. +- **One-command demo as plain bash, not docker-compose** (Phase 5, Claude's proposal, + accepted): `scripts/demo.sh` boots the stack in dependency order (tape → engine → + metrics DB → journal seed → API :8000 → dashboard :3000) with readiness polls, a port + pre-flight and a cleanup trap that tears both servers down on Ctrl-C — and drops + `next dev`'s generated-type scratch, which a mid-write kill can leave half-written + where `tsc` would trip over it. It reuses the + toolchain the project already requires; containerizing a three-language dev stack would + add a dependency without demoing anything better. Choices inside it: the dashboard runs + `next dev` (no build wait, identical charts); `METRICS_DB` is recreated on every run so + re-runs stay deterministic instead of duplicating journal seeds; `.venv`/`node_modules` + install only when missing, so the second boot takes seconds. +- **Screenshots captured from the running demo, committed to the repo** (Phase 5, + Claude's proposal, accepted): the README opens with `docs/images/dashboard-*.png`, + taken from this demo over the sample tape — a dashboard the README only describes is + one the reader takes on faith. ~600 KB of PNG in-repo is the accepted cost. +- **Root Makefile as an index, not a build system** (Phase 5, Claude's proposal, + accepted): the targets (`setup`/`demo`/`verify`/`build`/`test`/`lint`/`clean`) only + wrap the long per-component commands already documented in this README; CI keeps + invoking the underlying commands directly, so the Makefile cannot drift from what CI + actually runs. diff --git a/backtest/pyproject.toml b/backtest/pyproject.toml index 95ed492..0a893eb 100644 --- a/backtest/pyproject.toml +++ b/backtest/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "backtest" -version = "0.0.1" +version = "0.1.0" description = "Backtesting, aggregation and API layer on top of the microstructure engine metrics" requires-python = ">=3.11" dependencies = [ diff --git a/docs/images/dashboard-journal.png b/docs/images/dashboard-journal.png new file mode 100644 index 0000000..ad8907c Binary files /dev/null and b/docs/images/dashboard-journal.png differ diff --git a/docs/images/dashboard-metrics.png b/docs/images/dashboard-metrics.png new file mode 100644 index 0000000..a68ce44 Binary files /dev/null and b/docs/images/dashboard-metrics.png differ diff --git a/scripts/demo.sh b/scripts/demo.sh new file mode 100755 index 0000000..4b4e8bd --- /dev/null +++ b/scripts/demo.sh @@ -0,0 +1,145 @@ +#!/usr/bin/env bash +# One-command demo of the full stack on the synthetic sample data: +# +# tape -> Rust engine -> metrics.db -> journal seed -> FastAPI -> Next.js dashboard +# +# Regenerates the deterministic sample data, computes the microstructure metrics, +# seeds the trade journal, then serves the API and the dashboard until Ctrl-C. +# The first run installs .venv and dashboard/node_modules; later runs reuse them. +# The behavioral agent (POST /journal/analyze) needs ANTHROPIC_API_KEY — see +# .env.example; everything else in the demo works without it. +# +# Usage: bash scripts/demo.sh (or: make demo) +# PYTHON=... interpreter for the venv + generators (default python3) +# METRICS_DB=... SQLite path (default metrics.db; recreated on every run) +# API_PORT=... FastAPI port (default 8000) +# DASH_PORT=... dashboard port (default 3000) +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$ROOT" + +# Optional; carries ANTHROPIC_API_KEY (and friends) for the behavioral agent. +if [[ -f .env ]]; then + set -a + # shellcheck disable=SC1091 + . ./.env + set +a +fi + +PY="${PYTHON:-python3}" +DB="${METRICS_DB:-metrics.db}" +API_PORT="${API_PORT:-8000}" +DASH_PORT="${DASH_PORT:-3000}" + +for tool in "$PY" cargo npm curl; do + command -v "$tool" >/dev/null 2>&1 || { + echo "demo: '$tool' is required but not on PATH" >&2 + exit 1 + } +done + +for port in "$API_PORT" "$DASH_PORT"; do + if curl -s -o /dev/null --max-time 2 "http://localhost:$port"; then + echo "demo: port $port is already in use — is another demo still running?" >&2 + exit 1 + fi +done + +LOG_DIR="$(mktemp -d -t tme_demo.XXXXXX)" +API_PID="" +DASH_PID="" + +cleanup() { + trap - INT TERM EXIT + echo + echo "stopping the demo (server logs kept in $LOG_DIR)" + [[ -z "$DASH_PID" ]] || kill "$DASH_PID" 2>/dev/null || true + [[ -z "$API_PID" ]] || kill "$API_PID" 2>/dev/null || true + wait 2>/dev/null || true + # A kill landing while `next dev` writes its generated route types leaves + # them half-written, and dashboard/tsconfig.json includes them — drop the + # scratch (regenerated on every dev boot) so an interrupted demo can never + # break `npm run typecheck`. + rm -rf "$ROOT/dashboard/.next/dev/types" + exit 0 +} +trap cleanup INT TERM EXIT + +echo "1/6 generating the synthetic tape and journal (deterministic, all synthetic)" +"$PY" data/generate_ticks.py >/dev/null +"$PY" data/generate_journal.py >/dev/null + +echo "2/6 computing microstructure metrics into $DB (fresh on every run)" +rm -f "$DB" +cargo run --quiet --manifest-path engine/Cargo.toml --release -- \ + --input data/sample_mnq_ticks.csv --db "$DB" --window 1s + +echo "3/6 seeding the trade journal" +PYTHONPATH="$ROOT/backtest/src${PYTHONPATH:+:$PYTHONPATH}" "$PY" - "$DB" <<'PYCODE' +import sys + +from backtest import journal + +seeded = journal.import_csv("data/sample_journal.csv", sys.argv[1]) +print(f" seeded {seeded} journal entries") +PYCODE + +echo "4/6 preparing the Python env (.venv)" +if [[ ! -x .venv/bin/uvicorn ]]; then + "$PY" -m venv .venv + .venv/bin/pip install --quiet -e backtest +fi + +echo "5/6 starting the API on http://localhost:$API_PORT" +METRICS_DB="$DB" .venv/bin/uvicorn backtest.api:app --port "$API_PORT" \ + >"$LOG_DIR/api.log" 2>&1 & +API_PID=$! +for _ in $(seq 1 60); do + curl -sf "http://localhost:$API_PORT/metrics/ofi?limit=1" >/dev/null 2>&1 && break + if ! kill -0 "$API_PID" 2>/dev/null; then + echo "demo: the API exited during startup; last log lines:" >&2 + tail -n 20 "$LOG_DIR/api.log" >&2 + exit 1 + fi + sleep 0.5 +done +curl -sf "http://localhost:$API_PORT/metrics/ofi?limit=1" >/dev/null 2>&1 || { + echo "demo: the API did not answer within 30s; last log lines:" >&2 + tail -n 20 "$LOG_DIR/api.log" >&2 + exit 1 +} + +echo "6/6 starting the dashboard on http://localhost:$DASH_PORT" +if [[ ! -d dashboard/node_modules ]]; then + echo " installing dashboard dependencies (first run only)" + npm ci --prefix dashboard --no-audit --no-fund >"$LOG_DIR/npm.log" 2>&1 +fi +( + cd dashboard + BACKEND_URL="http://localhost:$API_PORT" \ + exec node_modules/.bin/next dev --port "$DASH_PORT" +) >"$LOG_DIR/dashboard.log" 2>&1 & +DASH_PID=$! +for _ in $(seq 1 120); do + curl -sf "http://localhost:$DASH_PORT" >/dev/null 2>&1 && break + if ! kill -0 "$DASH_PID" 2>/dev/null; then + echo "demo: the dashboard exited during startup; last log lines:" >&2 + tail -n 20 "$LOG_DIR/dashboard.log" >&2 + exit 1 + fi + sleep 0.5 +done +curl -sf "http://localhost:$DASH_PORT" >/dev/null 2>&1 || { + echo "demo: the dashboard did not answer within 60s; last log lines:" >&2 + tail -n 20 "$LOG_DIR/dashboard.log" >&2 + exit 1 +} + +echo +echo "demo up — open http://localhost:$DASH_PORT (API: http://localhost:$API_PORT)" +if [[ -z "${ANTHROPIC_API_KEY:-}" ]]; then + echo "note: ANTHROPIC_API_KEY is not set — everything works except behavioral analysis" +fi +echo "Ctrl-C stops both servers." +wait