diff --git a/planning/MARKET_INTERFACE.md b/planning/MARKET_INTERFACE.md new file mode 100644 index 00000000..4d4c4bb5 --- /dev/null +++ b/planning/MARKET_INTERFACE.md @@ -0,0 +1,132 @@ +# Market Data Interface — Unified Design + +Defines the single abstract interface (`MarketDataSource`, per PLAN.md §6) that both the Massive API client and the built-in simulator implement, so that price-cache and SSE code (and everything downstream) is agnostic to the source. Backed by research in `MASSIVE_API.md`; simulator internals are detailed in `MARKET_SIMULATOR.md`. + +## 1. Selection Rule + +```python +def build_market_data_source() -> "MarketDataSource": + if os.environ.get("MASSIVE_API_KEY"): + return MassiveMarketDataSource(api_key=os.environ["MASSIVE_API_KEY"]) + return SimulatorMarketDataSource() +``` + +- `MASSIVE_API_KEY` set and non-empty → `MassiveMarketDataSource` +- Absent/empty → `SimulatorMarketDataSource` (default, no external dependency) + +This selection happens once at process startup; the chosen source runs as the single background task described in PLAN.md §6 ("Shared Price Cache"). + +## 2. Shared Data Model + +```python +from dataclasses import dataclass + +@dataclass(frozen=True) +class PriceTick: + ticker: str + price: float # latest trade price + previous_close: float # prior session close, for daily % change + timestamp: float # unix seconds (epoch), UTC +``` + +Both implementations produce `PriceTick` objects in this exact shape — this is the contract the price cache and SSE stream depend on. Neither the simulator's GBM internals nor Massive's snapshot JSON shape leak past this boundary. + +## 3. Abstract Interface + +```python +from abc import ABC, abstractmethod +from typing import Iterable + +class MarketDataSource(ABC): + @abstractmethod + async def start(self, tickers: Iterable[str]) -> None: + """Begin producing price updates for the given tickers (initial watchlist).""" + + @abstractmethod + async def stop(self) -> None: + """Stop the background update loop and release any resources.""" + + @abstractmethod + def watch(self, ticker: str) -> None: + """Add a ticker to the set of symbols being updated (e.g. added to watchlist, or a position opened on a delisted-from-watchlist ticker).""" + + @abstractmethod + def unwatch(self, ticker: str) -> None: + """Stop updating a ticker once nothing references it (removed from watchlist and no open position).""" + + @abstractmethod + def latest(self, ticker: str) -> PriceTick | None: + """Return the most recent PriceTick for a ticker, or None if not yet available.""" + + @abstractmethod + def subscribe(self) -> "AsyncIterator[PriceTick]": + """Async stream of PriceTick updates as they occur, for the SSE endpoint to consume.""" +``` + +Both `MassiveMarketDataSource` and `SimulatorMarketDataSource` (§4/§5 below) implement this exact interface. + +## 4. `MassiveMarketDataSource` + +- `start()` launches an `asyncio` polling loop calling the Massive **Full Market Snapshot** endpoint (`MASSIVE_API.md` §3.1) once per tick with the comma-joined set of currently-watched tickers. +- Poll interval: env-configurable (`MASSIVE_POLL_INTERVAL_SECONDS`, default `15` to respect the free-tier 5 req/min limit; lower for paid tiers). +- Each response is parsed into one `PriceTick` per ticker: `price = lastTrade.p`, `previous_close = prevDay.c`, `timestamp = updated / 1e9` (Massive returns nanoseconds). +- `watch()`/`unwatch()` mutate the in-memory ticker set consulted on the next tick — no immediate network call, to stay within rate limits. (Exception: consider an immediate single-ticker snapshot call, `MASSIVE_API.md` §3.2, when a brand-new ticker is added, so its first price isn't delayed a full poll interval — flagged here as a possible enhancement, not required for v1.) +- On HTTP error or 429: log and skip the tick, keep serving the last cached `PriceTick` per ticker (never fall back to the simulator mid-session — see `MASSIVE_API.md` §4). +- Unknown/invalid tickers: if a requested ticker is absent from the response, mark it `None` in `latest()` rather than raising. + +```python +class MassiveMarketDataSource(MarketDataSource): + def __init__(self, api_key: str, poll_interval: float = 15.0): + self._api_key = api_key + self._poll_interval = poll_interval + self._tickers: set[str] = set() + self._cache: dict[str, PriceTick] = {} + self._subscribers: list[asyncio.Queue[PriceTick]] = [] + self._task: asyncio.Task | None = None + + async def start(self, tickers): + self._tickers = set(tickers) + self._task = asyncio.create_task(self._poll_loop()) + + async def _poll_loop(self): + async with httpx.AsyncClient(base_url="https://api.massive.com") as client: + while True: + if self._tickers: + await self._poll_once(client) + await asyncio.sleep(self._poll_interval) + + async def _poll_once(self, client): + try: + resp = await client.get( + "/v2/snapshot/locale/us/markets/stocks/tickers", + params={"tickers": ",".join(sorted(self._tickers))}, + headers={"Authorization": f"Bearer {self._api_key}"}, + ) + resp.raise_for_status() + except httpx.HTTPError: + return # skip this tick, keep last known prices + for entry in resp.json().get("tickers", []): + tick = PriceTick( + ticker=entry["ticker"], + price=entry["lastTrade"]["p"], + previous_close=entry["prevDay"]["c"], + timestamp=entry["updated"] / 1e9, + ) + self._cache[tick.ticker] = tick + self._publish(tick) +``` + +## 5. `SimulatorMarketDataSource` + +- `start()` launches an `asyncio` loop ticking every ~500ms (PLAN.md §6), generating `PriceTick`s via GBM per `MARKET_SIMULATOR.md`. +- `watch()`/`unwatch()` add/remove a ticker from the simulated set immediately, seeding new tickers from `SEED_PRICES` (or a synthesized default if unrecognized — see `MARKET_SIMULATOR.md` §5). +- No network calls, no rate limits, no failure mode to handle. + +## 6. Price Cache & SSE Integration + +- Whichever `MarketDataSource` is active, the same cache/SSE code in the backend calls `latest()` for REST responses (`/api/watchlist`, `/api/portfolio`) and `subscribe()` to fan out `PriceTick`s to connected `/api/stream/prices` clients. +- SSE event payload is `PriceTick` serialized to JSON plus a derived `direction` field (`"up" | "down" | "flat"`, computed by comparing to the previously cached price before overwriting). + +## 7. Watchlist / Position Lifecycle Hooks + +Per the open question in `planning/REVIEW.md` about deleting a watched ticker with an open position: the backend calls `source.watch(ticker)` whenever a ticker is added to the watchlist **or** a position is opened, and `source.unwatch(ticker)` only when a ticker is both off the watchlist **and** has zero position quantity. This keeps the `MarketDataSource` implementations simple (they just track a flat ticker set) while the lifecycle policy lives in the backend layer that calls them. diff --git a/planning/MARKET_SIMULATOR.md b/planning/MARKET_SIMULATOR.md new file mode 100644 index 00000000..9644dafa --- /dev/null +++ b/planning/MARKET_SIMULATOR.md @@ -0,0 +1,130 @@ +# Market Simulator — Design + +Design for `SimulatorMarketDataSource` (PLAN.md §6, `MARKET_INTERFACE.md` §5), the default no-API-key market data source. Produces plausible, continuously-updating stock prices without any external dependency. + +## 1. Goals + +- Look and feel like a real, moving market: correlated sector moves, occasional volatility spikes, no external calls. +- Deterministic enough to unit test (seedable RNG), realistic enough for a convincing demo. +- Conform exactly to the `MarketDataSource` interface (`MARKET_INTERFACE.md` §3) so the rest of the system can't tell it apart from the Massive-backed source. + +## 2. Price Model — Geometric Brownian Motion (GBM) + +Each ticker's price evolves per discrete-time GBM: + +``` +S(t + dt) = S(t) * exp((mu - 0.5 * sigma^2) * dt + sigma * sqrt(dt) * Z) +``` + +- `S(t)`: current price +- `mu`: drift (annualized), per-ticker, small (e.g. `0.05` = 5%/yr) so prices don't trend away over a demo session +- `sigma`: annualized volatility, per-ticker (e.g. `0.20`–`0.45`, higher for TSLA/NVDA-style names) +- `dt`: elapsed time as a fraction of a trading year for one tick (`tick_interval_seconds / (252 * 6.5 * 3600)`, i.e. treating each tick as a fraction of a 6.5-hour trading day) +- `Z`: standard normal random draw, independent per tick but **correlated across tickers** (§3) + +## 3. Correlated Moves + +To make sector co-movement visible (PLAN.md: "tech stocks move together"), tickers are grouped, and each tick draws: + +1. One shared market-wide factor `Z_market ~ N(0,1)` +2. One sector factor `Z_sector ~ N(0,1)` per group (e.g. `tech`, `finance`) +3. One idiosyncratic factor `Z_idio ~ N(0,1)` per ticker + +Each ticker's `Z` is a weighted blend: + +``` +Z_ticker = beta_market * Z_market + beta_sector * Z_sector + beta_idio * Z_idio +``` + +with `beta_market + beta_sector + beta_idio` chosen so `Z_ticker` stays unit-variance (e.g. weights `0.4, 0.3, sqrt(1 - 0.4^2 - 0.3^2)`). Default sector groups for the seed watchlist: + +| Sector | Tickers | +|---|---| +| tech | AAPL, GOOGL, MSFT, AMZN, NVDA, META, NFLX | +| finance | JPM, V | +| auto/other | TSLA | + +## 4. Random Events + +Each tick, per ticker, with small probability (e.g. `p_event = 0.001` at a 500ms tick → roughly one event per ~8 minutes per ticker), apply an independent shock: + +``` +S(t) *= (1 + sign * random.uniform(0.02, 0.05)) # sign = +1 or -1, 50/50 +``` + +This produces the "sudden 2–5% moves... for drama" called for in PLAN.md §6, layered on top of (not replacing) the GBM tick for that step. + +## 5. Seed Prices + +A single `SEED_PRICES: dict[str, float]` module-level map is the one place starting prices live (addresses the simplification flagged in `planning/REVIEW.md`): + +```python +SEED_PRICES: dict[str, float] = { + "AAPL": 190.00, "GOOGL": 175.00, "MSFT": 420.00, "AMZN": 185.00, + "TSLA": 245.00, "NVDA": 135.00, "META": 590.00, "JPM": 210.00, + "V": 275.00, "NFLX": 680.00, +} +DEFAULT_SEED_PRICE = 100.00 # used for any ticker not in the map (e.g. user/AI adds an unrecognized symbol) +``` + +When `watch(ticker)` is called for a ticker not already tracked, its starting price is `SEED_PRICES.get(ticker, DEFAULT_SEED_PRICE)`, and it's assigned to the `misc` sector group (`beta_sector = 0`, i.e. only market + idiosyncratic factors) unless explicitly added to a sector table. + +## 6. Tick Loop + +```python +class SimulatorMarketDataSource(MarketDataSource): + def __init__(self, tick_interval: float = 0.5, seed: int | None = None): + self._tick_interval = tick_interval + self._rng = random.Random(seed) + self._prices: dict[str, float] = {} + self._sectors: dict[str, str] = {} + self._cache: dict[str, PriceTick] = {} + self._session_open: dict[str, float] = {} # previous_close per ticker + self._subscribers: list[asyncio.Queue[PriceTick]] = [] + self._task: asyncio.Task | None = None + + async def start(self, tickers): + for t in tickers: + self.watch(t) + self._task = asyncio.create_task(self._tick_loop()) + + def watch(self, ticker: str) -> None: + if ticker in self._prices: + return + price = SEED_PRICES.get(ticker, DEFAULT_SEED_PRICE) + self._prices[ticker] = price + self._session_open[ticker] = price + self._sectors[ticker] = _sector_for(ticker) + + async def _tick_loop(self): + while True: + self._advance_all() + await asyncio.sleep(self._tick_interval) + + def _advance_all(self): + z_market = self._rng.gauss(0, 1) + z_sector = {s: self._rng.gauss(0, 1) for s in set(self._sectors.values())} + for ticker, price in self._prices.items(): + z = _blend(z_market, z_sector[self._sectors[ticker]], self._rng.gauss(0, 1)) + new_price = _gbm_step(price, mu=0.05, sigma=_sigma_for(ticker), dt=_DT, z=z) + if self._rng.random() < 0.001: + sign = self._rng.choice([-1, 1]) + new_price *= 1 + sign * self._rng.uniform(0.02, 0.05) + self._prices[ticker] = new_price + tick = PriceTick( + ticker=ticker, + price=round(new_price, 4), + previous_close=self._session_open[ticker], + timestamp=time.time(), + ) + self._cache[ticker] = tick + self._publish(tick) +``` + +`_gbm_step` and `_blend` implement §2/§3 directly; `_DT` is the per-tick fraction of a trading year derived from `tick_interval`. + +## 7. Testability + +- `seed` param makes the RNG deterministic for unit tests (`GBM math is correct`, per PLAN.md §12). +- Because `_advance_all` is a pure step function over `self._prices`, tests can call it directly N times and assert statistical properties (mean drift ≈ `mu`, no negative prices, event shocks bounded to 2–5%) without needing to run the async tick loop. +- Conformance test: a shared test suite (parametrized over both `SimulatorMarketDataSource` and a mocked `MassiveMarketDataSource`) asserts both satisfy the `MarketDataSource` ABC contract from `MARKET_INTERFACE.md` §3. diff --git a/planning/MASSIVE_API.md b/planning/MASSIVE_API.md new file mode 100644 index 00000000..04d039fd --- /dev/null +++ b/planning/MASSIVE_API.md @@ -0,0 +1,106 @@ +# Massive API (formerly Polygon.io) — Reference for FinAlly + +Research notes and code examples for the endpoints FinAlly's market-data layer needs: real-time/latest quotes for multiple tickers and end-of-day (previous close) prices. This document is the input to `MARKET_INTERFACE.md`, which designs the unified Python interface built on top of it. + +Sources: [API Docs](https://massive.com/docs), [Stocks REST Overview](https://massive.com/docs/rest/stocks/overview), [Full Market Snapshot](https://massive.com/docs/rest/stocks/snapshots/full-market-snapshot.md), [Unified Snapshot](https://massive.com/docs/rest/stocks/snapshots/unified-snapshot.md), [Single Ticker Snapshot](https://massive.com/docs/rest/stocks/snapshots/single-ticker-snapshot.md), [Previous Day Bar](https://massive.com/docs/rest/stocks/aggregates/previous-day-bar.md), [Rate Limits](https://massive.com/knowledge-base/article/what-is-the-request-limit-for-massives-restful-apis), [Python client](https://github.com/massive-com/client-python) + +> Polygon.io rebranded to **Massive** in early 2026. `MASSIVE_API_KEY` in `.env` refers to this service. + +## 1. Base URL & Authentication + +- Base URL: `https://api.massive.com` +- Auth: `Authorization: Bearer ` HTTP header (the official Python client sends this; the legacy Polygon.io `?apiKey=` query param may still work but the header form is documented as current). + +```bash +curl -H "Authorization: Bearer $MASSIVE_API_KEY" \ + "https://api.massive.com/v2/snapshot/locale/us/markets/stocks/tickers?tickers=AAPL,GOOGL,MSFT" +``` + +## 2. Rate Limits + +- **Free tier**: 5 requests/minute. +- **Paid tiers**: effectively unlimited, but Massive still asks clients to stay under ~100 req/s to avoid throttling. +- This is the reason PLAN.md §6 specifies polling (not per-request-per-tick) on a 15s interval for the free tier. + +## 3. Endpoints Relevant to FinAlly + +### 3.1 Full Market Snapshot (multi-ticker, one call) — **primary endpoint for polling the watchlist** + +``` +GET /v2/snapshot/locale/us/markets/stocks/tickers?tickers=AAPL,TSLA,GOOG +``` + +Params: +- `tickers` (optional, comma-separated, case-sensitive): restricts the snapshot to specific symbols — this is how FinAlly fetches all watchlist tickers in a single call. +- `include_otc` (optional, bool, default `false`) + +Response: + +```json +{ + "count": 1, + "status": "OK", + "tickers": [ + { + "ticker": "AAPL", + "todaysChange": -0.124, + "todaysChangePerc": -0.601, + "day": { "c": 20.506, "h": 20.64, "l": 20.506, "o": 20.64, "v": 37216, "vw": 20.616 }, + "prevDay": { "c": 20.63, "h": 21, "l": 20.5, "o": 20.79, "v": 292738 }, + "min": { "c": 20.506, "h": 20.506, "l": 20.506, "o": 20.506, "v": 5000 }, + "lastQuote": { "P": 20.6, "p": 20.5 }, + "lastTrade": { "p": 20.506, "s": 2416 }, + "updated": 1605192894630916600 + } + ] +} +``` + +Fields FinAlly needs: `ticker`, `lastTrade.p` (latest price), `prevDay.c` (previous close, for daily % change fallback), `day.c` (today's close-so-far), `updated` (nanosecond epoch timestamp). + +### 3.2 Single Ticker Snapshot + +``` +GET /v2/snapshot/locale/us/markets/stocks/tickers/{stocksTicker} +``` + +Same shape as one entry of the full snapshot's `tickers[]` array, nested under `ticker` instead. Useful for on-demand lookups (e.g., when the AI or user adds a new ticker to the watchlist and a price is needed immediately, without waiting for the next poll cycle). + +### 3.3 Unified Snapshot (multi-asset-class, newer endpoint) + +``` +GET /v3/snapshot?ticker.any_of=AAPL,GOOGL,MSFT&limit=10 +``` + +- `ticker.any_of`: comma-separated list, up to 250 tickers. +- Returns `last_quote.bid/ask`, `last_trade.price/size`, `session.open/high/low/close/volume`. +- Broader (covers options/crypto/forex too) but the field names differ from the v2 snapshot. **Recommendation: use 3.1 (`/v2/snapshot/.../tickers`) for FinAlly** since it's stocks-only, matches the schema FinAlly's `MarketDataSource` interface needs (see `MARKET_INTERFACE.md`), and keeps parsing simpler. + +### 3.4 Previous Day Bar (end-of-day OHLC) + +``` +GET /v2/aggs/ticker/{stocksTicker}/prev?adjusted=true +``` + +```json +{ + "results": [ + { "T": "AAPL", "c": 115.97, "h": 117.59, "l": 114.13, "o": 115.55, "t": 1605042000000, "v": 131704427, "vw": 116.3058 } + ], + "status": "OK", + "ticker": "AAPL" +} +``` + +Single-ticker only — for the watchlist's set of tickers this requires one call per ticker, so it's mainly useful for seeding a starting price for a newly-added ticker outside market hours, not for the live polling loop (use 3.1 for that, whose `prevDay` block already carries this data per ticker in one call). + +## 4. Recommended Polling Strategy for FinAlly + +1. On each poll tick, call **3.1 Full Market Snapshot** once with `tickers=` set to the comma-joined current watchlist (+ any tickers with open positions, per PLAN.md REVIEW.md note on delisted-from-watchlist positions). +2. Parse each entry into `{ticker, price: lastTrade.p, previous_close: prevDay.c, timestamp: updated}`. +3. Poll interval: 15s on free tier (5 req/min ceiling — one snapshot call per tick, well under the limit regardless of watchlist size since it's a single request). Configurable via env for paid tiers (down to 2s). +4. On request failure or rate-limit (HTTP 429): back off and retry next tick; do not fall back to the simulator mid-session (would produce a discontinuous price jump) — just skip the tick and keep the last known price in the cache. + +## 5. Ticker Validation + +Massive returns `status: "OK"` with an empty or partial `tickers[]` array for unknown symbols rather than erroring — so ticker existence should be validated via the snapshot response itself (empty result for that symbol = invalid/unrecognized ticker), not via a separate call to `/v3/reference/tickers`. diff --git a/planning/REVIEW.md b/planning/REVIEW.md new file mode 100644 index 00000000..2cb85ac2 --- /dev/null +++ b/planning/REVIEW.md @@ -0,0 +1,20 @@ +# Review — Changes Since Last Commit + +Review of uncommitted work on branch `start` (base commit `6b568a9`): three new planning documents, `MASSIVE_API.md`, `MARKET_INTERFACE.md`, and `MARKET_SIMULATOR.md`, researching and designing the market-data layer described in `PLAN.md` §6. + +## Summary of Changes + +- **`planning/MASSIVE_API.md`** (new): Research notes on the Massive (formerly Polygon.io) REST API — auth, rate limits, and the specific endpoints needed for multi-ticker real-time quotes and end-of-day prices. +- **`planning/MARKET_INTERFACE.md`** (new): Design for the unified `MarketDataSource` abstract interface selected by `MASSIVE_API_KEY` presence, implemented by both the Massive-backed source and the simulator. +- **`planning/MARKET_SIMULATOR.md`** (new): GBM-based price simulator design — correlated sector moves, random event shocks, seed prices, tick loop. + +## Findings + +- **Unverified external claims (`MASSIVE_API.md`)**: The endpoint shapes, auth header, and rate limits were pulled from Massive's public docs and a few third-party summaries via web search/fetch, not confirmed against a real API key/response. Before backend implementation begins, a quick live smoke test against the actual API (with a real `MASSIVE_API_KEY`) should confirm field names (`lastTrade.p`, `prevDay.c`, `updated` in nanoseconds) and the auth header format, since some of the fetched doc pages returned incomplete content and had to be corroborated from the Python client's debug output rather than an explicit curl example. +- **Interface not yet wired to PLAN.md's open questions**: `MARKET_INTERFACE.md` §7 proposes an answer to the "delete a watched ticker with an open position" question raised previously (keep it watched via position-derived `watch()` calls), and `MASSIVE_API.md` §5 proposes an answer to the "unrecognized ticker" question (treat an empty snapshot result as invalid rather than a separate validation call). Neither of these is yet reflected back into `PLAN.md` itself — worth folding in as explicit rule updates so `PLAN.md` stays the single source of truth rather than requiring readers to cross-reference three documents. +- **No rounding/precision decision made**: `MARKET_SIMULATOR.md` rounds simulated prices to 4 decimals ad hoc; `MASSIVE_API.md`/`MARKET_INTERFACE.md` don't specify a rounding convention for cash/price values at all. This was flagged as an open question against `PLAN.md` previously and remains unresolved by this round of docs — should be decided before the backend's trade-execution math is implemented, since simulator and Massive-backed prices need to agree on precision for `PriceTick.price` to be a true drop-in-compatible contract. +- **Poll-interval fallback behavior newly specified but not yet approved**: `MASSIVE_API.md` §4 states that on request failure/429 the poller should skip the tick and keep serving cached prices rather than falling back to the simulator. This is a reasonable default but is a product decision (not purely technical) and should get explicit sign-off, since PLAN.md itself left it as an open question rather than a decided behavior. + +## Verdict + +No code was changed — this is planning/documentation only, so there's no runtime risk yet. The main follow-up is reconciling the new design decisions back into `PLAN.md` (or leaving them here as an addendum) so there's one authoritative document, and validating the Massive API details against a live key before backend work starts.