This is the Inital workflow, this will change as the projects grow.
Below contain the workflow and actual approach to solve this Problem.
Gateway Exchange → Data Ingestion Service
-
The exchange (e.g., Deribit) provides market data.
-
The ingestion service connects to the exchange, subscribes to data, and normalizes it.
Data Distribution
- Data is pushed into NATS ( JetStream for durable streams & replay).
Consumers
-
Services and CLIs (Frontends) subscribe to NATS subjects to get live data.
-
This allows scaling multiple independent consumers without coupling to ingestion.
-
Build a Python CLI
- Start a basic ingestion service (connect to Deribit, fetch orderbook stream)
- Start and configure NATS JetStream
- Provide a CLI for consuming data (subscribe and display)
This project is managed by uv. Hence these are some steps to follow for setup.
# Create a virtual env
python3 -m venv .venv
# Activate the virtual environment
source .venv/bin/activate
# UV install
uv sync
# CLI Working
task click# To run service cli
task click service
# Start services
docker-compose up -d
# View logs
docker-compose logs -f
# Stop services
docker-compose down
# Stop and remove volume (ideal for removing cache)
docker-compose down -v