Drivetrain

Drivetrain is a small OpenAI-compatible proxy for inference servers that support llama.cpp/llama-server speculative decoding request fields. It chooses speculative decoding settings per request, injects missing speculative.* fields, forwards the request upstream, and records local telemetry for online policy adaptation.

It is designed for local/trusted inference deployments where different prompt/decode shapes benefit from different multi-token prediction (MTP) depths.

What it does

• Proxies GET /v1/models and POST /v1/chat/completions to an upstream OpenAI-compatible server.
• Classifies request shape from prompt size, decode budget, task hints, and streaming mode.
• Chooses one of three default profiles:

Bucket	Profile	Fields
short	short-n6	speculative.n_max=6, n_min=0, p_min=0.0, p_split=0.2
mid	mid-n4	speculative.n_max=4, n_min=0, p_min=0.0, p_split=0.2
long-safe	long-safe-n2	speculative.n_max=2, n_min=0, p_min=0.0, p_split=0.2

• Preserves caller-provided speculative fields, including falsey values such as 0 or null.
• Persists request telemetry to SQLite and policy observations to JSON.
• Supports deterministic adjacent exploration for online tuning.
• Supports basic SSE streaming passthrough.

Endpoints

• GET /health
• GET /dynamic-mtp/status
• GET /v1/models
• POST /v1/chat/completions

Install

python -m pip install -e '.[test]'

Run

DYNAMIC_MTP_CONFIG=configs/example.yaml   uvicorn --factory drivetrain.app:create_app --host 127.0.0.1 --port 8090

Then point a client at:

http://127.0.0.1:8090/v1

Configuration

See configs/example.yaml. Keep the service bound to localhost or behind an authenticated reverse proxy unless you have added your own access controls.

Tests

pytest -q

Security

This is an inference proxy intended for trusted local networks. It is not an authentication gateway. See SECURITY.md.