GAR — Green-Aware Routing for LLM Pools

Routes each inference request to the lowest-carbon model that still meets an accuracy floor and a p95-latency SLO. Implements the five policies from the GAR paper with a lightweight set of sklearn-based predictors.

---

How it works

For every incoming query x at time t, the router:

1. Predicts accuracy p̂_m, p95-latency ℓ̂_m, and carbon ĉ_m for each model.
2. Inflates latency and carbon estimates by safety margins γ_ℓ and γ_c (Eq. 2).
3. Gates on the service-level feasible set (Eq. 3):

   F(x,t) = { m : p̂_m ≥ τ  AND  ℓ̃_m,p95 ≤ L }
   

4. Selects the model in F(x,t) that minimises inflated carbon (Eq. 5), with lower latency / higher accuracy as tie-breakers.
5. Falls back to the highest-capacity model when F(x,t) is empty.

---

Routing policies

Policy	Description
GAR	Baseline — minimise carbon over the feasible set.
GAR-Fixed	Adds a hard per-request carbon cap c_cap to feasibility; falls back to uncapped set if cap is too tight.
GAR-ε	Restricts to models within ε accuracy points of the best feasible model, then minimises carbon.
GAR-Target	Binary-searches per-dataset accuracy floors τ_d on a calibration set to hit a desired macro-accuracy target, then minimises carbon.
GAR-PD	Online primal-dual: maintains dual variable λ_t that shadow-prices carbon overruns against a rolling W-request budget B. Dual is updated after each request via Eq. 9.

---

Estimators

Estimator	Method
LogisticAccuracyEstimator	Per-model logistic regression + temperature-scaling calibration.
QuantileLatencyEstimator	Per-model quantile regression (sklearn QuantileRegressor) at q = 0.95.
LinearCarbonEstimator	Closed-form: (α_m + β_m × tokens) × grid_intensity(region_m). No training needed.

All three implement a common ABC interface so they can be swapped for heavier models (e.g. gradient-boosted trees, neural calibrators) without changing the router.

---

Project structure

backend/
  api.py           # FastAPI server — live grid data + ESG extraction endpoints
  gar/             # GAR routing framework (models, estimators, router, metrics)
  esg/             # ESG extraction pipeline (Claude-powered PDF → GarSettings)
  esg_agent.py     # CLI wrapper for ESG extraction (dev/testing)
  demo.py          # Synthetic end-to-end routing simulation
  requirements.txt

frontend/          # Next.js dashboard (App Router)
  src/app/         # Pages and API proxy routes
  src/components/  # UI components including SettingsPanel with ESG import
  src/lib/         # Settings types, routing logic, analytics data

docs/
  ESG_AGENT.md     # ESG pipeline, HTTP API reference, frontend integration
  ESG_FIELDS.md    # Extraction field reference with risk classifications

---

Quickstart

# Python backend
pip install -r backend/requirements.txt
python backend/demo.py          # routing simulation (no server needed)

Sample output:

Policy          Acc    CO2 g/req   Lat OK  Violations
GAR            0.797    0.54968    0.970       0.036
GAR-Fixed      0.797    0.54866    0.970       0.116
GAR-ε          0.838    0.72533    0.968       0.036
GAR-Target     0.764    0.40191    0.984       0.012
GAR-PD         0.797    0.54790    0.978       0.036

CO₂ savings vs always-large-us baseline:  +51 – 64 %

---

Using with real data

Replace the synthetic samplers in demo.py with a log-loader that yields:

# One row per (query, model) observation
queries:    List[Query]             # features, estimated_tokens, dataset tag
labels:     Dict[str, np.ndarray]   # binary correctness per model
latencies:  Dict[str, np.ndarray]   # observed latency (ms) per model

Then train and run:

from gar import (ModelPool, Query,
                 LogisticAccuracyEstimator, QuantileLatencyEstimator,
                 LinearCarbonEstimator, GARRouter)

pool    = ModelPool.default()           # or build your own
acc_est = LogisticAccuracyEstimator(pool)
acc_est.fit(calib_queries, calib_labels)
acc_est.calibrate_temperature(calib_queries, calib_labels)

lat_est = QuantileLatencyEstimator(pool)
lat_est.fit(calib_queries, calib_latencies)

car_est = LinearCarbonEstimator()       # uses ModelMetadata + grid intensity

router  = GARRouter(pool, acc_est, lat_est, car_est, tau=0.7, L=2000.0)
model, violated = router.route(query)

Plug in live grid intensities (e.g. Electricity Maps API) by passing custom_grid={"us-east-1": 312.0, ...} to the router.

---

Default hyperparameters

Parameter	Value	Meaning
γ_c	0.10	Carbon safety margin
γ_ℓ	0.05	Latency safety margin
W	100	GAR-PD sliding-window size
η	0.05	GAR-PD dual step size
B	0.65 × C̄_baseline	Per-request carbon budget

---

Backend API

backend/api.py is a FastAPI service with three endpoints:

Endpoint	Description
GET /health	Liveness probe
GET /api/live-grid	Live carbon intensities (gCO2eq/kWh) and electricity prices for seven European bidding zones; cached 1 hour
POST /esg/extract	Accepts a PDF ESG report, returns GarSettings-compatible JSON extracted by Claude

Start the backend:

cd backend
pip install -r requirements.txt
export ANTHROPIC_API_KEY=sk-ant-...
uvicorn api:app --reload --port 8000

The frontend proxies both endpoints through Next.js API routes. Set GAR_BACKEND_URL in frontend/.env.local to point to a non-default backend address (defaults to http://localhost:8000).

See docs/ESG_AGENT.md for full ESG extraction documentation.

---

Local Development

Frontend app root: frontend/ (Next.js App Router via frontend/src/app).

Package manager: npm (lockfile: frontend/package-lock.json).

Recommended Node.js: 20.x LTS.

From repository root:

npm --prefix frontend ci --legacy-peer-deps
npm run dev
npm run build
npm run lint

From frontend folder:

cd frontend
npm ci --legacy-peer-deps
npm run dev
npm run build
npm run lint

Environment variables (frontend/.env.local, gitignored):

Variable	Default	Description
GAR_BACKEND_URL	http://localhost:8000	URL of the Python FastAPI backend