Add opt-in scope router to /chat/message (load the heavy background only when needed)

[anth-volk]

Summary

Adds an opt-in scope router to /chat/message: a cheap pre-check that decides which background a turn needs, while the model still answers the user's real message in every case.

It started as the off-topic "topic gate" salvaged from #95 (Sakshi) and has been generalised into a router. Two routes:

• compute — full background: SYSTEM_PROMPT + REFERENCE_DOC + all six tools. Anything needing a calculation or a grounded figure.
• lightweight — lean background: a small LIGHTWEIGHT_SYSTEM prompt, no reference doc, no tools. Off-topic, scope/capability, and explicitly-unmodelled questions get a tailored, model-generated reply — not a canned string — without loading the ~20k-token reference doc and the tool schemas.

Why

#102's scope reasoning currently runs inside the expensive call, entangled with the full background; the original gate proved a cheap pre-pass works but returned a fixed string. This unifies them: the router keeps the model generating from the user's prompt, and only the compute branch pays for the heavy background.

How (reuses the existing loop)

Routing only selects model / system_blocks / whether tools are sent — the same lever plan mode already pulls to drop tools. So billing, usage accounting, the done event, streaming, and the iteration cap are all unchanged. With no tools, the lightweight branch returns end_turn in one iteration and emits done normally. No parallel code path.

Safety properties

• Fail-safe to compute: empty input, any router error, or an unrecognised reply routes to the full background. A wrong compute only wastes the background we'd have loaded anyway; a wrong lightweight risks answering without the data, so we bias hard against it.
• No numbers from memory: the lean prompt forbids quoting figures/parameter values from memory, preserving the app's "every number comes from a computation" invariant.
• Off by default: opt in via POLICYENGINE_CHAT_SCOPE_ROUTER_ENABLED=true (model via POLICYENGINE_CHAT_SCOPE_ROUTER_MODEL). Disabled, behaviour is byte-identical to current main.

Engine-derived scope descriptor (no drift)

The router reasons over a compact SCOPE_DESCRIPTOR (~a few hundred tokens) instead of the full reference doc. It is now generated from the engine at deploy time: scripts/build_reference.py also writes scope_descriptor.md, deriving the modelled half from the Parameters schema (the authoritative reform-key list) and capabilities() (datasets), combined with a curated not-modelled boundary. The route layer loads it alongside reference.md and falls back to DEFAULT_SCOPE_DESCRIPTOR when absent (local dev). prompts.py owns the instruction text + scope_router_system() / lightweight_system() builders; chatbot.py loads the descriptor and assembles. scope_descriptor.md is gitignored like reference.md.

Sample generated descriptor (against the installed engine): modelled reform keys income_tax, national_insurance, universal_credit, child_benefit, pension_credit, …; datasets frs, efrs, lcfs, spi, was.

Eval coverage

New routing suite in the AI eval harness pins the routing deterministically. A RoutingCase asserts a prompt routes to compute or lightweight via _route_scope. Cases are live-only (requirements: [live_model]) since the router makes a real fast-model call. The canonical case is personal-allowance → inflation (must route compute — the modelled part needs the engine; the macro caveat is the main loop's job), plus contrasting pure-macro / off-topic / capability (→ lightweight) and reform-cost / household-calc (→ compute) cases.

Not included (Part 2 of #95)

Intentionally excluded — the /chat/backends cold-start work (model_backends.py, modal_app.py pre-import, the ChatPage.tsx dropdown, the workflow tweak), coupled to the closed backend-selector stack (#51).

Tests

• backend/tests/test_scope_router.py — router parser calibration (light/compute, fail-safe on empty/error/unrecognised), _last_user_text flattening, lightweight blocks exclude the reference doc, router off by default.
• Offline eval: 96 passed, 0 failed, 34 skipped across all five suites (routing cases skip offline). Backend unit + eval-harness tests green; 73 passed, 10 skipped across test_scope_router + test_prompts + test_api + test_evaluation.

Follow-ups

• Run the routing suite live (make eval-ai-live) to confirm the classifier hits the expected routes, and tune the router prompt if needed.
• Optional: split lightweight into explicit off-topic / unmodelled / confirm-first handling, tying into the confirm-first proposal on Add scope/refusal contract to chat system prompt (closes #101) #102.

Credit: original gate by @SakshiKekre in #95.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policyengine-uk-chat	Ready	Preview, Comment	Jun 15, 2026 7:57pm

[github-actions]

Beta preview is ready.

• Frontend: open preview
• Backend: open backend

[anth-volk]

@vahid-ahmadi — requesting your review, since this is closely related to your #102 (scope/refusal contract) and I want to be sure they compose rather than collide.

How they differ — they operate at different layers:

• Add scope/refusal contract to chat system prompt (closes #101) #102 reasons about scope inside the model call. The system prompt teaches the model to recognise and decline/scope out-of-bounds requests, with the full background (reference doc + tools) already loaded. It shapes what the model says when a request is partly or wholly out of scope.
• Add opt-in scope router to /chat/message (load the heavy background only when needed) #109 reasons about scope before the model call. A cheap Haiku pre-pass classifies the opening message and picks which background to load: compute (full system prompt + reference doc + all 6 tools) vs lightweight (lean prompt, no reference doc, no tools). It never changes the wording — the model still answers the user's real message either way. It's a cost/latency lever, not a refusal policy.

	#102	#109
Where	in-call (system prompt)	pre-call (classifier)
Goal	correct scoping/refusal wording	don't pay for the heavy background on off-topic turns
Changes default behaviour?	yes	no — opt-in flag, off by default
Touches refusal text?	yes	no

Why they reinforce each other: the lightweight branch in #109 is exactly where #102's scope contract — and the confirm-first idea I raised on #102 — would live, because that's the path where we've already decided not to compute. If #102 lands first, #109's lightweight prompt should adopt its language; if #109 lands first, #102 slots cleanly into the lightweight branch. No conflict, just sequencing.

Two things I'd value your eye on:

1. The fail-safe-to-compute bias: anything ambiguous, any router error, and all follow-up turns load the full background. A wrong compute only wastes background we'd have loaded anyway; a wrong lightweight risks answering without data. Conservative enough for you?
2. The engine-derived scope descriptor (build_reference.py → scope_descriptor.md) as the source of truth — how that sits alongside your version-stamp approach in Version-stamp reference.md and warn on engine drift (closes #104) #106.

Still draft while I run the routing eval suite live to confirm the classifier hits the expected routes.

[anth-volk]

@vahid-ahmadi — #106 and #109 are circling the same two questions from different angles, and I'd rather we agree a target shape than rebase around each other piecemeal. Can we sync? I think we need an affirmative, agreed pathway on two things before either merges:

1. How we handle irrelevant / unanswerable questions.
Three overlapping mechanisms are in flight: your scope/refusal contract (#102, in-call), my scope router (#109, pre-call — picks a light vs. full background), and the confirm-first idea I raised on #102. They can compose (different layers), but we should decide the canonical layering: what actually declines, what just loads a lighter background, and what the user sees in each case.

2. How we handle systemic metadata.
#106 (stamp-and-warn on engine drift) and #109 (derive the scope descriptor live from the Parameters schema + capabilities()) are two different answers to the same "stale-index" risk. They also edit the same two spots — build_reference.py and the REFERENCE_DOC load in chatbot.py — so they'll conflict on merge regardless. We should pick one philosophy (stamp-and-warn vs. derive-live) and apply it consistently across reference.md and the scope descriptor.

I'm putting #109 back into draft until we've talked, so we settle the architecture before either lands.