Add scope/refusal contract to chat system prompt (closes #101)

[vahid-ahmadi]

Summary

Adds a SCOPE & REFUSAL: section near the top of SYSTEM_PROMPT in backend/routes/chatbot.py that makes "out of scope" an explicit contract the model follows. It defines what is in scope (UK tax/benefit microsimulation over the datasets and years capabilities() reports) and out of scope (non-UK policy, macro forecasting, unannounced/future Budgets, legal/tax-filing advice, anything capabilities() reports as not modelled), with clear off-topic, unmodelled, and partial-answer rules.

Rationale

Today the chat has no first-class handling for off-topic or unmodelled questions:

• Off-topic questions are answered anyway, paying full input/output cost (system prompt + cached reference.md) for something that should be declined in one sentence.
• On-topic-but-unmodelled questions (macro/GDP/inflation, non-UK policy) degrade into re-running run_python and re-guessing API shapes instead of stopping after one capabilities() check and saying "not modelled."

The only prior guardrail was a single buried line. This section replaces that with an explicit in/out-of-scope list plus a stop-after-one-check rule.

A partial-answer rule and a personal-allowance/inflation example guard against false refusals: questions that touch a non-modelled dimension but can still be partially answered are answered with the limitation explained, not declined.

Notes

• Prompt-only change — no new tools, no change to _build_system_blocks, no run_python sandbox change. No added LLM call or latency.
• This is "Layer 2" of the broader layered plan for off-topic/uncomputable questions, complementing the off-topic pre-flight topic gate (PR Cut wasted tokens (off-topic gate) and cold-start latency (/chat/backends warmup) #95) and graceful failure on the iteration cap (Cap run_python iterations with a best-effort fallback message #84 / PR Cap chat iterations at 12 with graceful fallback (closes #84) #87).

Closes #101

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policyengine-uk-chat	Ready	Preview, Comment	Jun 15, 2026 2:56pm

[github-actions]

Beta preview is ready.

• Frontend: open preview
• Backend: open backend

[anth-volk]

@vahid-ahmadi You sure about this part?

[vahid-ahmadi]

@anth-volk heads-up on a fix I just pushed (1108bc2) to the SCOPE & REFUSAL block, in case it affects your review.

Contradiction in the original text: the decline list named inflation as flatly out-of-scope —

Out of scope (decline): … macroeconomic forecasting (GDP, inflation, employment, market reactions)

— but the flagship partial-answer example was an inflation question it said not to decline. So for the same query type the prompt gave opposite instructions; the model could swing between a curt refusal and a full simulation.

Fix:

1. Scoped the macro decline to the pure-forecast case — "what will inflation/GDP/employment be?" with no modelled tax-benefit lever in the question — so it no longer collides with the example.
2. Made the partial-answer rule explicitly take precedence when a modelled policy is in the question, so the tie-break is unambiguous.
3. Reworded the example so the answer makes clear it's addressing the modelled part (fiscal/distributional) and that the macro part is out of scope — rather than implying it answered the inflation question.

Note this PR also got rebased onto current main earlier (main had refactored SYSTEM_PROMPT out of chatbot.py into section constants in prompts.py), so the contract now lives as a SCOPE_AND_REFUSAL section there. Verified: prompt assembles correctly and test_prompts.py passes.

Still flagging, as before, that prompt-behaviour like this really wants an eval case (topic-gate / #52 harness) to lock it in rather than trusting wording alone — happy to follow up.

[anth-volk]

Following up on my review — a design suggestion on the partial-answer rule specifically (now that 1108bc2 has resolved the inflation contradiction).

As written, the rule is compute-first: when a question centres on a modelled reform but also touches a non-modelled dimension, the model runs the simulation immediately and caveats the unmodelled part inline. I'd like us to consider flipping it to confirm-first:

1. State the boundary up front — what it can and can't answer for this question.
2. Offer the modelled analysis.
3. Run the simulation only once the user agrees.

So "how will raising the personal allowance affect inflation?" would first return something like: "I can't model the inflation (second-round macro) effect — that's outside the microsimulation. I can show the fiscal and distributional impact of raising the personal allowance. Want that?" — and compute on confirmation, rather than computing immediately.

Why I think this is worth it:

• It avoids spending the expensive path on an unwanted answer. The partial-answer route runs run_economy_simulation, the heaviest tool we have. If the user actually wanted the inflation answer (which we can't give), compute-first burns a full distributional simulation and then tells them the thing they cared about is out of scope. Confirm-first spends nothing on the engine until the user says the modelled slice is useful.
• It sets expectations before presenting numbers, which reduces the risk of a partial answer reading as if it were complete — the exact failure mode the caveat is trying to prevent, but handled before the work rather than after.

The tradeoff is a round-trip of latency/friction on the common case where the user did want the modelled answer and would just say "yes" — and it cuts against the app's general eager-compute stance ("every number must come from a tool result you just computed").

One framing question for you: this confirm-first shape is conceptually a scoped Plan-mode turn — PLAN_MODE_DIRECTIVE already encodes "don't call tools, ask first." So rather than adding a fourth bespoke rule to SCOPE_AND_REFUSAL, it might be cleaner to express partial-answer cases as "enter a Plan-mode-style turn" and reuse that machinery. Worth deciding whether this is a new rule or a mode interaction.

Not blocking — the current compute-first version is internally consistent now. Flagging it as a behavioural choice I'd like us to make deliberately, ideally pinned by an eval case (the personal-allowance → inflation flow) once we settle where the eval harness lives post-#52.