Add AI gateway to /chat/message (5-outcome routing)#112
Draft
anth-volk wants to merge 3 commits into
Draft
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Beta preview is ready.
|
anth-volk
changed the title
A cheap forced-tool pre-pass builds a structured execution plan for the opening user turn and routes it to one of: irrelevant, out_of_scope, partial, needs_plan, ready. The model grounds each plan slot (source flag); the server gates via per-slot criticality. Any gateway error fails safe to compute. Supersedes the binary scope router (#109) and the reference.md drift stamp (#106; replaced by the engine-derived scope descriptor). - backend/gateway_config.py: criticality table, inferable set, promotions, gate() - backend/gateway.py: run_gateway (forced emit_plan), verdict, writer/plan helpers - backend/prompts.py: gateway + lightweight prompts, scope descriptor - backend/routes/chatbot.py: gateway routing in generate_stream - backend/scripts/build_reference.py: build_scope_descriptor -> scope_descriptor.md - evaluation: GatewayCase + _run_gateway + 14 live cases (evals/cases/gateway) - backend/tests/test_gateway.py: 33 offline tests (gate, parser, fail-safe) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
anth-volk
force-pushed
the
feat/chat-gateway
branch
from
ea06514 to
a153910
Compare
The gateway's needs_plan outcome is now the only source of clarifying questions, so the manual plan-mode toggle is removed end to end: - backend: plan_mode request field, PLAN_MODE_DIRECTIVE, the not-plan_mode guards, and plan-mode handling in the eval runner/schemas - frontend: the /plan slash command + handler, the Plan toggle button, plan_mode in the request bodies, and the now-unused IconBulb import - tests + eval cases for plan mode, and stale plan-mode references in the skill docs and the cap-hit fallback message Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add backend/model_config.py with DEFAULT_TEMPERATURE (0) and SUGGESTION_TEMPERATURE (1), and route every model call through them. Previously only the compute loop and eval harness pinned temperature; the gateway, titling, and follow-up suggestions silently inherited the SDK default of 1.0. The gateway is a classifier, so it now runs deterministically at 0. Removes the old CHAT_TEMPERATURE constant (env var ANTHROPIC_CHAT_TEMPERATURE -> ANTHROPIC_TEMPERATURE). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #111
Summary
Adds an AI gateway to
/chat/message: a cheap pre-pass (fast model, forced tool-use) that runs once on the opening user turn, builds a structured execution plan, and routes the turn into one of five outcomes instead of always paying for the full background.The first four reply on the lean lightweight path (no reference doc, no tools, one turn). Only
readypays for the heavy background. Any gateway error fails safe to compute. The gateway is uncalibrated until the live eval runs, so treat that live run as a pre-merge gate.Design — the model grounds, the server gates
The gateway model emits, per plan slot, only a
valueand asource(prompt/default/assumed). It does not judge importance. The server looks up a static per-slot criticality and applies a deterministic gate:This split (in
gateway_config.py) keeps the gate auditable, unit-testable offline, and independent of poorly-calibrated model self-confidence. Over-asking is bounded by the inferable set (schema-required-but-derivable slots likebenunit/householdare inferred, never asked) and by context promotions (e.g. a wealth-tax question promotesdatasetto high so an assumed FRS default gets flagged).Safety
ready/compute on any parse/timeout/API error, an empty/garbage plan, or a plan missing the routing decision — a wrong refusal is the worst outcome, so admissibility biases toward not refusing.partial/needs_planprompt has a prior assistant turn, so_is_followupskips the gateway and the turn flows straight to compute with full context.doneevents carryroute+outcome.Supersedes #109 and #106
compute/lightweightscope router) → generalised into the 5-outcome gateway.reference.md) → replaced by derive-live:build_reference.pynow generatesscope_descriptor.mdfrom the engine, so it can't drift.Recommend closing both once this lands.
Removes manually-selected plan mode
The manual plan-mode toggle (the
/plancommand + button and theplan_moderequest field) is removed end to end — backend, frontend, tests, eval cases, and skill docs. The gateway'sneeds_planoutcome is now the sole source of clarifying questions, so a manual override is redundant. (charts_modeis unaffected.)Tests
backend/tests/test_gateway.py— 34 offline tests (gate logic, criticality, inferable clause, promotions,run_gatewayparser, fail-safe paths, writer directives, flag-off). Full backend suite: 162 passed, 10 skipped.gatewayeval suite — 14 live cases across all 5 outcomes, with minimal-pair discriminators (wealth-tax specified vs not; FRS-correct vs WAS-required dataset; partial vs out_of_scope; inferable household slots must-not-gate). Offline eval: 96 passed, 0 failed, 42 skipped (gateway cases are live-only).Follow-ups
make eval-ai-live --suites gatewayto confirm the classifier hits the expected outcomes; tune the gateway prompt + promotions against any misroutes. Criticality is deterministic, so most tuning is the grounding prompt.reference.mddrift warning on top of derive-live, if wanted.🤖 Generated with Claude Code