Steer tool-choice eval (Ollama): switch/end/reply routing has no repeatable LLM-in-the-loop test [spec 11 §5]

[wine-fall]

What it is

Spec 11 gives the reply turn action tools (switch_music, end_broadcast,
submit_reply). Whether the model calls the right tool on the right turn is
stochastic behavior: the unit suite covers only the deterministic scaffolding
(tool handlers, Director wiring, epoch guard), and the one real-SDK smoke
(scratch/smoke_steer.ts, throwaway) passed 3/3 on 2026-08-01 — a switch
request called switch_music with a hint, a mood remark called nothing, an
explicit off-request armed without closing. That is an on-demand probe, not a
repeatable eval: there is no LLM-in-the-loop test that would catch a prompt
regression (e.g. a rewrite of STEER_SWITCH_RULE making the model narrate
instead of act, or call end_broadcast on mood remarks).

The repo currently has no eval-track infrastructure at all (DESIGN §10.3 /
§11.4 name the convention: Ollama-preferred, canned turns, assert on tool-call
traces, never on wording) — so this is the first consumer.

Spec

specs/spec11/11-agentic-steer.md §5 Testing (the "LLM-in-the-loop (eval)"
item); DESIGN §10.3 / §11.4 for the eval-track convention.

Done when

A runnable eval exists (Ollama-preferred per §11.4) that feeds canned steer
turns — at minimum: a plain-chat turn, a skip request, a hinted/style request,
a mood remark, an explicit off request, and a confirm-after-armed turn — and
asserts on the tool-call traces (which tools were called, with what argument
shape), not on reply wording. It runs on demand (not in the fast unit gate)
and is documented in spec 11 §5. Until then, scratch/smoke_steer.ts (or a
re-created equivalent) is the manual stand-in.