th-562b6d: clamp chat max_tokens to model /model/info ceiling#190
Open
brentrager wants to merge 10 commits into
Open
th-562b6d: clamp chat max_tokens to model /model/info ceiling#190brentrager wants to merge 10 commits into
brentrager wants to merge 10 commits into
Conversation
…nymous) The operator's resolve_ws_access degrades a missing/invalid ?token= to an ANONYMOUS connection rather than rejecting it, so a token verifier gates only ACL scope, not connections — a real gap for single-tenant local/tailnet deployments (a tokenless peer can drive the agent). Surfaced by the smooth daemon's live e2e. - AppState.strict_auth + with_strict_auth (default false — K8s/widget anonymous flows unchanged). - resolve_ws_access -> Result<ConnectionAuth, ()>: in strict mode a missing/invalid token returns Err; ws_upgrade then responds 401 instead of upgrading. Lenient (default) is byte-for-byte unchanged. - LocalServerBuilder::strict_auth(bool) threads it; the smooth daemon's local flavor opts in. Tested via the daemon's live e2e (smooth repo): tokenless /ws REJECTED, valid-token accepted + a real LLM turn still runs. Unit test: strict off by default, opt-in threads to AppState. (Couldn't run the operator's own suite in this worktree — a corrupt rustls-pki-types 1.14.1 in the local cargo cache; validated through the daemon build, which pins the working 1.14.0.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wBikCFJyoowRokiWK5rX1
…sistence) The local flavor hardcoded InMemoryStorageAdapter, so an always-on, self-hosted daemon lost all conversations/sessions/checkpoints on restart — and the only durable backend was Postgres (the cloud flavor). Add a seam so an embedder can inject ANY StorageAdapter (e.g. a local sqlite/dolt one) without standing up Postgres: - AppState::with_storage(Arc<dyn StorageAdapter>) — builder, mirrors with_auth. - LocalServerBuilder::storage(...) — when set, build() installs it in place of the in-memory default; unset → in-memory (unchanged). This is the keystone for collapsing the smooth daemon onto a single operator runtime: with durable local storage the operator path can absorb what the bespoke serve_persistent path provided, and the bespoke path retires. Tested: storage_seam_installs_a_durable_adapter (injected adapter installed; default still in-memory). 33 lib tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wBikCFJyoowRokiWK5rX1
…rve_spa Three additive, behavior-preserving seams the smooth-daemon (EPIC th-c89c2a) needs; all default to the prior behavior when unused: - default_persona (AppState + LocalServer.persona()): a host-installed default system prompt the WS chat turn falls back to when no per-org persona is set. Unset → the const customer-support prompt, byte-for-byte unchanged. - CORS on /admin (admin_cors): allow cross-origin GET/POST with the auth header so a same-origin-or-dev SPA can read /admin/me. /ws (no preflight) untouched; auth unchanged. - serve_spa(Router): mount a host-supplied SPA as the router fallback (so /ws, /health, /admin still win), token-injection left to the host (SPA-agnostic, no cross-repo dep). Tests added for each seam (state default/trim, admin CORS header + preflight, SPA-vs-operator routing). Full operator-server suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RuPU8KGE7exadVU6Gi9hBs
…rotocol msg Reasoning tokens now arrive on AgentEvent::ReasoningDelta (smooth-operator-core th-4d8682). Route them to a new stream_reasoning protocol message — shaped like stream_token but on a distinct type — so clients render reasoning as 'thinking' and never as the answer. Test added. Pearl th-4d8682. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RuPU8KGE7exadVU6Gi9hBs
…+ /admin/model-costs
Foundation for Smooth Modes (th-f512b1, th-2a6330), all additive + back-compat:
- send_message gains optional 'model' → that turn runs on the chosen model
(sets llm.model for the turn only; absent = server default).
- eventual_response carries data.data.usage.{costUsd,promptTokens,completionTokens}
(from AgentEvent::Completed) so clients accumulate live session cost.
- GET /admin/model-costs (ungated) returns model->{inputCostPerToken,outputCostPerToken,
tier,useCases} from the gateway /v1/model/info, cached; {} (200) on gateway error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RuPU8KGE7exadVU6Gi9hBs
…alongside /ws,/admin,/ Lets a host (the smooth-daemon) mount its own routes (e.g. /search for @ mentions) into the operator router as real merged routes, CORS-matched to /admin. admin_cors made pub(crate) for reuse. Additive; default behavior unchanged. (th-58b5fe) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RuPU8KGE7exadVU6Gi9hBs
smooth-operator-server: send_message accepts an optional 'images' array of data-URL media; runner threads them via TurnRequest.user_images into AgentConfig::with_user_images so the engine emits OpenAI image_url content parts. Image/PDF turns auto-route to gemini-2.5-flash-lite (explicit model still wins). Bump core path-dep 0.14->0.15 (multimodal API landed in core 0.15). Fill Message.images:vec![] at the two EngineMessage build sites and the TurnRequest call/test sites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chat path now clamps max_tokens to the resolved model's output ceiling: - map_model_info carries maxOutputTokens from /model/info - model_output_ceiling(state, model) reuses the cached /model/info fetch (one fetch/process), best-effort -> None on any gateway error - handler threads the resolved model's ceiling into TurnRequest.model_max_output - runner applies AgentConfig::with_model_ceiling Groq-compound (8192 out) is the load-bearing case; high-ceiling models are no-ops at the current 32768 budget. EPIC th-1cc9fa. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Closing: this PR merges the entire long-lived localflavor-hardening branch (conflicts with main), not just the D clamp change. The clamp (th-562b6d) is committed on the branch and already live on the smoo-hub daemon via path-dep. Landing it to hardening-main should be a cherry-picked PR off current main, or ride the branch's own merge — not this whole-branch merge. |
…92, iterations 6->20) The 512/6 defaults are chat-widget sizing that starve reasoning models (they burn the budget on reasoning_content and return empty). Safe to raise now that the per-model output-ceiling clamp caps max_tokens to what each model can emit — a higher cap only bounds runaway output, concise answers stay concise. OSS default benefits every consumer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Chat path clamps
max_tokensto the resolved model's output ceiling (max_output_tokens) from the gateway/model/info, via the engine'sAgentConfig::with_model_ceiling(smooth-operator-core #69).map_model_infonow carriesmaxOutputTokensmodel_output_ceiling(state, model)reuses the existing cached/model/infofetch (one fetch/process), best-effort →Noneon any gateway error (graceful passthrough, zero behaviour change when unknown)TurnRequest.model_max_output; runner applies itWhy
A budget
max_tokens(tuned high, or per-org via@smooai/configlimits) must never exceed what a model can emit — else a reasoning model returns empty or the upstream 400s. Load-bearing case:groq-compound(8192 out) under a 32768 budget. High-ceiling models are no-ops.Part of EPIC th-1cc9fa. Already running on the smoo-hub daemon via path-dep (verified:
/model/infosurfaces ceilings for 48 models, turns unaffected).🤖 Generated with Claude Code