diff --git a/CHANGELOG.md b/CHANGELOG.md index af63efcf4c..84431d2dd2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,42 @@ All notable changes to GBrain will be documented in this file. +## [Unreleased] + +**A brain can now run start to finish on local models, with no hosted API key anywhere.** + +gbrain has supported Ollama since v0.32, but only for embeddings. That meant a local install could index and search a brain and then hit a wall: every step that needed to *reason* — `gbrain think`, query expansion, `gbrain dream`, `gbrain agent run` — routed to Anthropic, because that was the only thing the model defaults knew how to name. It reported itself as a missing Anthropic key, which reads as "gbrain requires Anthropic" rather than "nothing told me what to use." + +Ollama and llama.cpp's `llama-server` now serve chat and query expansion as well as embeddings, with tool calling, so the subagent loop runs on them. The model defaults follow your configured chat model when there is no Anthropic credential, instead of naming a model you have no way to reach. The subagent loop routes non-Anthropic models through the provider-agnostic tool loop automatically — that loop has shipped since v0.38, but a config key stood between it and every brain that needed it. And a worker on a keyless brain now starts, where before it failed during startup regardless of which models its jobs targeted. + +Brains with an Anthropic key are unaffected: every default resolves exactly as it did. + +Remote Ollama and Ollama Cloud work through the same recipe via `OLLAMA_BASE_URL`. One thing worth knowing: Ollama model ids ending in `-cloud` or `:cloud` run on Ollama's servers even when your base URL points at the local daemon, because the daemon proxies them. The model id is the only thing that tells you, so it is worth reading before you pick one. + +Four hosted providers join the registry: xAI (Grok), Cerebras, Fireworks AI, and SambaNova. Fireworks serves embeddings as well as chat; the other three are chat-only and pair with an embedding provider. + +**`gbrain models autotune` gives each model tier a model sized for its job.** gbrain routes work through four tiers — classification, the default workhorse, expensive reasoning, and the tool loop. A keyless brain used to run all four on one model, which wastes a local fleet in both directions. autotune reads what you have pulled and assigns each tier, printing its reasoning and the context window it measured. It runs automatically during `gbrain init` when you pick an Ollama chat model, never overwrites a tier you set by hand, and is safe to re-run after pulling a model. Discovery happens once and writes config, so model resolution stays a pure config read with no network call in the path. + +### To take advantage of this release + +```bash +gbrain upgrade +``` + +To move a brain onto local models: + +```bash +ollama serve & ollama pull nomic-embed-text && ollama pull qwen3 +gbrain config set chat_model ollama:qwen3 +gbrain doctor +``` + +`gbrain doctor` now probes local endpoints directly — a stopped daemon is the failure mode no key check can catch. Read [docs/guides/local-models.md](docs/guides/local-models.md) first if you are doing this on a real brain; it covers the two settings most likely to bite you (context window and prompt caching) and what local costs you in retrieval quality. + +### For contributors + +`agent.use_gateway_loop` still exists and still means something — it opts *Anthropic* models into the gateway loop too. It is simply no longer what stands between a local model and a loop that already supported it. Cost estimates for Cerebras, Fireworks, and SambaNova report unknown rather than carrying a guessed rate, since none publishes a stable public per-token table; a fabricated rate corrupts a `--max-usd` gate silently, which is worse than an absent one. + ## [0.42.74.0] - 2026-08-07 **Two fixes for agents that reach a brain over the network: takes-holder visibility now works the way you set it, and the voice recipe is safe by default.** diff --git a/CLAUDE.md b/CLAUDE.md index e57f5214d2..efff7a025a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -115,6 +115,7 @@ detail on demand.) | any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry | | search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` | | search modes / cost knobs | `docs/guides/search-modes.md` | +| local / non-Anthropic models (ollama, llama-server, provider recipes) | `docs/guides/local-models.md` | | embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` | | push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` | | schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` | diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index db9896f42b..ce5032d865 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -119,7 +119,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; the threaded `'query'|'document'` crosses the SDK boundary via the module-level `__embedInputTypeStore` AsyncLocalStorage populated in `embedSubBatch()`, because the AI SDK's openai-compatible adapter strips `input_type` from `providerOptions` before building the wire body — #1400; `voyageCompatFetch` injects it opt-in the same way, and `openAICompatAsymmetricFetch` is the fallthrough shim for every other openai-compat recipe — llama-server/litellm/ollama — a strict pass-through when nothing was threaded) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). Provider-agnostic plumbing: `resolveNativeBaseUrl(provider, cfg)` normalizes a configured `ANTHROPIC_BASE_URL` / `OPENAI_BASE_URL` to carry the `/v1` suffix and is passed explicitly at every native `createAnthropic` / `createOpenAI` site (chat/expansion/embedding), so an env-injected bare host doesn't 404; returns `undefined` when unset so the SDK default is preserved (Google deferred until its native suffix is verified). `diagnoseEmbedding` fails closed with `user_provided_dims_unset` when a user-provided / zero-default recipe (litellm/llama-server) has no configured `embedding_dimensions` — this REPLACED the old `user_provided_model_unset` guard, which was structurally unreachable (parseModelId throws on a bare provider) and only ever false-positived for `litellm:`, silently disabling vector search. `configureGateway` no longer backfills `embedding_dimensions` (readers default it themselves), keeping the "no dims set" signal honest for that guard and the multimodal skip. - `src/core/ai/recipes/zeroentropyai.ts` — ZeroEntropy openai-compatible recipe declaring BOTH `embedding` (`zembed-1`, 7 Matryoshka dims: 2560/1280/640/320/160/80/40) AND `reranker` (`zerank-2` flagship + `zerank-1` + `zerank-1-small`, 5MB payload cap) touchpoints. `implementation: 'openai-compatible'` (pinned by regression in `test/ai/zeroentropy-recipe.test.ts`). `base_url_default: 'https://api.zeroentropy.dev/v1'` already ends with `/v1`, so the `zeroEntropyCompatFetch` URL rewrite `/embeddings → /models/embed` produces `…/v1/models/embed` (NOT `…/v1/v1/…` — pinned by regression). `chars_per_token: 1` + `safety_factor: 0.5` match Voyage's dense-content hedge. - `src/core/ai/recipes/llama-server-reranker.ts` — sibling of `llama-server` (the embedding recipe) for llama.cpp in `--reranking` mode. Distinct recipe rather than dual-touchpoint extension because `--reranking` and `--embeddings` are mutually exclusive at server-launch time, so the two backends need independent base URLs (default 8081 here vs 8080 there). Declares `reranker` touchpoint with `models: []` (user-provided id matching the `--alias` the user launched with), `path: '/rerank'` (leaf-only; consumes `RerankerTouchpoint.path` override; gateway concatenates with `base_url_default` which ends in `/v1`, producing `…/v1/rerank`), `default_timeout_ms: 30_000` (consumed by `src/core/search/mode.ts`'s reranker timeout chain — CPU-only first-call warmup headroom; the 5s mode-bundle default would fail-open as `timeout`), `cost_per_1m_tokens_usd: 0` (recognized by `FREE_LOCAL_RERANK_PROVIDERS` in `src/core/budget/budget-tracker.ts` so `--max-cost` callers don't hard-fail on local rerank). Setup hint emphasizes `--alias` because llama-server's `/v1/models` defaults model id to the gguf file path without it. Covers Qwen3-Reranker via llama.cpp AND self-hosted ZE weights via llama.cpp — same recipe, different `--model` at launch. Pinned by `test/ai/recipe-llama-server-reranker.test.ts`. Voyage / Cohere / vLLM rerankers stay out of scope (different wire shapes). Same wave adds: `path?: string` + `default_timeout_ms?: number` on `RerankerTouchpoint` in `src/core/ai/types.ts`; consumed by the URL build at `src/core/ai/gateway.ts:rerank()` and by mode-resolution at `src/core/search/mode.ts:resolveSearchMode` (precedence: per-call > config-key > recipe touchpoint default > mode bundle); `LLAMA_SERVER_RERANKER_BASE_URL` env passthrough in `src/cli.ts:buildGatewayConfig`; `FREE_LOCAL_RERANK_PROVIDERS` set in `src/core/budget/budget-tracker.ts:lookupPricing` (rerank-kind-only zero-pricing for the local provider prefix); doctor-fix at `src/commands/models.ts:probeRerankerConfig` reads `search.reranker.model` via `loadSearchModeConfig` + `resolveSearchMode` (closes file-plane / DB-plane divergence where doctor said "not configured" while live search was actively reranking — the field-plane `getRerankerModel()` read nothing writes); `probeRerankerReachability` reads the recipe's `default_timeout_ms` so CPU-only cold-start doesn't false-fail. -- `src/core/ai/recipes/openrouter.ts` — OpenRouter openai-compatible recipe: single key, many providers via `openrouter:/` strings. `base_url_default: 'https://openrouter.ai/api/v1'`. Embedding touchpoint: `openai/text-embedding-3-small` at 1536 dims with Matryoshka `dims_options: [512, 768, 1024, 1536]`; `max_batch_tokens: 300_000` = OpenAI's aggregate-per-request token cap (NOT per-input). Chat touchpoint declares 8 curated entry points (gpt-5.2, gpt-5.2-chat, gpt-5.5, claude-haiku-4.5, claude-sonnet-4.6, claude-opus-4.7, gemini-3-flash-preview, deepseek-chat) but openai-compat tier accepts any model ID; deliberately no `max_context_tokens` because OR's catalog spans 128K to 1M+. `supports_subagent_loop: false` is INFORMATIONAL — the real gate is `isAnthropicProvider()` in `src/core/model-config.ts` which hard-pins gbrain's subagent infra to Anthropic-direct. Declares `resolveDefaultHeaders(env)` returning OR's three attribution headers: `HTTP-Referer` (required for OR app-attribution), `X-OpenRouter-Title` (preferred), `X-Title` (back-compat alias); defaults to `https://gbrain.ai` / `gbrain`; forks override via `OPENROUTER_REFERER` / `OPENROUTER_TITLE` env vars. Smoke-tested by `test/ai/recipe-openrouter.test.ts` (incl. the shape-test regression guard: every model in the chat list matches `^[a-z0-9-]+\/[a-z0-9._-]+$`). +- `src/core/ai/recipes/openrouter.ts` — OpenRouter openai-compatible recipe: single key, many providers via `openrouter:/` strings. `base_url_default: 'https://openrouter.ai/api/v1'`. Embedding touchpoint: `openai/text-embedding-3-small` at 1536 dims with Matryoshka `dims_options: [512, 768, 1024, 1536]`; `max_batch_tokens: 300_000` = OpenAI's aggregate-per-request token cap (NOT per-input). Chat touchpoint declares 8 curated entry points (gpt-5.2, gpt-5.2-chat, gpt-5.5, claude-haiku-4.5, claude-sonnet-4.6, claude-opus-4.7, gemini-3-flash-preview, deepseek-chat) but openai-compat tier accepts any model ID; deliberately no `max_context_tokens` because OR's catalog spans 128K to 1M+. `supports_subagent_loop: false` is INFORMATIONAL — the operative gate is `classifyCapabilities()` in `src/core/ai/capabilities.ts`, consulted by the subagent queue and handler; a non-Anthropic model that clears it auto-routes to the provider-agnostic gateway tool loop. Declares `resolveDefaultHeaders(env)` returning OR's three attribution headers: `HTTP-Referer` (required for OR app-attribution), `X-OpenRouter-Title` (preferred), `X-Title` (back-compat alias); defaults to `https://gbrain.ai` / `gbrain`; forks override via `OPENROUTER_REFERER` / `OPENROUTER_TITLE` env vars. Smoke-tested by `test/ai/recipe-openrouter.test.ts` (incl. the shape-test regression guard: every model in the chat list matches `^[a-z0-9-]+\/[a-z0-9._-]+$`). - `src/core/rerank-audit.ts` — failure-only JSONL audit at `~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation, mirrors `src/core/audit-slug-fallback.ts`). Exports `logRerankFailure({reason, model, query_hash, doc_count, error_summary})` + `readRecentRerankFailures(days)`. Deliberately no `logRerankSuccess`: writing once per tokenmax search is hot-path I/O churn AND success events leak query volume + timing into a local audit file. `gbrain doctor`'s `reranker_health` check reads `search.reranker.enabled` first so "no events in window" is interpreted correctly (disabled → ok; enabled → ok). Query text SHA-256-prefix-hashed (8 hex chars) for privacy. `GBRAIN_AUDIT_DIR` env override honored via the shared `resolveAuditDir()`. - `src/core/search/embedding-column.ts` — single source of truth for "which `content_chunks.*` column does this query rank against?" Pure functions, no engine I/O: `loadRegistry(cfg)` walks the `embedding_columns` config (DB plane, JSON map keyed by column name with `{provider, dimensions, type}` entries), seeds the OpenAI `embedding` builtin when unset, validates everything before it lands (column-name regex, type ∈ `vector | halfvec`, dims in [1, 8192], provider format) using `Object.create(null)` + `Object.hasOwn` so a key like `constructor` rejects instead of resolving to `Object.prototype.constructor`. `resolveColumn(registry, override?, cfg)` is the boundary call: returns a frozen `ResolvedColumn` descriptor (`{name, provider, dimensions, type}`) honoring per-call override → `search_embedding_column` config → `'embedding'` default; throws `UnknownEmbeddingColumnError` with the list of registered names on miss. `isCacheSafe(resolved, cfg)` compares the full embedding SPACE (provider + dimensions + name) against cfg's default so a repointed `embedding` builtin doesn't serve OpenAI-shaped cache rows. `validateResolvedColumn(descriptor)` re-validates hand-rolled descriptors that bypass the registry (internal-SDK passthrough) so the SQL-injection escape hatch through the descriptor field is closed. Consumed by `hybridSearch`, `gateway.embedQuery(text, {embeddingModel, dimensions})`, `cosineReScore`, and the `query` MCP op (per-call `embedding_column` param). Pinned by `test/search/embedding-column.test.ts` (prototype-pollution, descriptor passthrough, env-only Postgres install, empty-brain coverage gate, cache-space comparison). - `src/core/search/rerank.ts` — the call-site abstraction. `applyReranker(query, results, opts)` slots between `dedupResults()` and `enforceTokenBudget()` in `src/core/search/hybrid.ts`. Slices `opts.topNIn` (default 30) by current RRF order, sends to `gateway.rerank()`, reorders by `relevanceScore` desc, appends the un-reranked tail unchanged (recall protection). Fail-open on every `RerankError.reason`: any error logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items so downstream telemetry sees the new ordering signal. `topNOut: null` is the explicit "don't truncate" signal — semantically distinct from `undefined` ("fall through to mode bundle"). Test seam: `opts.rerankerFn` stubs `gateway.rerank` without the network. @@ -136,7 +136,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/diarize/payload-fitter.ts` — generic fit-arbitrarily-large-items-into-per-call-token-budget utility. `'batch'` strategy is deterministic token-budgeted chunking with no LLM calls. `'summarize'` strategy embed-clusters into ceil(items/4) groups via cheap deterministic nearest-neighbor on cosine, Haiku-summarizes each cluster via `Promise.allSettled` at parallelism=4. Each Haiku call composes the active BudgetTracker via the AsyncLocalStorage. Quality gate: when `success_ratio < min_success_ratio` (default 0.75), result is flagged `degraded: true` — the fitter preserves the successful subset; the caller decides whether to surface a partial result or abort. - `src/core/brainstorm/checkpoint.ts` — crash-resilient checkpoint for `gbrain brainstorm` and `gbrain lsd`. Persists FULL idea bodies (~50KB/run) so resume MERGES pre-crash ideas with post-resume ideas before the judge runs (a resume that produces only second-run output is silent partial output). `run_id = sha256(question + profile + sort(close_slugs) + sort(far_slugs)).slice(0,16)` — NO embedding bits, stable across embedding-model swaps. Atomic write via `.tmp + rename`. ONE resume flag (`--resume ` covers both failed AND never-attempted crosses); `--list-runs` prints run_ids mtime-newest-first; `--force-resume` bypasses the 7-day staleness gate. Cycle purge phase (`gbrain dream --phase purge`) GCs checkpoints older than 7 days via `gcStaleCheckpoints(7)`. Pinned by `test/e2e/brainstorm-resume.test.ts` (20 unit + 3 E2E cases incl. the merge contract). - `src/core/remediation-checkpoint.ts` — `doctor --remediate` checkpoint at `~/.gbrain/remediation/.json`. `plan_hash = sha256(JSON.stringify(sorted recommendation ids)).slice(0,16)`. Schema-versioned, atomic `.tmp + rename`. `gbrain doctor --remediate --resume ` (no arg picks newest matching) loads it and skips completed steps. Mismatched plan_hash refuses with a paste-ready message. Cleared on clean completion. Pinned by 13 unit cases. -- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). Four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. 8-step resolution chain: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern (routes through `splitProviderModelId` from `src/core/model-id.ts` so slash-form ids like `anthropic/claude-sonnet-4-6` classify correctly). `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves non-Anthropic, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` (the Anthropic Messages API tool-loop can't run on OpenAI/Gemini). `_resetDeprecationWarningsForTest()` also clears `_subagentTierWarningsEmitted`. Pinned by `test/model-config.serial.test.ts`. +- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). Four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. 8-step resolution chain: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `resolveTierDefault(tier)` → caller fallback. `resolveTierDefault(tier)` is the step-7 fallback: it returns `TIER_DEFAULTS[tier]` whenever an Anthropic credential resolves (`hasAnthropicKey()` — env OR config file), so keyed brains are unchanged; with NO Anthropic credential it returns the config file's `chat_model` instead, because an `anthropic:*` default is a guaranteed failure for a brain deliberately run on a local or non-Anthropic provider. Falls back to `TIER_DEFAULTS` when `chat_model` is absent or is itself Anthropic (the honest missing-key error beats a substitution), and never probes — readiness is the doctor's job, not the resolver's. Emits a once-per-`(tier, model)` stderr notice naming the `models.tier.` key to pin. `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern (routes through `splitProviderModelId` from `src/core/model-id.ts` so slash-form ids like `anthropic/claude-sonnet-4-6` classify correctly). `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves non-Anthropic, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` (the Anthropic Messages API tool-loop can't run on OpenAI/Gemini). `_resetDeprecationWarningsForTest()` also clears `_subagentTierWarningsEmitted`. Pinned by `test/model-config.serial.test.ts`. - `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` takes an optional 4th `extendedModels: ReadonlySet`: when the modelId is in that set the native-recipe allowlist throw is bypassed (user explicitly opted in via config, so provider rejection surfaces as `model_not_found` at HTTP call time and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — source typos still fail fast (the fail-fast contract for chat + expand + embed stays intact). - `src/core/ai/gateway.ts` extension — module-scoped `_extendedModels: Map>>` registry feeds `assertTouchpoint`'s extended-model path without broadening unrelated surfaces. `reconfigureGatewayWithEngine(engine)` (async, called from `cli.ts` after `engine.connect()`, before every command except `CLI_ONLY` no-DB commands) re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to both. `registerConfigSelectedChatModel(model)` is the narrow runtime seam for a dedicated contextual-synopsis model: the ID joins the chat allowlist but remains rejected for embedding, expansion, and reranking. `DEFAULT_CHAT_MODEL` is `anthropic:claude-sonnet-4-6`. `__setChatTransportForTests` mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport. - `src/core/minions/queue.ts` extension — `MinionQueue.add()` rejects `subagent` jobs whose `data.model` resolves via `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (layers 2+3: `model-config.ts:enforceSubagentAnthropic` runtime fallback + `src/commands/doctor.ts` `subagent_provider` check). Pinned by `test/agent-cli.test.ts`. @@ -262,7 +262,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values. - `src/core/minions/handlers/supervisor-audit.ts` — supervisor lifecycle JSONL audit at `~/.gbrain/audit/supervisor-YYYY-Www.jsonl` (ISO-week rotation; shares `computeIsoWeekName()` with `shell-audit.ts`). `writeSupervisorEvent(emission, supervisorPid)` appends one line per event (`started`, `worker_spawned`, `worker_exited`, `backoff`, `health_warn`, `health_error`, `max_crashes_exceeded`, `shutting_down`, `stopped`, `worker_spawn_failed`). `readSupervisorEvents({sinceMs})` is the readback for `gbrain doctor`. Exports `isCrashExit(event)`, `summarizeCrashes(events)`, `CrashSummary` type, and `CLEAN_EXIT_CAUSES` denylist (`'clean_exit' | 'graceful_shutdown'`). Single regression point — both `gbrain doctor` (supervisor check at `doctor.ts:1011-1043`) and `gbrain jobs supervisor status` (`jobs.ts:803-826`) import from here so the two surfaces can't drift. `isCrashExit` classifies a single `worker_exited` against the denylist: clean/graceful are NON-crashes; everything else (incl. any future `likely_cause` from `child-worker-supervisor.ts`) is a crash; audit lines lacking `likely_cause` fall back to `code !== 0`. `summarizeCrashes` returns `{total, by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}, clean_exits}` — the `legacy` bucket catches both old fallback entries AND unrecognized future causes (fail-loud, not silent underreport); denylist-over-allowlist is deliberate. Pinned by `test/supervisor-audit.test.ts` (14 cases) and 4 source-grep wiring assertions in `test/doctor.test.ts`. - `src/core/minions/backpressure-audit.ts` — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. One line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the maxWaiting guard introduced. -- `src/core/minions/handlers/subagent.ts` — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Per-turn output cap resolves via `resolveMaxOutputTokens` (`data.max_tokens` → `agent.max_output_tokens` config → 8192 default); a `stop_reason: 'max_tokens'` final turn surfaces as `SubagentStopReason 'max_tokens'` (not a silent `end_turn`), and a max_tokens stop mid-tool-round injects a truncation note into the tool-result turn so the model re-issues the dropped call. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times. Catches both initial-prompt overflow and turn-N tool-loop accumulation that `synthesize.ts`'s chunker can't bound ahead of time. +- `src/core/minions/handlers/subagent.ts` — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. The default Anthropic client is built LAZILY on first legacy-path use and must stay that way — `new Anthropic({apiKey: undefined})` throws in the SDK constructor, so building it at factory (worker-registration) time made a brain with no Anthropic credential unable to register the handler at all, even when every job it would run targeted a local model through the gateway loop. Routing: `useGatewayLoop = agent.use_gateway_loop || !isAnthropicProvider(model)` — a non-Anthropic model auto-routes to the provider-agnostic `gateway.toolLoop()` (the legacy path calls the Anthropic Messages API directly and could never run it), with a once-per-model stderr notice; the config flag remains meaningful as the opt-in that routes ANTHROPIC models through the gateway loop too. `classifyCapabilities()` at handler entry still rejects tool-less models and unknown providers, so auto-routing can only reach a provider the loop can drive. Per-turn output cap resolves via `resolveMaxOutputTokens` (`data.max_tokens` → `agent.max_output_tokens` config → 8192 default); a `stop_reason: 'max_tokens'` final turn surfaces as `SubagentStopReason 'max_tokens'` (not a silent `end_turn`), and a max_tokens stop mid-tool-round injects a truncation note into the tool-result turn so the model re-issues the dropped call. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times. Catches both initial-prompt overflow and turn-N tool-loop accumulation that `synthesize.ts`'s chunker can't bound ahead of time. - `src/core/minions/handlers/subagent-aggregator.ts` — `subagent_aggregator` handler. Claims AFTER all children resolve (queue guarantees every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds a deterministic mixed-outcome markdown summary. No LLM call. - `src/core/minions/handlers/subagent-audit.ts` — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback for `gbrain agent logs`. - `src/core/minions/rate-leases.ts` — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms). diff --git a/docs/guides/local-models.md b/docs/guides/local-models.md new file mode 100644 index 0000000000..8360477c44 --- /dev/null +++ b/docs/guides/local-models.md @@ -0,0 +1,276 @@ +# Running gbrain on local models + +You can run a whole brain — ingest, embed, search, `think`, and the subagent +tool loop — with no hosted API key anywhere. This guide covers how, and what +you trade away. + +## The short version + +```bash +# 1. Serve the models. Two pulls: one embedder, one chat model with tool calling. +ollama serve & +ollama pull nomic-embed-text +ollama pull qwen3 + +# 2. Point every touchpoint at Ollama. +gbrain init --pglite \ + --embedding-model ollama:nomic-embed-text \ + --embedding-dimensions 768 \ + --chat-model ollama:qwen3 \ + --expansion-model ollama:qwen3 + +# 3. Confirm the daemon is actually reachable and the wiring resolves. +gbrain doctor +``` + +`gbrain init` will not auto-pick a local provider for you. That is deliberate: +picking Ollama silently when you have `OPENAI_API_KEY` set is a +silent-broken-state class — the daemon may not be running, and you probably +meant the hosted provider. Local providers are always available by explicit +flag or through the interactive picker. + +## Why this needed a fix + +gbrain has shipped Ollama support since v0.32, but only for **embeddings**. The +`ollama` recipe declared one touchpoint. So a local install could index a brain +and search it, and then every synthesis step — `gbrain think`, query expansion, +`gbrain dream`, `gbrain agent run` — routed to Anthropic, because that was the +only thing the tier defaults knew how to name. The failure surfaced as +`NO_ANTHROPIC_API_KEY`, which reads as *gbrain requires Anthropic* rather than +*nothing told me what to use*. + +Four separate things had to change for the local path to be real. They are +worth knowing about because each one is a place the behavior could regress: + +1. **`ollama` and `llama-server` now declare `chat` and `expansion` + touchpoints**, with tool calling. Without a chat touchpoint the gateway + refuses to route synthesis to them at all. +2. **Tier defaults consult your `chat_model`.** The resolution chain's last + step used to hardcode `anthropic:*`. It still does when you have an + Anthropic key — behavior for existing brains is unchanged — but with no key + it now falls through to whatever `chat_model` you configured. +3. **The subagent loop auto-routes non-Anthropic models.** gbrain has had a + provider-agnostic tool loop since v0.38, gated behind + `agent.use_gateway_loop`. A non-Anthropic model used to be *refused* with a + pointer to that config key. It is now routed through the gateway loop + automatically, because the legacy Anthropic-direct path could never have run + it anyway. The flag still exists and still means something: it opts + *Anthropic* models into the gateway loop too. +4. **The Anthropic SDK client is constructed lazily.** It used to be built when + the worker registered its handlers, and the SDK constructor throws on a + missing key — so `gbrain jobs work` died at startup on a keyless brain + before any routing decision was reached. + +## Giving each tier the right model + +gbrain routes work through four model tiers — `utility` (classification, +verdicts), `reasoning` (the default workhorse), `deep` (expensive reasoning), +and `subagent` (the tool loop). With no per-tier config, a keyless brain runs +all four on your `chat_model`, which wastes the fleet in both directions: a 3B +model is the right classifier and the wrong deep reasoner. + +`gbrain models autotune` reads what you have pulled and assigns each tier: + +``` +$ gbrain models autotune +Local fleet at http://localhost:11434/v1: 14 model(s), 7 tool-capable. + + ✔ utility ollama:llama3.2:latest 128k ctx 2.0GB, smallest tool-capable without thinking + ✔ reasoning ollama:gpt-oss:20b 128k ctx 13.8GB, runner-up by size + ✔ deep ollama:qwen3.6:35b-mlx 128k ctx 21.9GB, largest tool-capable + ✔ subagent ollama:gpt-oss:20b 128k ctx 13.8GB, same workhorse +``` + +It runs automatically during `gbrain init` when you pick an Ollama chat model, +so a fresh local brain is tiered without a second command. Re-run it after +pulling a new model. `--dry-run` previews; `--json` is machine-readable. + +**It never overwrites a tier you set by hand.** A hand-tuned tier is a +decision, so re-running after a pull is safe. Use `--force` to overwrite. + +**Discovery runs once and writes config.** Model resolution stays a pure config +read — putting a probe in the resolution path would add a network round-trip to +every unconfigured call, which is worse than the flat defaults it fixes. + +### What the ranking is, and is not + +Ranking is by on-disk size, which is a proxy for capability, not a quality +measure — a 9B model tuned for reasoning may well beat a 12B generalist. That's +why every assignment is printed with its justification and stays overridable: + +```bash +gbrain config set models.tier.deep ollama:your-preferred-model +``` + +Size specifically means *bytes*, not parameter count: quantized and MLX builds +frequently report no parameter count at all, so a parameter-based sort silently +drops them to the bottom of the fleet. + +Two tier rules are deliberate. `utility` prefers the smallest model **without** +a `thinking` capability — classification returns a label, and reasoning tokens +spent on it are pure overhead. `reasoning` takes the runner-up rather than the +largest, so deep-tier latency isn't paid on every ordinary call. + +### Why capability detection matters more than it sounds + +Ollama reports per-model capabilities, and several **embedding** models +advertise `tools` while lacking `completion` — `qwen3-embedding:8b` reports +`[tools,embedding]`. At 7.6B it outranks most genuine chat models by size, so +selecting on `tools` alone lands a model that cannot generate text into a +reasoning tier. autotune requires `completion` **and** `tools`, and prints +every rejected model with the reason: + +``` + Not eligible for chat tiers: + - qwen3-embedding:8b embedding model (advertises tools but has no completion) +``` + +`llama-server` and `litellm` are not autotuned: the former serves one model +chosen at launch, and the latter proxies opaque backends with no capability +API. Both keep the single-model behavior. + +## What you give up + +Be clear-eyed about this. Local is not free of cost, it just moves the cost. + +**Prompt caching.** Anthropic's ephemeral cache markers are what keep a long +multi-turn subagent loop cheap. No other provider in the registry honors them, +so every turn re-sends the whole conversation. On a hosted non-Anthropic +provider that shows up as a linear cost increase. On a local model it shows up +as latency — a 20-turn loop re-processes a growing prompt 20 times. + +**Context window.** The `ollama` and `llama-server` recipes declare a +conservative 4096-token context, because that is the un-tuned default for both. +That is *small* — smaller than a single `tokenmax` search payload. Raise the +daemon, then raise gbrain's budget to match: + +```bash +# Ollama: raise the daemon's default, then tell gbrain about it. +OLLAMA_CONTEXT_LENGTH=32768 ollama serve +gbrain config set search.token_budget 12000 +``` + +Leaving these mismatched is the most common way a local setup produces +plausible-but-truncated answers: the daemon silently drops the front of the +prompt, which is where the retrieved context lives — no error is raised +anywhere. + +`gbrain models autotune` reports the context it measured per tier, so you can +see the real number rather than the recipe's conservative default. It reads the +length the daemon actually **serves** (which is `min(trained length, OLLAMA_CONTEXT_LENGTH)`), +not what the model was trained for — those differ by 32× on an un-tuned daemon, +and only the served value predicts truncation. The reading requires a chat +model to be resident, so run one query first if autotune reports it as unknown. + +**Tool-calling quality.** Declaring `supports_tools: true` on the recipe says +the *server* speaks the protocol, not that your model uses it well. Small +models (4B and under) frequently emit malformed tool calls, re-call the same +tool in a loop, or ignore tools entirely. If `gbrain agent run` spins, try a +larger model before assuming a gbrain bug. + +**Embedding quality.** `nomic-embed-text` at 768 dimensions is a real step down +from `voyage-3` or `text-embedding-3-large` on retrieval benchmarks. A common +middle path is local chat with hosted embeddings — embeddings are computed once +per document and are the cheaper half of the bill. + +## Remote and cloud Ollama + +The same recipe covers three deployments: + +| Setup | Configuration | +|---|---| +| Local daemon | nothing — `http://localhost:11434/v1` is the default | +| Remote daemon | `OLLAMA_BASE_URL=http://gpu-box:11434/v1` (add `OLLAMA_API_KEY` if it sits behind an authenticating proxy) | +| Ollama Cloud | `OLLAMA_BASE_URL=https://ollama.com/v1` + `OLLAMA_API_KEY` | + +**One trap worth stating plainly:** Ollama model ids ending in `-cloud` or +`:cloud` run on Ollama's servers *even when your base URL points at the local +daemon* — the daemon proxies them. Picking one sends your brain's contents +off-device. That is a privacy decision, not a performance one, and the model id +is the only thing that tells you. + +## llama.cpp (`llama-server`) + +`llama-server` serves **one model per process**. Running embeddings and chat +both locally through llama.cpp therefore means two processes on two ports: + +```bash +./llama-server --model embed.gguf --embeddings --port 8080 & +./llama-server --model chat.gguf --jinja --port 8081 & + +gbrain config set base_urls.llama-server http://localhost:8080/v1 +``` + +`--jinja` is required for tool calling — it loads the model's chat template, +which is what emits `tool_calls`. Without it the subagent loop gets a model +that can never call a tool. + +Because gbrain resolves each touchpoint independently, the more common setup is +mixing: llama.cpp for chat (where you want a specific GGUF) and Ollama for +embeddings (where you want convenience), or either one for chat with hosted +embeddings. + +## Hosted non-Anthropic providers + +Everything above about the tier-default and subagent-routing fixes applies +equally to hosted providers that aren't Anthropic. A brain on DeepSeek, xAI, or +Cerebras hit the exact same dead end. The registry now covers: + +| Provider | id | key | chat | embeddings | +|---|---|---|---|---| +| xAI (Grok) | `xai` | `XAI_API_KEY` | yes | no | +| Cerebras | `cerebras` | `CEREBRAS_API_KEY` | yes | no | +| Fireworks AI | `fireworks` | `FIREWORKS_API_KEY` | yes | yes | +| SambaNova | `sambanova` | `SAMBANOVA_API_KEY` | yes | no | + +Plus the previously-shipped OpenAI, Google, DeepSeek, Groq, Together, Mistral, +Moonshot, Zhipu, MiniMax, NVIDIA, Perplexity, OpenRouter, Azure OpenAI, and the +LiteLLM proxy escape hatch. Run `gbrain providers list` for the live registry — +that command reads the same recipes this table was written from, so it cannot +drift. + +Fireworks is the only one of the four new providers that can serve a whole +brain alone; the other three ship no embedding model and need to be paired. + +### Providers deliberately not added + +Three more were considered and left out, because each needs transport work +rather than a recipe — a recipe is pure data over an OpenAI-compatible +endpoint, and these are not that: + +- **AWS Bedrock** — requests are SigV4-signed against a regional host. That is + a new `implementation` in the gateway's factory switch, not a `base_url`. +- **GitHub Copilot** — auth is an OAuth device flow with short-lived tokens, so + there is no static key for `auth_env` to name. +- **poolside** — access is enterprise-gated; the endpoint contract could not be + verified against public documentation, and shipping an unverifiable recipe + is how stale model ids and wrong prices get in. + +All three are reachable today through the `litellm` recipe, which is the +existing escape hatch for exactly this: run LiteLLM in front of them and point +gbrain at the proxy. + +### A note on cost estimates + +`cerebras`, `fireworks`, and `sambanova` do not publish stable per-1M-token +rates in a public table, so their recipes report cost as **unknown** rather +than carrying a guess. `--max-usd` pre-flights and `est_cost_usd` audit rows +will show no estimate for these providers. That is intentional: a fabricated +rate corrupts a budget gate silently, which is worse than an absent one. + +xAI publishes rates, and the recipe carries the sub-200k-prompt tier. xAI +doubles its rate above a 200k-token prompt; gbrain search payloads top out +around 20k, so the recorded tier is the one you will actually pay — unless you +are deliberately stuffing enormous contexts, in which case estimates run 2x low. + +## Verifying it works + +```bash +gbrain doctor # probes the local endpoint; warns if unreachable +gbrain providers list # confirms which recipes are ready +gbrain think "what did I write about retrieval?" +``` + +`gbrain doctor`'s `subagent_capability` check probes local endpoints directly. +A stopped daemon is the failure mode that no API-key check can catch, and it is +the single most common cause of a local brain that "stopped working". diff --git a/docs/integrations/embedding-providers.md b/docs/integrations/embedding-providers.md index ffd8114bf1..8b03b176b4 100644 --- a/docs/integrations/embedding-providers.md +++ b/docs/integrations/embedding-providers.md @@ -4,6 +4,12 @@ GBrain ships with 16 embedding-provider recipes covering OpenAI, ZeroEntropy, Vo This page is the human-readable counterpart: capability per provider, env-var setup, dimensions, cost, and known constraints. +> **Embeddings are one touchpoint of four.** A provider that embeds does not +> necessarily chat, and vice versa. If you are trying to run a brain with no +> hosted API key at all, read **[Running gbrain on local models](../guides/local-models.md)** +> — it covers the chat / expansion / subagent side, which is where the hosted +> dependency actually used to live. + ## Quick start ``` @@ -34,6 +40,7 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom | `zhipu` | `ZHIPUAI_API_KEY` | 1024 | varies | no | no | | `ollama` | (none — runs locally) | 768 | 0 | yes | no | | `llama-server` | (none — runs locally) | user-set | 0 | yes | no | +| `fireworks` | `FIREWORKS_API_KEY` | 768 (Matryoshka to 512/256/128/64) | varies | no | no | | `litellm` | `LITELLM_API_KEY` (optional) | user-set | varies | yes (proxy) | yes (backend permitting) | | `together` | `TOGETHER_API_KEY` | 768 | varies | no | no | | `anthropic` | (no embedding model — chat only) | — | — | — | — | diff --git a/llms-full.txt b/llms-full.txt index b89b90a309..e173822c62 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -264,6 +264,7 @@ detail on demand.) | any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry | | search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` | | search modes / cost knobs | `docs/guides/search-modes.md` | +| local / non-Anthropic models (ollama, llama-server, provider recipes) | `docs/guides/local-models.md` | | embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` | | push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` | | schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` | diff --git a/src/commands/agent.ts b/src/commands/agent.ts index deebbfc323..f973e91811 100644 --- a/src/commands/agent.ts +++ b/src/commands/agent.ts @@ -67,8 +67,8 @@ SUBMITTING gbrain agent run --subagent-def Named plugin subagent (from GBRAIN_PLUGIN_PATH) --model Model id as provider:model (default: subagent tier model, - anthropic:claude-sonnet-4-6). Non-Anthropic providers need - agent.use_gateway_loop enabled — see NOTES below. + anthropic:claude-sonnet-4-6). Non-Anthropic providers run + on the gateway loop automatically — see NOTES below. --max-turns Max assistant turns (default 20) --tools a,b,c Subset of registered tool names (comma list) --timeout-ms Per-job wall-clock timeout @@ -92,17 +92,21 @@ NOTES This CLI path is trusted-only. (Remote MCP callers reach subagents through the scoped submit_agent operation, not through this command.) - By default the worker runs the legacy Anthropic-direct path, which needs an + An Anthropic --model runs the legacy Anthropic-direct path, which needs an Anthropic key — from ANTHROPIC_API_KEY or from anthropic_api_key in ~/.gbrain/config.json — or the first LLM turn of a claimed job fails. - To run --model on a non-Anthropic provider, enable the provider-neutral - gateway loop first, then supply whatever credential that provider needs - (an API key for most; some recipes use OAuth or a local endpoint): + A non-Anthropic --model runs the provider-neutral gateway loop instead, with + no extra configuration: the Anthropic-direct path could not have run it. You + still supply whatever credential that provider needs (an API key for most; + local recipes like ollama and llama-server need none, just a reachable + endpoint). + + To route ANTHROPIC models through the gateway loop as well: gbrain config set agent.use_gateway_loop true Accepted values: true / 1 / yes / on. - The gateway loop needs a provider whose recipe supports chat WITH tool + Either way the loop needs a provider whose recipe supports chat WITH tool calling — not every recipe under src/core/ai/recipes/ qualifies. A model that cannot call tools is refused at job start with the reason named. `); diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index e7c4465790..3bd7b045dc 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -3125,33 +3125,45 @@ export async function checkSubagentCapability(engine: BrainEngine): Promise null); - const gatewayLoopEnabled = isConfigTruthy(gatewayLoopRaw); const { isAnthropicProvider } = await import('../core/model-config.ts'); - if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY && !gatewayLoopEnabled) { - return { - name: 'subagent_capability', - status: 'warn', - message: - `chat_model is "${chatModel}" (non-Anthropic) and ANTHROPIC_API_KEY is not set. ` + - `Subagent features (gbrain dream, gbrain agent run, gbrain autopilot) will fail at job submission ` + - `unless agent.use_gateway_loop=true. Chat alone (gbrain think) still works. ` + - `Either set ANTHROPIC_API_KEY or enable: \`gbrain config set agent.use_gateway_loop true\`.`, - }; + if (chatModel && !isAnthropicProvider(chatModel)) { + const { resolveRecipe } = await import('../core/ai/model-resolver.ts'); + const { recipe } = resolveRecipe(chatModel); + // Only local-server recipes carry a probe. Hosted providers surface + // auth/availability at call time with a clear provider error. + if (recipe.probe) { + // `provider_base_urls` is the config-file field; the gateway merges + // it over the `*_BASE_URL` env vars into its own `base_urls`. Pass + // the same resolved URL the gateway would use, else the probe checks + // localhost while live traffic goes to a remote daemon. + const baseURL = cfg?.provider_base_urls?.[recipe.id]; + const verdict = await Promise.race([ + recipe.probe(baseURL), + new Promise<{ ready: boolean; hint?: string }>(resolve => + setTimeout(() => resolve({ ready: false, hint: 'probe timed out after 2s' }), 2000)), + ]).catch(() => ({ ready: false, hint: 'probe threw' })); + if (!verdict.ready) { + return { + name: 'subagent_capability', + status: 'warn', + message: + `chat_model is "${chatModel}" but its endpoint is not reachable. ` + + `${verdict.hint ?? ''} ` + + `Subagent features (gbrain dream, gbrain agent run, gbrain autopilot) will fail until it is.`, + }; + } + } } - } catch { /* loadConfig may throw; fall through */ } + } catch { /* loadConfig / resolveRecipe may throw; fall through to ok */ } return { name: 'subagent_capability', diff --git a/src/commands/init-autotune.ts b/src/commands/init-autotune.ts new file mode 100644 index 0000000000..60cf4bf09e --- /dev/null +++ b/src/commands/init-autotune.ts @@ -0,0 +1,46 @@ +/** + * Install-time tier assignment for local brains. + * + * Fires from `gbrain init` after `initSchema()` (so config writes are valid) + * when the chosen chat model is an Ollama one. Without it, a fresh local brain + * runs every tier on one model until the user discovers `gbrain models + * autotune` — and most never would, because nothing points at it. + * + * Deliberately narrow: + * - Ollama only. It is the sole local provider exposing per-model + * capabilities; llama-server serves one model and LiteLLM proxies opaque + * backends, so neither has a fleet to rank. + * - Never overwrites a tier the user already set (autotune's own rule). + * - Fail-open. A daemon that is unreachable at init time is normal — people + * configure before starting `ollama serve`. It prints the one-line fix and + * moves on; init must not fail over an optimization. + */ + +import type { BrainEngine } from '../core/engine.ts'; + +export async function maybeAutotuneLocalTiers( + engine: BrainEngine, + opts: { chatModel?: string; jsonOutput?: boolean }, +): Promise { + const chat = opts.chatModel?.trim(); + if (!chat || !chat.startsWith('ollama:')) return; + + try { + const { runModelsAutotune } = await import('./models.ts'); + if (!opts.jsonOutput) { + process.stderr.write('\n[init] Assigning model tiers from your local Ollama fleet…\n'); + } + // exitOnError:false is load-bearing. The CLI path exits non-zero when the + // daemon is unreachable; inheriting that here would abort `gbrain init` + // for the most ordinary reason imaginable — configuring gbrain before + // starting `ollama serve`. + await runModelsAutotune(engine, { json: opts.jsonOutput, exitOnError: false }); + } catch (e) { + if (!opts.jsonOutput) { + process.stderr.write( + `[init] Could not assign per-tier models (${e instanceof Error ? e.message : String(e)}). ` + + `Every tier will use ${chat}. Run \`gbrain models autotune\` once Ollama is up.\n`, + ); + } + } +} diff --git a/src/commands/init-provider-picker.ts b/src/commands/init-provider-picker.ts index f2d851a476..eed779201f 100644 --- a/src/commands/init-provider-picker.ts +++ b/src/commands/init-provider-picker.ts @@ -49,22 +49,38 @@ export interface PickProviderOpts { } /** - * Surface the subagent-Anthropic caveat (D7) when the user picks a - * non-Anthropic chat-capable recipe without `ANTHROPIC_API_KEY` set. + * Surface what running on a non-Anthropic chat model means, when the user + * picks one without `ANTHROPIC_API_KEY` set. * - * Exported so `initPGLite` can reuse the same message in its post-init - * stderr summary path (auto-pick branch doesn't run the picker but still - * needs to surface the caveat). One source of truth keeps the message - * format aligned across the three D7 surfaces (picker / init summary / - * doctor). + * This used to say subagent features "require ANTHROPIC_API_KEY regardless of + * which chat model you pick". That is no longer true — the subagent handler + * auto-routes non-Anthropic models through the provider-agnostic gateway tool + * loop — and leaving it in place would tell a user who just deliberately chose + * a local model that their choice doesn't count. + * + * What IS still worth saying is the part that actually differs between the two + * loops: prompt caching. Anthropic's ephemeral cache markers are what keep a + * long multi-turn loop cheap; providers without them re-send the whole + * conversation every turn. On a local model that costs latency rather than + * money, which is a real trade-off but not a blocker. + * + * Exported so `initPGLite` can reuse the same message in its post-init stderr + * summary path (the auto-pick branch doesn't run the picker but still needs to + * surface this). One source of truth keeps the message aligned across the + * picker / init summary / doctor surfaces. */ export function printSubagentAnthropicCaveat(write: (s: string) => void): void { write( '\n' + 'Note: subagent features (gbrain dream, gbrain agent run, gbrain autopilot)\n' + - ' require ANTHROPIC_API_KEY regardless of which chat model you pick.\n' + - ' Chat alone (gbrain think, gbrain query expansion) works without it.\n' + - ' Set ANTHROPIC_API_KEY before running those commands.\n\n', + ' will run on this model via the provider-agnostic tool loop — no\n' + + ' ANTHROPIC_API_KEY needed. Two things to know:\n' + + ' - The model must support tool calling. Local models vary; small\n' + + ' ones tend to loop badly even when they advertise support.\n' + + ' - No prompt caching outside Anthropic, so each turn re-sends the\n' + + ' conversation. Long loops get slower (and, on hosted providers,\n' + + ' more expensive) than the same loop on Anthropic.\n' + + ' Run `gbrain doctor` to confirm the endpoint is reachable.\n\n', ); } diff --git a/src/commands/init.ts b/src/commands/init.ts index ca01c611c5..04d076f70b 100644 --- a/src/commands/init.ts +++ b/src/commands/init.ts @@ -1064,6 +1064,15 @@ async function initPGLite(opts: { printSubagentAnthropicCaveat((s) => process.stderr.write(s)); } + // Local-fleet tier assignment. Same placement rationale as the mode picker + // below: after initSchema so config writes land. No-op unless chat_model is + // an Ollama one; fail-open when the daemon is down. + const { maybeAutotuneLocalTiers } = await import('./init-autotune.ts'); + await maybeAutotuneLocalTiers(engine, { + chatModel: opts.aiOpts?.chat_model, + jsonOutput: opts.jsonOutput, + }); + // v0.32.3 search-lite install-time mode picker. Runs AFTER initSchema so // DB config writes are valid. Idempotent: skipped on re-init if already set. // Non-TTY auto-selects; --json emits a structured event. @@ -1311,6 +1320,13 @@ async function initPostgres(opts: { printSubagentAnthropicCaveat((s) => process.stderr.write(s)); } + // Local-fleet tier assignment — same shape as the PGLite path above. + const { maybeAutotuneLocalTiers: autotunePostgres } = await import('./init-autotune.ts'); + await autotunePostgres(engine, { + chatModel: opts.aiOpts?.chat_model, + jsonOutput: opts.jsonOutput, + }); + // v0.32.3 search-lite install-time mode picker. Same shape as the // PGLite path above — runs AFTER initSchema, idempotent on re-init. const { runModePicker: runPostgresModePicker } = await import('./init-mode-picker.ts'); diff --git a/src/commands/models.ts b/src/commands/models.ts index 03a4b5634d..1fecb30143 100644 --- a/src/commands/models.ts +++ b/src/commands/models.ts @@ -646,13 +646,20 @@ export async function runModels(engine: BrainEngine, args: string[]): Promise # global hammer gbrain config set models.tier. # per-tier (utility/reasoning/deep/subagent) gbrain config set models.aliases. # custom alias -Tiers: utility (haiku-class) | reasoning (sonnet) | deep (opus) | subagent (Anthropic-only) +Tiers: utility (classification) | reasoning (default workhorse) | deep (expensive +reasoning) | subagent (tool loop). Without per-tier config, a brain with an +Anthropic key uses the Anthropic defaults; a keyless brain falls back to its +configured chat_model for every tier — which \`autotune\` improves on by giving +each tier a model sized for its job. `); return; } + if (sub === 'autotune') { + await runModelsAutotune(engine, { + json, + dryRun: args.includes('--dry-run'), + force: args.includes('--force'), + }); + return; + } + if (sub === 'read') { const report = await buildReport(engine); if (json) { @@ -754,3 +779,145 @@ Tiers: utility (haiku-class) | reasoning (sonnet) | deep (opus) | subagent (Anth process.exit(1); } } + +// ── autotune ──────────────────────────────────────────────────────────────── + +const AUTOTUNE_TIERS = ['utility', 'reasoning', 'deep', 'subagent'] as const; + +/** + * Assign the four model tiers from the local Ollama fleet. + * + * Runs discovery ONCE and persists the result to `models.tier.*`. Model + * resolution deliberately stays a pure config read — putting this probe in + * the resolution path would add a network round-trip to every unconfigured + * LLM call, which is a worse problem than the flat defaults it fixes. + * + * Existing per-tier settings are preserved unless `--force`: a hand-tuned tier + * is a decision, and silently overwriting it on a re-run would make the + * command unsafe to repeat after pulling a model. + */ +export async function runModelsAutotune( + engine: BrainEngine, + opts: { + json?: boolean; + dryRun?: boolean; + force?: boolean; + /** + * Exit the process on a discovery failure. True for the CLI, where a + * non-zero code is the contract. MUST be false for in-process callers: + * `gbrain init` invokes this as an optimization, and an unreachable + * daemon at install time — the normal case, since people configure + * before starting `ollama serve` — would otherwise abort the whole init. + */ + exitOnError?: boolean; + } = {}, +): Promise { + const exitOnError = opts.exitOnError !== false; + const { + discoverOllamaModels, rankForTiers, observeServedContext, effectiveContext, + } = await import('../core/ai/local-discovery.ts'); + const { loadConfig } = await import('../core/config.ts'); + const { getRecipe } = await import('../core/ai/recipes/index.ts'); + + const cfg = loadConfig(); + const baseURL = cfg?.provider_base_urls?.ollama + ?? process.env.OLLAMA_BASE_URL + ?? getRecipe('ollama')?.base_url_default + ?? 'http://localhost:11434/v1'; + + let models; + try { + models = await discoverOllamaModels(baseURL); + } catch (e) { + const msg = `Ollama not reachable at ${baseURL}: ${e instanceof Error ? e.message : String(e)}`; + if (opts.json) process.stdout.write(JSON.stringify({ status: 'error', message: msg }, null, 2) + '\n'); + else process.stderr.write(`${msg}\nStart it with \`ollama serve\`, or set OLLAMA_BASE_URL.\n`); + if (exitOnError) process.exit(1); + return; + } + + const assignment = rankForTiers(models); + if (!assignment) { + // Every model rejected. Naming why beats "no models found" — the usual + // cause is a fleet of embedding-only models, which looks non-empty. + const msg = models.length === 0 + ? `No models pulled. Try \`ollama pull qwen3\`.` + : `No tool-calling chat model among ${models.length} pulled model(s). The subagent loop dispatches brain ops via tool calls, so a model without tool support cannot drive it. Try \`ollama pull qwen3\`.`; + if (opts.json) process.stdout.write(JSON.stringify({ status: 'error', message: msg, models }, null, 2) + '\n'); + else process.stderr.write(`${msg}\n`); + if (exitOnError) process.exit(1); + return; + } + + const served = await observeServedContext(baseURL, { models }); + const existing: Record = {}; + for (const t of AUTOTUNE_TIERS) { + existing[t] = await engine.getConfig(`models.tier.${t}`).catch(() => null); + } + + const plan = AUTOTUNE_TIERS.map(tier => { + const model = assignment.tiers[tier]; + const info = models.find(m => `ollama:${m.name}` === model); + const held = existing[tier]; + const skipped = Boolean(held && held.trim() && !opts.force); + return { + tier, + model, + reason: assignment.reasons[tier], + context: info ? effectiveContext(info, served.servedContext) : undefined, + existing: held ?? null, + applied: !skipped && !opts.dryRun, + skipped_reason: skipped ? `already set to "${held}" — pass --force to overwrite` : undefined, + }; + }); + + if (!opts.dryRun) { + for (const p of plan) { + if (p.applied) await engine.setConfig(`models.tier.${p.tier}`, p.model); + } + } + + if (opts.json) { + process.stdout.write(JSON.stringify({ + status: 'ok', + base_url: baseURL, + dry_run: Boolean(opts.dryRun), + served_context: served.servedContext ?? null, + served_context_definitive: served.definitive, + served_context_observed_from: served.observedFrom ?? null, + tiers: plan, + rejected: assignment.rejected, + }, null, 2) + '\n'); + return; + } + + const out = process.stdout; + out.write(`Local fleet at ${baseURL}: ${models.length} model(s), ${models.filter(m => m.chatCapable).length} tool-capable.\n\n`); + for (const p of plan) { + const mark = p.applied ? '✔' : p.skipped_reason ? '–' : ' '; + const ctx = p.context ? `${Math.round(p.context / 1024)}k ctx` : 'ctx unknown'; + out.write(` ${mark} ${p.tier.padEnd(10)} ${p.model.padEnd(28)} ${ctx.padStart(11)} ${p.reason}\n`); + if (p.skipped_reason) out.write(` ${' '.repeat(10)} kept: ${p.skipped_reason}\n`); + } + if (assignment.rejected.length > 0) { + out.write(`\n Not eligible for chat tiers:\n`); + for (const r of assignment.rejected) out.write(` - ${r.name.padEnd(32)} ${r.reason}\n`); + } + if (served.servedContext !== undefined) { + out.write( + served.definitive + ? `\n Daemon serves at most ${served.servedContext} tokens (measured on ${served.observedFrom}); ` + + `contexts above are clamped to it.\n` + : `\n Daemon served ${served.servedContext} tokens to ${served.observedFrom}, which is a FLOOR, not the cap ` + + `(that model got its full trained length). Contexts are clamped to it — safe, possibly pessimistic. ` + + `Raise with OLLAMA_CONTEXT_LENGTH.\n`, + ); + } else { + out.write(`\n No chat model resident, so the daemon's served context is unknown and trained lengths are shown ` + + `unclamped. An un-tuned daemon serves 4096 regardless of what a model was trained for — run one query, then ` + + `re-run autotune for a measured reading.\n`); + } + out.write(opts.dryRun + ? `\n Dry run — nothing written. Re-run without --dry-run to apply.\n` + : `\n Wrote ${plan.filter(p => p.applied).length} tier(s). Override any with: gbrain config set models.tier. \n`); +} diff --git a/src/core/ai/local-discovery.ts b/src/core/ai/local-discovery.ts new file mode 100644 index 0000000000..ddafcfd94c --- /dev/null +++ b/src/core/ai/local-discovery.ts @@ -0,0 +1,269 @@ +/** + * Local-model discovery for tier assignment. + * + * `resolveTierDefault` gives a keyless brain ONE model for all four tiers. + * That works but wastes the local fleet: a 3B model is the right utility-tier + * classifier and the wrong deep-tier reasoner, and vice versa. This module + * discovers what the user has actually pulled and proposes a per-tier mapping. + * + * The design constraint that shapes everything here: **model resolution must + * stay a pure config read.** Discovery therefore runs ONCE, at setup, and + * writes `models.tier.*`; it is never consulted from the resolution path. A + * network round-trip in front of every unconfigured LLM call would be a far + * worse regression than the imperfect defaults it fixed. + * + * Ollama-only, deliberately. It is the one local provider exposing an + * authoritative per-model capability list (`/api/show` → `capabilities`). + * `llama-server` serves a single model chosen at launch, so there is nothing + * to rank; LiteLLM proxies arbitrary backends and exposes no capability API. + * Both keep the single-model behavior. + * + * NOTE: `/api/tags`, `/api/show` and `/api/ps` are Ollama-NATIVE endpoints, + * not the OpenAI-compatible surface the gateway uses. The recipe's base URL + * ends in `/v1`; `ollamaApiRoot()` strips it. + */ + +import type { ModelTier } from '../model-config.ts'; + +/** One model as the Ollama daemon reports it. */ +export interface LocalModelInfo { + /** Ollama model id, e.g. `qwen3:32b`. Use verbatim — tags are significant. */ + name: string; + /** Declared capabilities, e.g. ['completion','tools','thinking']. */ + capabilities: string[]; + /** On-disk size in bytes. The ranking signal — see chatCapable's doc. */ + bytes: number; + /** Reported parameter count, e.g. '20.9B'. Absent for some builds (MLX). */ + parameterSize?: string; + /** Context length the model was TRAINED for. Not necessarily what is served. */ + trainedContext?: number; + /** + * Usable as a chat / subagent model. + * + * Requires BOTH `completion` and `tools`, and the conjunction is + * load-bearing rather than defensive: several embedding models advertise + * `tools` WITHOUT `completion` (e.g. `qwen3-embedding:8b` reports + * `[tools,embedding]`). Filtering on `tools` alone admits a multi-billion + * parameter embedding model into a chat tier, where it ranks high by size + * and cannot generate text at all. + */ + chatCapable: boolean; +} + +export interface TierAssignment { + tiers: Record; + /** Human-readable justification per tier, for the command's output. */ + reasons: Record; + /** Models considered but rejected, with why. Surfaced so the pick is auditable. */ + rejected: Array<{ name: string; reason: string }>; +} + +/** Strip the OpenAI-compat `/v1` suffix to reach Ollama's native API root. */ +export function ollamaApiRoot(baseURL: string): string { + return baseURL.replace(/\/v1\/?$/, '').replace(/\/$/, ''); +} + +async function getJson(url: string, timeoutMs: number, init?: RequestInit): Promise { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + try { + const res = await fetch(url, { ...init, signal: controller.signal }); + if (!res.ok) throw new Error(`HTTP ${res.status}`); + return await res.json(); + } finally { + clearTimeout(timer); + } +} + +/** Pull the `*.context_length` entry out of Ollama's model_info bag. */ +function trainedContextFrom(modelInfo: Record | undefined): number | undefined { + if (!modelInfo) return undefined; + const key = Object.keys(modelInfo).find(k => k.endsWith('.context_length')); + if (!key) return undefined; + const v = modelInfo[key]; + return typeof v === 'number' && v > 0 ? v : undefined; +} + +/** + * Enumerate the daemon's models with capabilities. One `/api/tags` call plus + * one `/api/show` per model — metadata only, nothing is loaded into memory. + */ +export async function discoverOllamaModels( + baseURL: string, + opts: { timeoutMs?: number } = {}, +): Promise { + const root = ollamaApiRoot(baseURL); + const timeoutMs = opts.timeoutMs ?? 5000; + const tags = await getJson(`${root}/api/tags`, timeoutMs); + const models: LocalModelInfo[] = []; + for (const m of tags.models ?? []) { + let capabilities: string[] = []; + let trainedContext: number | undefined; + try { + const show = await getJson(`${root}/api/show`, timeoutMs, { + method: 'POST', + body: JSON.stringify({ model: m.name }), + headers: { 'content-type': 'application/json' }, + }); + capabilities = Array.isArray(show.capabilities) ? show.capabilities : []; + trainedContext = trainedContextFrom(show.model_info); + } catch { + // A model whose /api/show fails stays in the list with no capabilities, + // so it is rejected rather than silently dropped from the audit trail. + } + models.push({ + name: m.name, + capabilities, + bytes: typeof m.size === 'number' ? m.size : 0, + ...(m.details?.parameter_size ? { parameterSize: m.details.parameter_size } : {}), + ...(trainedContext !== undefined ? { trainedContext } : {}), + chatCapable: capabilities.includes('completion') && capabilities.includes('tools'), + }); + } + return models; +} + +/** + * Observe the context length the daemon actually SERVES, which is + * `min(trained context, OLLAMA_CONTEXT_LENGTH)`. + * + * This matters more than the trained context and is the number budget math + * needs. An un-tuned daemon serves 4096 while the model advertises 131072; + * recording the trained value there makes gbrain send a prompt Ollama + * silently truncates, and a truncated prompt loses the retrieved context at + * the front — plausible answers, missing evidence, no error anywhere. + * + * Only `/api/ps` reveals it, and only for a LOADED model. We read whatever is + * already resident (free) rather than loading anything: a caller that wants a + * reading for a cold model must load it first. + * + * Returns the smallest served length observed among CHAT-CAPABLE models. The + * chat-capable filter is not a detail — embedding models carry tiny contexts + * (nomic-embed-text is trained for 2048), and a plain minimum across + * everything resident clamps every chat tier to an embedder's window. Taking + * the minimum is the right instinct (under-reporting costs capacity, + * over-reporting costs correctness) but only within the population the number + * describes. + * + * `opts.models` is what identifies chat-capability, so a caller that omits it + * gets no reading rather than a wrong one. + */ +export async function observeServedContext( + baseURL: string, + opts: { timeoutMs?: number; models?: LocalModelInfo[] } = {}, +): Promise { + const root = ollamaApiRoot(baseURL); + const chatNames = new Set((opts.models ?? []).filter(m => m.chatCapable).map(m => m.name)); + try { + const ps = await getJson(`${root}/api/ps`, opts.timeoutMs ?? 5000); + const loaded = (ps.models ?? []).filter( + (m: any) => typeof m.context_length === 'number' && chatNames.has(m.name), + ); + if (loaded.length === 0) return { definitive: false }; + const best = loaded.reduce((a: any, b: any) => (a.context_length <= b.context_length ? a : b)); + // Definitiveness: if the observed model was served LESS than it was + // trained for, the daemon cap is exactly that number. If it was served + // its full trained length, we learned only that the cap is at least that + // — a bigger model might legitimately get more. Callers clamp either way + // (under-reporting is the safe direction) but should say which they have. + const trained = opts.models?.find(m => m.name === best.name)?.trainedContext; + return { + servedContext: best.context_length, + observedFrom: best.name, + definitive: trained !== undefined && best.context_length < trained, + }; + } catch { + return { definitive: false }; + } +} + +export interface ServedContextObservation { + /** Context length the daemon served for `observedFrom`. */ + servedContext?: number; + /** Which loaded model the reading came from. */ + observedFrom?: string; + /** + * True when `servedContext` is the daemon's actual cap (the observed model + * was clamped below its trained length). False when it is only a lower + * bound — the observed model got everything it asked for, so a larger model + * may be allowed more than this reading suggests. + */ + definitive: boolean; +} + +/** + * Effective usable context for a model: the trained length, clamped by what + * the daemon was observed to serve. Unknowns fall through to the other value, + * and when both are unknown the caller keeps its own conservative default. + */ +export function effectiveContext( + model: Pick, + servedContext?: number, +): number | undefined { + if (model.trainedContext !== undefined && servedContext !== undefined) { + return Math.min(model.trainedContext, servedContext); + } + return model.trainedContext ?? servedContext; +} + +/** + * Assign the four tiers from a discovered fleet. Pure — no I/O — so the policy + * is testable against fixtures without a daemon. + * + * Ranking is by on-disk BYTES, not `parameterSize`: quantized and MLX builds + * frequently omit the parameter count entirely (every `*-mlx` model in a real + * fleet reported none), so a parameter-based sort silently drops them to the + * bottom. Bytes are always present and correlate with capability within a + * fleet. It is a proxy, not a quality ranking — a 9B tuned for reasoning may + * well beat a 12B generalist, which is why the assignment is printed for + * review and every tier stays overridable. + */ +export function rankForTiers(models: LocalModelInfo[]): TierAssignment | null { + const rejected: Array<{ name: string; reason: string }> = []; + const chat: LocalModelInfo[] = []; + for (const m of models) { + if (m.chatCapable) { chat.push(m); continue; } + const why = m.capabilities.includes('embedding') + ? m.capabilities.includes('tools') + ? 'embedding model (advertises tools but has no completion)' + : 'embedding model' + : m.capabilities.length === 0 + ? 'capabilities unavailable' + : `no tool calling (${m.capabilities.join(',')})`; + rejected.push({ name: m.name, reason: why }); + } + if (chat.length === 0) return null; + + const bySizeDesc = [...chat].sort((a, b) => b.bytes - a.bytes); + const largest = bySizeDesc[0]; + // With three or more, the workhorse is the runner-up: the largest model is + // reserved for the deep tier, where its latency is the point rather than a + // tax paid on every ordinary call. + const workhorse = bySizeDesc.length >= 3 ? bySizeDesc[1] : largest; + // Utility runs classification and verdicts, where `thinking` is pure + // overhead — it spends reasoning tokens on a job whose answer is a label. + // Prefer the smallest NON-thinking model, falling back to the smallest. + const bySizeAsc = [...bySizeDesc].reverse(); + const utility = bySizeAsc.find(m => !m.capabilities.includes('thinking')) ?? bySizeAsc[0]; + + const q = (m: LocalModelInfo) => `ollama:${m.name}`; + const gb = (m: LocalModelInfo) => `${(m.bytes / 1e9).toFixed(1)}GB`; + + return { + tiers: { + utility: q(utility), + reasoning: q(workhorse), + deep: q(largest), + subagent: q(workhorse), + }, + reasons: { + utility: utility.capabilities.includes('thinking') + ? `${gb(utility)}, smallest tool-capable (every candidate has thinking)` + : `${gb(utility)}, smallest tool-capable without thinking`, + reasoning: `${gb(workhorse)}, ${bySizeDesc.length >= 3 ? 'runner-up by size' : 'largest available'}`, + deep: `${gb(largest)}, largest tool-capable`, + subagent: `${gb(workhorse)}, same workhorse — the tool loop needs reliability, not peak size`, + }, + rejected, + }; +} diff --git a/src/core/ai/model-resolver.ts b/src/core/ai/model-resolver.ts index f7cc338bdd..a4fa21b356 100644 --- a/src/core/ai/model-resolver.ts +++ b/src/core/ai/model-resolver.ts @@ -117,8 +117,8 @@ export function assertTouchpoint( `Provider "${recipe.id}" does not support touchpoint "${touchpoint}".`, touchpoint === 'embedding' && recipe.id === 'anthropic' ? 'Anthropic has no embedding model. Use openai or google for embeddings.' - : touchpoint === 'chat' && (recipe.id === 'voyage' || recipe.id === 'ollama') - ? `${recipe.name} is configured here only for embeddings. Use openai/anthropic/google/deepseek/groq/together for chat.` + : touchpoint === 'chat' && recipe.id === 'voyage' + ? `${recipe.name} is configured here only for embeddings. Use a chat-capable provider — ollama or llama-server to stay local, or openai/anthropic/google/deepseek/groq/together for hosted.` : undefined, ); } diff --git a/src/core/ai/probes.ts b/src/core/ai/probes.ts index ba17b67691..fae0b4bbcc 100644 --- a/src/core/ai/probes.ts +++ b/src/core/ai/probes.ts @@ -36,8 +36,15 @@ export async function probeOpenAICompat(baseUrl: string, timeoutMs: number = 100 } } -export async function probeOllama(): Promise { - const url = process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434/v1'; +/** + * Probe an Ollama daemon's OpenAI-compatible endpoint. Defaults to the local + * daemon on 11434. Pass `baseURL` (from `cfg.base_urls['ollama']`) so the + * probe checks the same endpoint the gateway will call — a config-only URL + * override would otherwise be invisible here (the codex-#5 class fixed for + * llama-server). + */ +export async function probeOllama(baseURL?: string): Promise { + const url = baseURL ?? process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434/v1'; return probeOpenAICompat(url); } diff --git a/src/core/ai/recipes/cerebras.ts b/src/core/ai/recipes/cerebras.ts new file mode 100644 index 0000000000..df4880e533 --- /dev/null +++ b/src/core/ai/recipes/cerebras.ts @@ -0,0 +1,58 @@ +import type { Recipe } from '../types.ts'; + +/** + * Cerebras Inference — wafer-scale hardware, the fastest tokens/sec tier + * available. Same role in the routing table as Groq: the latency lane. + * + * Chat + expansion only; Cerebras serves no embedding model. + * + * NOTE on the base URL: Cerebras documents the host as + * `https://api.cerebras.ai` and the OpenAI-compatible routes under `/v1`. + * The recipe pins the `/v1` suffix because `createOpenAICompatible` appends + * only the route (`/chat/completions`), never the version segment. + * + * NOTE on pricing: Cerebras does not publish per-1M-token rates on its public + * pricing page (self-serve starts at a $10 credit; per-model rates live behind + * the dashboard). `cost_per_1m_*` is therefore left undefined rather than + * guessed — same choice litellm-proxy makes. Cost views treat undefined as + * "unknown", which is honest; a fabricated number would silently corrupt + * `--max-usd` pre-flights. + */ +export const cerebras: Recipe = { + id: 'cerebras', + name: 'Cerebras Inference', + tier: 'openai-compat', + implementation: 'openai-compatible', + base_url_default: 'https://api.cerebras.ai/v1', + auth_env: { + required: ['CEREBRAS_API_KEY'], + optional: ['CEREBRAS_BASE_URL'], + setup_url: 'https://cloud.cerebras.ai', + }, + touchpoints: { + expansion: { + models: ['gpt-oss-120b', 'gemma-4-31b'], + cost_per_1m_tokens_usd: undefined, + price_last_verified: '2026-08-08', + }, + chat: { + // `zai-glm-4.7` is deliberately omitted: Cerebras scheduled it for + // deprecation on 2026-08-17. Listing a model that dies in days would + // make it a wizard-pickable default. + models: ['gpt-oss-120b', 'gemma-4-31b'], + supports_tools: true, + supports_subagent_loop: true, + supports_prompt_cache: false, + // Cerebras documents `response_format: json_schema` and recommends it + // over json_object for models that support it. + supports_structured_outputs: true, + // Paid tier. The free tier caps at 65k — a free-tier key will see + // provider-side truncation before gbrain's budget math kicks in. + max_context_tokens: 131_072, + cost_per_1m_input_usd: undefined, + cost_per_1m_output_usd: undefined, + price_last_verified: '2026-08-08', + }, + }, + setup_hint: 'Get an API key at https://cloud.cerebras.ai, then `export CEREBRAS_API_KEY=...`. Pair with a separate embedding provider — Cerebras ships none.', +}; diff --git a/src/core/ai/recipes/fireworks.ts b/src/core/ai/recipes/fireworks.ts new file mode 100644 index 0000000000..5907d786b0 --- /dev/null +++ b/src/core/ai/recipes/fireworks.ts @@ -0,0 +1,64 @@ +import type { Recipe } from '../types.ts'; + +/** + * Fireworks AI — hosted open-weight models with an OpenAI-compatible surface, + * covering chat AND embeddings. The one provider in this wave that can serve a + * whole brain on its own. + * + * Model ids are fully-qualified paths (`accounts/fireworks/models/`), + * which contain no colon, so `provider:model` parsing is unambiguous: + * `fireworks:accounts/fireworks/models/kimi-k2-instruct-0905`. Users on a + * dedicated deployment substitute their own account segment — that's why the + * model lists here stay advisory (`tier: 'openai-compat'` never rejects an + * unlisted id). + * + * NOTE on pricing: Fireworks prices per-model by parameter-count band and + * changes the bands as the model library rotates. Rather than pin a number + * that goes stale silently, `cost_per_1m_*` is left undefined (the + * litellm-proxy choice). Cost views report unknown instead of wrong. + */ +export const fireworks: Recipe = { + id: 'fireworks', + name: 'Fireworks AI', + tier: 'openai-compat', + implementation: 'openai-compatible', + base_url_default: 'https://api.fireworks.ai/inference/v1', + auth_env: { + required: ['FIREWORKS_API_KEY'], + optional: ['FIREWORKS_BASE_URL'], + setup_url: 'https://fireworks.ai/account/api-keys', + }, + touchpoints: { + embedding: { + models: ['nomic-ai/nomic-embed-text-v1.5'], + // Matryoshka: trained to truncate anywhere in 64..768. Fireworks honors + // the `dimensions` parameter, so the standard ladder is selectable and + // the Matryoshka allowlist governs (no `trust_custom_dims` needed). + dims_options: [768, 512, 256, 128, 64], + default_dims: 768, + cost_per_1m_tokens_usd: undefined, + price_last_verified: '2026-08-08', + // Batch capacity tracks the serving deployment, not a documented + // static cap. Same posture as together/litellm. + no_batch_cap: true, + }, + expansion: { + models: ['accounts/fireworks/models/kimi-k2-instruct-0905'], + cost_per_1m_tokens_usd: undefined, + price_last_verified: '2026-08-08', + }, + chat: { + models: ['accounts/fireworks/models/kimi-k2-instruct-0905'], + supports_tools: true, + supports_subagent_loop: true, + supports_prompt_cache: false, + // Documented: `response_format` accepts `json_schema` with a supplied schema. + supports_structured_outputs: true, + max_context_tokens: 131_072, + cost_per_1m_input_usd: undefined, + cost_per_1m_output_usd: undefined, + price_last_verified: '2026-08-08', + }, + }, + setup_hint: 'Get an API key at https://fireworks.ai/account/api-keys, then `export FIREWORKS_API_KEY=...`. Model ids are full paths: --model fireworks:accounts/fireworks/models/.', +}; diff --git a/src/core/ai/recipes/index.ts b/src/core/ai/recipes/index.ts index ccdf487d7c..bdb6686d6e 100644 --- a/src/core/ai/recipes/index.ts +++ b/src/core/ai/recipes/index.ts @@ -29,6 +29,10 @@ import { moonshot } from './moonshot.ts'; import { mistral } from './mistral.ts'; import { nvidia } from './nvidia.ts'; import { perplexity } from './perplexity.ts'; +import { xai } from './xai.ts'; +import { cerebras } from './cerebras.ts'; +import { fireworks } from './fireworks.ts'; +import { sambanova } from './sambanova.ts'; const ALL: Recipe[] = [ openai, @@ -54,6 +58,10 @@ const ALL: Recipe[] = [ mistral, nvidia, perplexity, + xai, + cerebras, + fireworks, + sambanova, ]; /** Map from `provider:id` key to recipe. */ diff --git a/src/core/ai/recipes/llama-server.ts b/src/core/ai/recipes/llama-server.ts index a51650b271..8745b2266f 100644 --- a/src/core/ai/recipes/llama-server.ts +++ b/src/core/ai/recipes/llama-server.ts @@ -2,12 +2,18 @@ import type { Recipe } from '../types.ts'; import { probeLlamaServer } from '../probes.ts'; /** - * llama.cpp's `llama-server --embeddings` (also published as - * `@llama.cpp/llama-server`). Exposes an OpenAI-compatible /v1/embeddings - * endpoint. Distinct from Ollama: different default port (8080), different + * llama.cpp's `llama-server` (also published as `@llama.cpp/llama-server`). + * Exposes OpenAI-compatible `/v1/embeddings` and `/v1/chat/completions` + * endpoints. Distinct from Ollama: different default port (8080), different * model-management story (you launch it with `--model `; the server * serves whatever model was passed). * + * One server instance serves one model. Running embeddings AND chat locally + * therefore means either two `llama-server` processes on different ports — + * one with `--embeddings`, one with `--jinja` for tool calling — or pairing + * this recipe with `ollama` for the other touchpoint. `gbrain init` can point + * each touchpoint at a different provider, so the mixed setup is supported. + * * Like LiteLLM, this recipe ships with `models: []` because the model * identity is whatever the user launched llama-server with. They MUST * pass `--embedding-model llama-server:` and `--embedding-dimensions @@ -42,6 +48,36 @@ export const llamaServer: Recipe = { // server launched with a larger `-b` can raise this. v0.32 (#779). max_batch_items: 32, }, + // Same server, same OpenAI-compatible surface. `models: []` for the same + // reason embedding declares it: model identity is whatever the server was + // launched with (`--model `), so there is nothing to enumerate. + expansion: { + models: [], + cost_per_1m_tokens_usd: 0, + price_last_verified: '2026-08-08', + }, + chat: { + models: [], + // llama.cpp's server implements OpenAI-style `tools` / `tool_calls` and + // ships per-family chat templates (`--jinja`) that emit them. Whether a + // given GGUF actually calls tools well depends on the model, not the + // server — same caveat as Ollama. + supports_tools: true, + supports_subagent_loop: true, + // KV-cache reuse across requests is a server-side prefix optimization, + // not an Anthropic-style cache_control contract. Nothing to mark up. + supports_prompt_cache: false, + // llama.cpp honors a strict GBNF grammar, but its OpenAI-compat + // `response_format: json_schema` handling varies by build. Stay on the + // schemaless path — correct everywhere, at the cost of one retry. + supports_structured_outputs: false, + // Set by `--ctx-size` at launch (default 4096). Declaring the launch + // default keeps token-budget math honest for an un-tuned server. + max_context_tokens: 4096, + cost_per_1m_input_usd: 0, + cost_per_1m_output_usd: 0, + price_last_verified: '2026-08-08', + }, }, /** * Probe via the OpenAI-compatible /v1/models endpoint. Caller passes the @@ -55,7 +91,7 @@ export const llamaServer: Recipe = { if (!result.reachable) { return { ready: false, - hint: `llama-server not reachable at ${url}. Start it with \`./llama-server --model --embeddings\` or set LLAMA_SERVER_BASE_URL.`, + hint: `llama-server not reachable at ${url}. Start it with \`./llama-server --model --embeddings\` (embeddings) or \`./llama-server --model --jinja\` (chat/tools), or set LLAMA_SERVER_BASE_URL.`, }; } if (!result.models_endpoint_valid) { @@ -67,5 +103,5 @@ export const llamaServer: Recipe = { return { ready: true }; }, setup_hint: - 'Build llama.cpp, then `llama-server --model --embeddings`. Set --embedding-model llama-server: + --embedding-dimensions .', + 'Build llama.cpp, then `llama-server --model --embeddings` for embeddings (set --embedding-model llama-server: + --embedding-dimensions ), or `llama-server --model --jinja` for chat with tool calling. One model per server — run two ports, or pair with ollama.', }; diff --git a/src/core/ai/recipes/ollama.ts b/src/core/ai/recipes/ollama.ts index 45db095d52..b7d1d941c6 100644 --- a/src/core/ai/recipes/ollama.ts +++ b/src/core/ai/recipes/ollama.ts @@ -1,5 +1,28 @@ import type { Recipe } from '../types.ts'; +import { probeOllama } from '../probes.ts'; +/** + * Ollama — the zero-cost local lane. + * + * Serves an OpenAI-compatible API on `http://localhost:11434/v1` by default. + * Three deployment shapes, all the same recipe: + * + * 1. Local daemon (default) — `ollama serve` on this machine. + * 2. Remote daemon — set `OLLAMA_BASE_URL=http://gpu-box:11434/v1`. + * Set `OLLAMA_API_KEY` too when the remote host + * sits behind an authenticating reverse proxy. + * 3. Ollama Cloud — `OLLAMA_BASE_URL=https://ollama.com/v1` plus + * `OLLAMA_API_KEY`. NOTE: cloud-suffixed model + * ids (`…-cloud`, `…:cloud`) run on Ollama's + * servers even when the base URL points at the + * LOCAL daemon — the daemon proxies them. Picking + * one sends brain content off-device; that is a + * privacy decision, not a performance one. + * + * All four touchpoints are declared, so a keyless install can run the full + * brain — embeddings, query expansion, chat/`think`, and the subagent tool + * loop — with no hosted API key anywhere. + */ export const ollama: Recipe = { id: 'ollama', name: 'Ollama (local)', @@ -45,6 +68,82 @@ export const ollama: Recipe = { // OLLAMA_NUM_PARALLEL config; no static cap to declare. v0.32 (#779). no_batch_cap: true, }, + // Same OpenAI-compatible endpoint as chat. Declared so an explicit + // `expansion_model: ollama:` resolves instead of silently dropping + // expansion (the #1135 class). A small instruct model is the natural + // pick — multi-query rewrites need no tool calling. + expansion: { + models: ['llama3.2', 'qwen3:4b', 'qwen3:8b', 'mistral-small3.2'], + cost_per_1m_tokens_usd: 0, + price_last_verified: '2026-08-08', + }, + chat: { + // Advisory, not an allowlist: `tier: 'openai-compat'` means + // assertTouchpoint never rejects an unlisted id, so any model the + // user has pulled works. These are the tool-calling-capable families + // worth defaulting to — Ollama exposes tool calling only for models + // whose template declares it, and the subagent loop is useless without. + models: [ + 'llama3.3', + 'llama3.1', + 'qwen3', + 'qwen3:8b', + 'qwen3:14b', + 'qwen3:32b', + 'qwen2.5', + 'mistral-small3.2', + 'mistral-nemo', + 'gpt-oss:20b', + 'gpt-oss:120b', + 'devstral', + ], + supports_tools: true, + // The loop's crash-replay no longer depends on provider-native tool_use + // ids (v0.38 D11 moved stable-id generation gbrain-side), so an + // openai-compatible local backend is replay-safe. Tool-call QUALITY + // still varies by model — a 4B model will loop badly where a 32B one + // won't. That's a model-selection problem, not a capability gate. + supports_subagent_loop: true, + // No cross-request prompt cache. Ollama keeps the model resident and + // reuses the KV cache for a shared prefix within a session, but there + // is no Anthropic-style `cache_control` marker to honor, so the loop + // must not inject one. + supports_prompt_cache: false, + // Ollama's OpenAI-compat layer accepts `response_format: json_object` + // but does NOT honor a strict `json_schema`. Leaving this false routes + // expansion through the schemaless text path, which it can satisfy. + supports_structured_outputs: false, + // Ollama truncates to the model's `num_ctx` (default 4096) unless the + // Modelfile or `OLLAMA_CONTEXT_LENGTH` raises it. Declaring the + // conservative default keeps the token-budget math honest; users who + // raised it can override with `search.token_budget`. + max_context_tokens: 4096, + cost_per_1m_input_usd: 0, + cost_per_1m_output_usd: 0, + price_last_verified: '2026-08-08', + }, + }, + /** + * Probe the OpenAI-compatible /v1/models endpoint. Caller passes the + * resolved baseURL (from cfg.base_urls['ollama'] or env) so the probe + * checks the same endpoint live traffic will use. + */ + async probe(baseURL?: string) { + const url = baseURL ?? process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434/v1'; + const result = await probeOllama(url); + if (!result.reachable) { + return { + ready: false, + hint: `Ollama not reachable at ${url}. Start it with \`ollama serve\`, or set OLLAMA_BASE_URL to a remote daemon.`, + }; + } + if (!result.models_endpoint_valid) { + return { + ready: false, + hint: `Ollama reached but /v1/models returned an unexpected shape: ${result.error ?? 'unknown'}.`, + }; + } + return { ready: true }; }, - setup_hint: 'Install Ollama from https://ollama.ai, then `ollama pull nomic-embed-text` and `ollama serve`.', + setup_hint: 'Install Ollama from https://ollama.ai, then `ollama serve` and pull a model for each touchpoint you use — e.g. `ollama pull nomic-embed-text` (embeddings) and `ollama pull qwen3` (chat). Remote daemon or Ollama Cloud: set OLLAMA_BASE_URL (+ OLLAMA_API_KEY).', }; diff --git a/src/core/ai/recipes/sambanova.ts b/src/core/ai/recipes/sambanova.ts new file mode 100644 index 0000000000..49575f504b --- /dev/null +++ b/src/core/ai/recipes/sambanova.ts @@ -0,0 +1,65 @@ +import type { Recipe } from '../types.ts'; + +/** + * SambaNova Cloud — RDU-hosted open-weight models behind an OpenAI-compatible + * chat surface. + * + * Chat + expansion only. SambaNova's OpenAI-compatibility guide covers chat + * completions and the Responses API; it does not document an embeddings route, + * so declaring one would be a guess that only fails at first insert. + * + * NOTE on the base URL: SambaNova issues the base URL alongside the API key + * (dedicated deployments get their own host), so the pinned default is the + * public shared endpoint and `SAMBANOVA_BASE_URL` / `base_urls.sambanova` + * overrides it. Users on a dedicated endpoint MUST override. + * + * NOTE on pricing: per-model rates are not published in a stable public table. + * `cost_per_1m_*` stays undefined rather than fabricated. + * + * Known API limitation: `n > 1` is rejected when `tools` is present. gbrain + * never sets `n`, so this doesn't bite the subagent loop — recorded so a + * future multi-sample caller doesn't rediscover it in production. + */ +export const sambanova: Recipe = { + id: 'sambanova', + name: 'SambaNova Cloud', + tier: 'openai-compat', + implementation: 'openai-compatible', + base_url_default: 'https://api.sambanova.ai/v1', + auth_env: { + required: ['SAMBANOVA_API_KEY'], + optional: ['SAMBANOVA_BASE_URL'], + setup_url: 'https://cloud.sambanova.ai/apis', + }, + touchpoints: { + expansion: { + models: ['Meta-Llama-3.3-70B-Instruct', 'gpt-oss-120b'], + cost_per_1m_tokens_usd: undefined, + price_last_verified: '2026-08-08', + }, + chat: { + // Production models only. The preview tier (DeepSeek-V3.2 at 32k, + // gemma-4-31B-it) is omitted so the wizard can't default onto a model + // whose id or availability moves without notice. + models: [ + 'MiniMax-M2.7', + 'DeepSeek-V3.1', + 'Meta-Llama-3.3-70B-Instruct', + 'gpt-oss-120b', + ], + supports_tools: true, + supports_subagent_loop: true, + supports_prompt_cache: false, + // Not documented in the OpenAI-compatibility guide. Schemaless path. + supports_structured_outputs: false, + // Conservative: the smallest production context in the list above. + // MiniMax-M2.7 carries 192k; a brain pinned to it can raise + // `search.token_budget` accordingly. + max_context_tokens: 131_072, + cost_per_1m_input_usd: undefined, + cost_per_1m_output_usd: undefined, + price_last_verified: '2026-08-08', + }, + }, + setup_hint: 'Get an API key at https://cloud.sambanova.ai/apis, then `export SAMBANOVA_API_KEY=...` (and SAMBANOVA_BASE_URL if you are on a dedicated endpoint). Pair with a separate embedding provider.', +}; diff --git a/src/core/ai/recipes/xai.ts b/src/core/ai/recipes/xai.ts new file mode 100644 index 0000000000..8b08e91adb --- /dev/null +++ b/src/core/ai/recipes/xai.ts @@ -0,0 +1,59 @@ +import type { Recipe } from '../types.ts'; + +/** + * xAI (Grok). OpenAI-compatible chat endpoint at https://api.x.ai/v1. + * + * Chat + expansion only: xAI publishes no embedding model, so a brain routed + * here for chat still needs a separate embedding provider (openai, voyage, or + * ollama to stay local). `gbrain init` resolves each touchpoint independently, + * so the mixed setup is the normal case, not a workaround. + * + * Pricing note: xAI tiers by PROMPT SIZE, not by model — the published rate + * doubles once a request crosses 200k prompt tokens. The flat + * `cost_per_1m_*_usd` fields carry the sub-200k tier, which is the only one + * gbrain search payloads reach (tokenmax tops out around 20k). A caller + * deliberately stuffing a 200k+ context will under-estimate by 2x. + */ +export const xai: Recipe = { + id: 'xai', + name: 'xAI (Grok)', + tier: 'openai-compat', + implementation: 'openai-compatible', + base_url_default: 'https://api.x.ai/v1', + auth_env: { + required: ['XAI_API_KEY'], + optional: ['XAI_BASE_URL'], + setup_url: 'https://console.x.ai', + }, + touchpoints: { + expansion: { + models: ['grok-4.3', 'grok-build-0.1'], + cost_per_1m_tokens_usd: 1.25, + price_last_verified: '2026-08-08', + }, + chat: { + models: [ + 'grok-4.5', + 'grok-4.3', + 'grok-4.20-0309-reasoning', + 'grok-4.20-0309-non-reasoning', + 'grok-4.20-multi-agent-0309', + 'grok-build-0.1', + ], + supports_tools: true, + supports_subagent_loop: true, + // No published prompt-cache contract. Don't inject cache_control. + supports_prompt_cache: false, + // xAI documents tool calling but not a strict `json_schema` + // response_format. Stay on the schemaless expansion path until the + // docs commit to it — a wrong `true` here breaks expand(), a wrong + // `false` only costs a retry. + supports_structured_outputs: false, + max_context_tokens: 500_000, // grok-4.5; 4.3 and the 4.20 family carry 1M + cost_per_1m_input_usd: 2.00, // grok-4.5, sub-200k-prompt tier + cost_per_1m_output_usd: 6.00, + price_last_verified: '2026-08-08', + }, + }, + setup_hint: 'Get an API key at https://console.x.ai, then `export XAI_API_KEY=...`. Pair with a separate embedding provider — xAI ships none.', +}; diff --git a/src/core/minions/handlers/subagent.ts b/src/core/minions/handlers/subagent.ts index a490900f3d..ff81835834 100644 --- a/src/core/minions/handlers/subagent.ts +++ b/src/core/minions/handlers/subagent.ts @@ -172,6 +172,28 @@ interface PersistedToolExec { error: string | null; } +/** + * One-shot-per-model stderr notice when a non-Anthropic model is auto-routed + * through the gateway loop. Not a warning — the routing is correct and there + * is nothing to fix. It exists so an operator reading worker logs can see + * WHICH loop ran without turning on debug logging, since the two paths have + * different cost and prompt-cache characteristics. + */ +const _gatewayAutoRouteNoticed = new Set(); +function logGatewayAutoRoute(model: string): void { + if (_gatewayAutoRouteNoticed.has(model)) return; + _gatewayAutoRouteNoticed.add(model); + process.stderr.write( + `[subagent] "${model}" is non-Anthropic — running the provider-agnostic gateway tool loop. ` + + `(Set agent.use_gateway_loop=true to route Anthropic models through it as well.)\n`, + ); +} + +/** Test-only: clear the auto-route notice memo so tests can assert it re-emits. */ +export function _resetGatewayAutoRouteNoticeForTest(): void { + _gatewayAutoRouteNoticed.clear(); +} + // ── Public handler factory ────────────────────────────────── /** @@ -191,7 +213,16 @@ export function makeSubagentHandler(deps: SubagentDeps) { // new Anthropic() only reads env, so launchd/MCP workers whose key lives // in the gbrain config file would fail auth (#2048). const makeAnthropic = deps.makeAnthropic ?? (() => new Anthropic({ apiKey: resolveAnthropicKey() })); - const client: MessagesClient = deps.client ?? makeAnthropic().messages; + // LAZY, and it must stay lazy. `new Anthropic({apiKey: undefined})` throws + // in the SDK constructor, so building the client here — at worker + // REGISTRATION time — meant a brain with no Anthropic credential could not + // register the subagent handler at all. `gbrain jobs work` died on startup + // even when every job it would ever run targeted a local Ollama model + // through the provider-agnostic gateway loop, which never touches this + // client. Constructed on first legacy-path use instead; injected `deps.client` + // still short-circuits it for tests. + let _client: MessagesClient | undefined = deps.client; + const getClient = (): MessagesClient => (_client ??= makeAnthropic().messages); const config = deps.config ?? loadConfig() ?? ({ engine: 'postgres' } as GBrainConfig); const rateLeaseKey = deps.rateLeaseKey ?? DEFAULT_RATE_KEY; const maxConcurrent = deps.maxConcurrent ?? DEFAULT_MAX_CONCURRENT; @@ -250,20 +281,29 @@ export function makeSubagentHandler(deps: SubagentDeps) { // v0.38 S1.10 — feature flag for the gateway-native tool loop. When ON, // route ALL subagent jobs through gateway.toolLoop() (works for every - // provider in src/core/ai/recipes/). When OFF, route through the legacy - // Anthropic-direct path AND refuse non-Anthropic models loudly. + // provider in src/core/ai/recipes/). When OFF, Anthropic models take the + // legacy Anthropic-direct path. const useGatewayLoopRaw = await engine.getConfig('agent.use_gateway_loop').catch(() => null); // #2753: share the doctor's truthiness set. Before this, the doctor accepted // yes/on but the worker did not, so `config set ... yes` reported healthy // here and still refused the job below. - const useGatewayLoop = isConfigTruthy(useGatewayLoopRaw); - if (!useGatewayLoop && !isAnthropicProvider(model)) { - throw new Error( - `subagent job: resolved model "${model}" is non-Anthropic but agent.use_gateway_loop is not enabled. ` + - `Enable the gateway-native loop to run on this provider: ` + - `\`gbrain config set agent.use_gateway_loop true\`. ` + - `Or use an Anthropic model (e.g. anthropic:claude-sonnet-4-6).`, - ); + const explicitGatewayLoop = isConfigTruthy(useGatewayLoopRaw); + // A non-Anthropic model CANNOT run the legacy path — that path calls the + // Anthropic Messages API directly. Through v0.42 this threw and told the + // user to set `agent.use_gateway_loop`, which made a working, + // provider-agnostic loop look like an unsupported configuration and left + // every local-model brain one undiscoverable config key away from + // functioning. The flag stays meaningful (it opts ANTHROPIC models into + // the gateway loop too, which is the actual rollout decision); it is just + // no longer the thing standing between a local model and a loop that + // already supports it. + // + // Safety is unchanged: classifyCapabilities() above already rejected + // models without tool calling and unknown providers, so auto-routing here + // can only reach a provider the gateway loop can actually drive. + const useGatewayLoop = explicitGatewayLoop || !isAnthropicProvider(model); + if (useGatewayLoop && !explicitGatewayLoop) { + logGatewayAutoRoute(model); } // Build the tool registry bound to THIS job as the owning subagent. @@ -593,7 +633,7 @@ export function makeSubagentHandler(deps: SubagentDeps) { }; const combinedSignal = mergeSignals(ctx.signal, ctx.shutdownSignal); - assistantMsg = await client.create(params, { signal: combinedSignal }); + assistantMsg = await getClient().create(params, { signal: combinedSignal }); } catch (err) { // Release lease eagerly on error so we don't starve capacity. await releaseLease(engine, lease.leaseId!).catch(() => {}); diff --git a/src/core/model-config.ts b/src/core/model-config.ts index 9b993c8d0e..d9a4858e26 100644 --- a/src/core/model-config.ts +++ b/src/core/model-config.ts @@ -78,6 +78,73 @@ export const TIER_DEFAULTS: Record = { subagent: 'anthropic:claude-sonnet-4-6', }; +const _tierFallbackNoticesEmitted = new Set(); + +/** + * Last-resort tier default, consulted only when NOTHING in the resolution + * chain matched (no CLI flag, no `models.*` config, no env var). + * + * Why this exists: `TIER_DEFAULTS` is Anthropic across the board. For a brain + * with an Anthropic credential that is the right answer and this function is a + * pass-through — historical behavior, byte for byte. For a brain deliberately + * run without one (local Ollama / llama-server, or a hosted non-Anthropic + * provider), an `anthropic:*` default is not a default at all: it is a + * guaranteed failure that reports itself as `NO_ANTHROPIC_API_KEY`, which + * reads as "gbrain requires Anthropic" rather than "nothing told me what to + * use". Users hit this after correctly pointing `chat_model` at Ollama, + * because the gateway's `chat_model` and this tier resolver were two + * independent notions of "the default model" and only one of them was + * consulted here. + * + * So: when no Anthropic credential resolves, prefer the brain's configured + * `chat_model` — the value `gbrain init` already wrote when the user picked a + * provider. Falling back to `TIER_DEFAULTS` when that is absent too is + * deliberate: the resulting error names the missing key honestly, which is the + * correct outcome for someone who simply hasn't finished setup. + * + * Deliberately NOT a probe. This runs during model resolution; reaching out to + * a local daemon here would put a network round-trip in the path of every + * unconfigured call. Readiness is the doctor's job. + */ +export function resolveTierDefault(tier: ModelTier): string { + const anthropicDefault = TIER_DEFAULTS[tier]; + let hasKey = true; + let configuredChat: string | undefined; + try { + // Lazy require for the same cycle-safety reason as capabilities.ts below: + // anthropic-key → config, and config is imported broadly. + // eslint-disable-next-line @typescript-eslint/no-require-imports + const keys = require('./ai/anthropic-key.ts') as typeof import('./ai/anthropic-key.ts'); + hasKey = keys.hasAnthropicKey(); + if (!hasKey) { + // eslint-disable-next-line @typescript-eslint/no-require-imports + const cfgMod = require('./config.ts') as typeof import('./config.ts'); + const cfg = cfgMod.loadConfig(); + const chat = cfg?.chat_model; + if (chat && chat.trim()) configuredChat = chat.trim(); + } + } catch { + // Config unreadable (first-run install) — keep the historical default. + return anthropicDefault; + } + + if (hasKey || !configuredChat) return anthropicDefault; + // An anthropic:* chat_model with no key is the same dead end as the tier + // default; don't dress it up as a substitution. + if (isAnthropicProvider(configuredChat)) return anthropicDefault; + + const key = `${tier}:${configuredChat}`; + if (!_tierFallbackNoticesEmitted.has(key)) { + _tierFallbackNoticesEmitted.add(key); + process.stderr.write( + `[models] no Anthropic credential and no models.tier.${tier} set — using the configured chat_model ` + + `"${configuredChat}" for the ${tier} tier. Pin it explicitly with: ` + + `gbrain config set models.tier.${tier} :\n`, + ); + } + return configuredChat; +} + /** * v0.31.12 subagent runtime enforcement (layer 2). * @@ -106,6 +173,24 @@ export function isAnthropicProvider(modelString: string): boolean { const _subagentTierWarningsEmitted = new Set(); +/** + * True when the resolved model runs on hardware the user already owns, so its + * marginal token cost is zero. Read from the recipe's declared chat pricing + * rather than an id allowlist, so a future local recipe is covered without + * editing this list. + */ +function isFreeLocalProvider(resolved: string): boolean { + try { + // eslint-disable-next-line @typescript-eslint/no-require-imports + const mr = require('./ai/model-resolver.ts') as typeof import('./ai/model-resolver.ts'); + const { recipe } = mr.resolveRecipe(resolved); + const chat = recipe.touchpoints.chat; + return chat?.cost_per_1m_input_usd === 0 && chat?.cost_per_1m_output_usd === 0; + } catch { + return false; + } +} + // Module-level set of deprecated config keys we've already warned about. // Reset on process restart; one warning per (key, process) per Codex P1 #11. const _deprecationWarningsEmitted = new Set(); @@ -191,9 +276,16 @@ export async function resolveModel( } // 7. Tier default (v0.31.12 — when no override beats us, the tier's - // canonical model wins over caller-supplied fallback) + // canonical model wins over caller-supplied fallback). Routed through + // resolveTierDefault so a brain with no Anthropic credential lands on + // its configured chat_model instead of an unreachable anthropic:* id. if (opts.tier && TIER_DEFAULTS[opts.tier]) { - return await resolveAlias(engine, TIER_DEFAULTS[opts.tier]); + const tierDefault = resolveTierDefault(opts.tier); + const resolved = await resolveAlias(engine, tierDefault); + // Same capability gate the config-sourced branches get: a chat_model + // pointed at a tool-less model must not silently become the subagent + // driver just because it arrived via this path. + return enforceSubagentCapable(resolved, opts.tier, `chat_model (tier.${opts.tier} default)`); } // 8. Hardcoded fallback (caller-supplied) @@ -269,10 +361,18 @@ function enforceSubagentCapable(resolved: string, tier: ModelTier | undefined, s if (verdict === 'degraded:no_caching') { if (!_subagentTierWarningsEmitted.has(key)) { _subagentTierWarningsEmitted.add(key); + // A free provider pays for the missing cache in latency, not dollars. + // Telling someone running Ollama that their loop is expensive — and + // that Anthropic would be cheaper — is simply wrong, and it is the + // message a deliberately-local brain would see on every process. + const free = isFreeLocalProvider(resolved); process.stderr.write( `[models] tier.subagent resolved to "${resolved}" via "${source}" — provider does not support prompt caching. ` + - `The loop will run hot (cost scales linearly with conversation length). ` + - `For lower cost on long loops, set models.tier.subagent to an Anthropic model.\n`, + (free + ? `The loop re-sends the whole conversation each turn, so long loops get progressively slower. ` + + `Raise the model's context window if turns start truncating.\n` + : `The loop will run hot (cost scales linearly with conversation length). ` + + `For lower cost on long loops, set models.tier.subagent to an Anthropic model.\n`), ); } } diff --git a/src/core/think/index.ts b/src/core/think/index.ts index 13f451ebca..aca70db0d9 100644 --- a/src/core/think/index.ts +++ b/src/core/think/index.ts @@ -488,7 +488,11 @@ export async function runThink( question: opts.question, answer: modelProblem ? `(model "${modelUsed}" not usable — ${detail}${fix})` - : '(no LLM available — set ANTHROPIC_API_KEY or pass `client`)', + // The warning CODE stays NO_ANTHROPIC_API_KEY for back-compat (it is + // a stable machine-readable signal), but the prose must not imply + // Anthropic is the only option — any chat-capable provider works, + // including a local one with no key at all. + : '(no LLM available — configure a chat model: `gbrain config set chat_model ollama:qwen3` for local, or set a hosted key such as ANTHROPIC_API_KEY / OPENAI_API_KEY)', citations: [], gaps: [ modelProblem @@ -833,7 +837,7 @@ function buildGracefulMessage(modelStr: string): { type: 'message', role: 'assistant', model: modelStr, - content: [{ type: 'text', text: '(no LLM available — set anthropic_api_key via gbrain config or ANTHROPIC_API_KEY env)' }], + content: [{ type: 'text', text: '(no LLM available — set a chat model via `gbrain config set chat_model :`, or provide that provider\'s key via gbrain config / env)' }], usage: { input_tokens: 0, output_tokens: 0 }, stop_reason: 'end_turn', }; diff --git a/test/ai/gateway-chat.test.ts b/test/ai/gateway-chat.test.ts index ff94a65768..aa77db9456 100644 --- a/test/ai/gateway-chat.test.ts +++ b/test/ai/gateway-chat.test.ts @@ -60,9 +60,25 @@ describe('chat touchpoint — recipe registry', () => { } }); - test('embedding-only providers (voyage, ollama) do NOT declare chat', () => { + test('embedding-only providers (voyage) do NOT declare chat', () => { expect(getRecipe('voyage')!.touchpoints.chat).toBeUndefined(); - expect(getRecipe('ollama')!.touchpoints.chat).toBeUndefined(); + }); + + test('local providers declare chat so a keyless brain can run end-to-end', () => { + // Ollama and llama-server were embedding-only through v0.42, which meant + // a local install could index a brain but never `think` over it — the + // synthesis step always needed a hosted key. Both now declare chat with + // tool calling, which is what makes the no-hosted-key path real. + for (const id of ['ollama', 'llama-server']) { + const chat = getRecipe(id)!.touchpoints.chat; + expect(chat, `${id} must declare a chat touchpoint`).toBeDefined(); + expect(chat!.supports_tools, `${id} chat must support tools`).toBe(true); + expect(chat!.supports_subagent_loop).toBe(true); + // Local inference is free at the margin; a nonzero cost here would + // corrupt every --max-usd pre-flight for local brains. + expect(chat!.cost_per_1m_input_usd).toBe(0); + expect(chat!.cost_per_1m_output_usd).toBe(0); + } }); test('openai-compat chat recipes have base_url_default', () => { @@ -172,8 +188,23 @@ describe('chat touchpoint — model resolver + aliases (Codex F-OV-5)', () => { test('assertTouchpoint rejects chat on embedding-only providers with a fix hint', () => { expect(() => assertTouchpoint(getRecipe('voyage')!, 'chat', 'voyage-3')) .toThrow(AIConfigError); - expect(() => assertTouchpoint(getRecipe('ollama')!, 'chat', 'nomic-embed-text')) - .toThrow(AIConfigError); + // The hint points at the local lane first — a user who hit this wall is + // usually trying to avoid a hosted key, not shopping for a hosted one. + try { + assertTouchpoint(getRecipe('voyage')!, 'chat', 'voyage-3'); + throw new Error('should have thrown'); + } catch (e) { + expect((e as AIConfigError).fix ?? '').toContain('ollama'); + } + }); + + test('assertTouchpoint accepts chat on local providers (arbitrary model ids)', () => { + // openai-compat tier: the model list is advisory, so any model the user + // has actually pulled/launched resolves. Rejection surfaces at the + // provider, not in gbrain. + expect(() => assertTouchpoint(getRecipe('ollama')!, 'chat', 'qwen3')).not.toThrow(); + expect(() => assertTouchpoint(getRecipe('ollama')!, 'chat', 'some-model-i-pulled')).not.toThrow(); + expect(() => assertTouchpoint(getRecipe('llama-server')!, 'chat', 'local-gguf')).not.toThrow(); }); test('assertTouchpoint rejects unknown native model with the model list in the fix hint', () => { diff --git a/test/ai/local-discovery.test.ts b/test/ai/local-discovery.test.ts new file mode 100644 index 0000000000..b5ba274564 --- /dev/null +++ b/test/ai/local-discovery.test.ts @@ -0,0 +1,180 @@ +/** + * Local-model discovery + tier ranking. + * + * `rankForTiers` is pure, so the policy is pinned here against fixtures rather + * than against whatever happens to be pulled on the machine running CI. + * + * The fixture is modelled on a real fleet, including the trap that motivated + * the `completion AND tools` predicate: several Ollama embedding models + * advertise `tools` without `completion`, and the largest of them outranks + * most genuine chat models by size. + */ + +import { describe, expect, test } from 'bun:test'; +import { + rankForTiers, + effectiveContext, + ollamaApiRoot, + type LocalModelInfo, +} from '../../src/core/ai/local-discovery.ts'; + +function model( + name: string, + caps: string[], + gb: number, + trainedContext?: number, +): LocalModelInfo { + return { + name, + capabilities: caps, + bytes: gb * 1e9, + ...(trainedContext !== undefined ? { trainedContext } : {}), + chatCapable: caps.includes('completion') && caps.includes('tools'), + }; +} + +/** Mirrors a real fleet: 7 chat-capable models and 7 embedders. */ +const FLEET: LocalModelInfo[] = [ + model('qwen3.6:35b-mlx', ['completion', 'vision', 'thinking', 'tools'], 21.9, 262144), + model('gpt-oss:20b', ['completion', 'tools', 'thinking'], 13.8, 131072), + model('gemma4:12b-mlx', ['completion', 'tools', 'thinking'], 7.7, 262144), + model('ornith:latest', ['completion', 'tools', 'thinking'], 5.6, 262144), + model('lfm2.5:latest', ['completion', 'tools', 'thinking'], 5.2, 128000), + model('granite4.1:3b', ['completion', 'tools'], 2.1, 131072), + model('llama3.2:latest', ['completion', 'tools'], 2.0, 131072), + // The trap: 7.6B, advertises tools, cannot generate text. + model('qwen3-embedding:8b', ['tools', 'embedding'], 4.7, 40960), + model('qwen3-embedding:4b', ['tools', 'embedding'], 2.5, 40960), + model('qwen3-embedding:0.6b', ['tools', 'thinking', 'embedding'], 0.6, 40960), + model('nomic-embed-text:latest', ['embedding'], 0.3, 2048), + model('mxbai-embed-large:335m', ['embedding'], 0.7, 512), + model('embeddinggemma:latest', ['embedding'], 0.6, 2048), + model('nomic-embed-text-v2-moe:latest', ['embedding'], 1.0, 2048), +]; + +describe('rankForTiers — embedding models never reach a chat tier', () => { + test('a tools-advertising embedding model is rejected, not ranked', () => { + const a = rankForTiers(FLEET)!; + const assigned = Object.values(a.tiers); + for (const id of assigned) { + expect(id, 'no embedding model may be assigned').not.toContain('embedding'); + } + // And the rejection is explained, not silent — the reason has to name the + // actual disqualifier so a user can tell it from "model not pulled". + const trap = a.rejected.find(r => r.name === 'qwen3-embedding:8b'); + expect(trap).toBeDefined(); + expect(trap!.reason).toContain('no completion'); + }); + + test('filtering on tools alone WOULD have picked one (the trap is real)', () => { + // Guards the predicate itself: if someone relaxes `completion AND tools` + // to just `tools`, this documents exactly what breaks. + const toolsOnly = FLEET.filter(m => m.capabilities.includes('tools')) + .sort((a, b) => b.bytes - a.bytes); + const wouldRank = toolsOnly.map(m => m.name); + expect(wouldRank).toContain('qwen3-embedding:8b'); + // It outranks four genuine chat models by size. + expect(wouldRank.indexOf('qwen3-embedding:8b')).toBeLessThan(wouldRank.indexOf('granite4.1:3b')); + }); + + test('all 7 embedders are rejected and all 7 chat models are candidates', () => { + const a = rankForTiers(FLEET)!; + expect(a.rejected).toHaveLength(7); + expect(new Set(Object.values(a.tiers)).size).toBeGreaterThanOrEqual(3); + }); +}); + +describe('rankForTiers — tier policy', () => { + const a = rankForTiers(FLEET)!; + + test('deep gets the largest chat model', () => { + expect(a.tiers.deep).toBe('ollama:qwen3.6:35b-mlx'); + }); + + test('reasoning gets the runner-up, so deep-tier latency is not paid on every call', () => { + expect(a.tiers.reasoning).toBe('ollama:gpt-oss:20b'); + expect(a.tiers.reasoning).not.toBe(a.tiers.deep); + }); + + test('subagent shares the reasoning workhorse', () => { + expect(a.tiers.subagent).toBe(a.tiers.reasoning); + }); + + test('utility prefers the smallest NON-thinking model', () => { + // Classification returns a label; reasoning tokens are pure overhead. + // llama3.2 (2.0GB, no thinking) beats granite (2.1GB, no thinking) on size. + expect(a.tiers.utility).toBe('ollama:llama3.2:latest'); + }); + + test('utility falls back to smallest when every candidate thinks', () => { + const allThinking = FLEET.filter(m => m.chatCapable && m.capabilities.includes('thinking')); + const b = rankForTiers(allThinking)!; + expect(b.tiers.utility).toBe('ollama:lfm2.5:latest'); // 5.2GB, smallest + expect(b.reasons.utility).toContain('every candidate has thinking'); + }); +}); + +describe('rankForTiers — degenerate fleets', () => { + test('a single chat model maps every tier to it (matches the no-profile behavior)', () => { + const one = [model('qwen3', ['completion', 'tools'], 5)]; + const a = rankForTiers(one)!; + expect(new Set(Object.values(a.tiers))).toEqual(new Set(['ollama:qwen3'])); + }); + + test('two chat models do not strand the reasoning tier on the smaller one', () => { + const two = [ + model('big', ['completion', 'tools'], 20), + model('small', ['completion', 'tools'], 2), + ]; + const a = rankForTiers(two)!; + expect(a.tiers.deep).toBe('ollama:big'); + expect(a.tiers.reasoning).toBe('ollama:big'); // runner-up rule needs 3+ + expect(a.tiers.utility).toBe('ollama:small'); + }); + + test('no chat-capable model returns null rather than a bogus assignment', () => { + expect(rankForTiers(FLEET.filter(m => !m.chatCapable))).toBeNull(); + expect(rankForTiers([])).toBeNull(); + }); + + test('ranks by bytes, so quantized/MLX builds with no parameter_size still place', () => { + // Every *-mlx model in a real fleet reported no parameter_size; a + // parameter-based sort would silently drop them to the bottom. + const mlx = FLEET.filter(m => m.chatCapable && m.name.includes('mlx')); + expect(mlx.length).toBeGreaterThan(0); + for (const m of mlx) expect(m.parameterSize).toBeUndefined(); + expect(rankForTiers(FLEET)!.tiers.deep).toBe('ollama:qwen3.6:35b-mlx'); + }); +}); + +describe('effectiveContext — clamp to what the daemon serves', () => { + test('clamps a model trained higher than the daemon serves', () => { + // The dangerous direction: an un-tuned daemon serves 4096 while the model + // advertises 131072. Recording the trained value makes gbrain send a + // prompt Ollama silently truncates from the front, losing the retrieved + // context with no error anywhere. + expect(effectiveContext({ trainedContext: 131072 }, 4096)).toBe(4096); + }); + + test('does not inflate a model trained below what the daemon allows', () => { + expect(effectiveContext({ trainedContext: 8192 }, 131072)).toBe(8192); + }); + + test('falls through when either side is unknown', () => { + expect(effectiveContext({ trainedContext: 8192 }, undefined)).toBe(8192); + expect(effectiveContext({}, 4096)).toBe(4096); + expect(effectiveContext({}, undefined)).toBeUndefined(); + }); +}); + +describe('ollamaApiRoot — native endpoints live beside the OpenAI-compat surface', () => { + test('strips the /v1 suffix the gateway base URL carries', () => { + expect(ollamaApiRoot('http://localhost:11434/v1')).toBe('http://localhost:11434'); + expect(ollamaApiRoot('http://localhost:11434/v1/')).toBe('http://localhost:11434'); + }); + + test('leaves a root without /v1 alone, and never leaves a trailing slash', () => { + expect(ollamaApiRoot('http://gpu-box:11434')).toBe('http://gpu-box:11434'); + expect(ollamaApiRoot('http://gpu-box:11434/')).toBe('http://gpu-box:11434'); + }); +}); diff --git a/test/ai/local-model-providers.serial.test.ts b/test/ai/local-model-providers.serial.test.ts new file mode 100644 index 0000000000..c9907be324 --- /dev/null +++ b/test/ai/local-model-providers.serial.test.ts @@ -0,0 +1,253 @@ +/** + * Local / non-Anthropic provider support. + * + * The invariant under test is a single user-visible claim: a brain configured + * for a local model must be able to run WITHOUT any hosted API key. Through + * v0.42 that claim was false in four independent places, and each one is + * pinned below so it can't silently come back: + * + * 1. `ollama` / `llama-server` declared embeddings only — no chat touchpoint, + * so synthesis could never route to a local model at all. + * 2. `resolveTierDefault` (step 7 of the model-resolution chain) hardcoded + * `anthropic:*`, so a brain that correctly set `chat_model` to Ollama + * still resolved every tier to an unreachable Anthropic id. + * 3. The subagent handler REFUSED non-Anthropic models unless an + * undiscoverable config key was set, even though the provider-agnostic + * gateway loop it was refusing to use supports them. + * 4. The Anthropic SDK client was constructed at worker-REGISTRATION time, + * so a keyless brain threw before any routing decision was made. + */ + +import { describe, expect, test, afterEach } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { getRecipe, listRecipes } from '../../src/core/ai/recipes/index.ts'; +import { assertTouchpoint } from '../../src/core/ai/model-resolver.ts'; +import { + TIER_DEFAULTS, + resolveTierDefault, + _resetDeprecationWarningsForTest, + type ModelTier, +} from '../../src/core/model-config.ts'; +import { classifyCapabilities } from '../../src/core/ai/capabilities.ts'; + +// ── env/config isolation ──────────────────────────────────────────────────── +// Same hazard `test/helpers/no-anthropic-key.ts` documents: a developer machine +// with a real key in ~/.gbrain/config.json makes the "no key" path untestable. +// These tests additionally need to WRITE a config, so they own a temp home +// rather than reusing that helper. + +const _cleanups: Array<() => void> = []; + +function withTempBrainHome(config: Record | null): void { + const origHome = process.env.GBRAIN_HOME; + const origKey = process.env.ANTHROPIC_API_KEY; + const tmp = mkdtempSync(join(tmpdir(), 'gbrain-local-providers-')); + if (config) { + mkdirSync(join(tmp, '.gbrain'), { recursive: true }); + writeFileSync(join(tmp, '.gbrain', 'config.json'), JSON.stringify(config), 'utf-8'); + } + process.env.GBRAIN_HOME = tmp; + delete process.env.ANTHROPIC_API_KEY; + _cleanups.push(() => { + if (origHome !== undefined) process.env.GBRAIN_HOME = origHome; + else delete process.env.GBRAIN_HOME; + if (origKey !== undefined) process.env.ANTHROPIC_API_KEY = origKey; + else delete process.env.ANTHROPIC_API_KEY; + try { rmSync(tmp, { recursive: true, force: true }); } catch { /* best-effort */ } + }); +} + +afterEach(() => { + while (_cleanups.length) _cleanups.pop()!(); + // resolveTierDefault memoizes its one-shot stderr notice; clear it so each + // test observes a fresh resolution rather than a suppressed one. + _resetDeprecationWarningsForTest(); +}); + +// ── 1. local recipes can actually chat ────────────────────────────────────── + +describe('local providers declare every touchpoint a brain needs', () => { + test('ollama covers embedding + expansion + chat', () => { + const r = getRecipe('ollama')!; + expect(r.touchpoints.embedding).toBeDefined(); + expect(r.touchpoints.expansion).toBeDefined(); + expect(r.touchpoints.chat).toBeDefined(); + }); + + test('ollama and llama-server chat can drive the subagent tool loop', () => { + for (const id of ['ollama', 'llama-server']) { + // classifyCapabilities is the gate the subagent queue + handler consult. + // `degraded:no_caching` is the expected verdict — usable, just uncached. + // Anything in the `unusable:*` / `unknown` family means the loop refuses + // the job, which is the pre-fix behavior we are removing. + const verdict = classifyCapabilities(`${id}:some-model`); + expect(verdict, `${id} must be loop-capable`).toBe('degraded:no_caching'); + } + }); + + test('local chat is free — a nonzero cost would corrupt --max-usd pre-flights', () => { + for (const id of ['ollama', 'llama-server']) { + const chat = getRecipe(id)!.touchpoints.chat!; + expect(chat.cost_per_1m_input_usd).toBe(0); + expect(chat.cost_per_1m_output_usd).toBe(0); + } + }); + + test('local recipes accept arbitrary model ids (the user pulled/launched it, not us)', () => { + expect(() => assertTouchpoint(getRecipe('ollama')!, 'chat', 'qwen3:32b')).not.toThrow(); + expect(() => assertTouchpoint(getRecipe('llama-server')!, 'chat', 'whatever.gguf')).not.toThrow(); + }); + + test('local recipes carry a reachability probe — a stopped daemon is the real failure mode', () => { + // No key can be missing for a local provider, so key checks catch nothing. + // The probe is what `gbrain doctor` uses instead. + expect(typeof getRecipe('ollama')!.probe).toBe('function'); + expect(typeof getRecipe('llama-server')!.probe).toBe('function'); + }); +}); + +// ── 2. tier resolution without an Anthropic credential ────────────────────── + +const ALL_TIERS: ModelTier[] = ['utility', 'reasoning', 'deep', 'subagent']; + +describe('resolveTierDefault — the step-7 fallback', () => { + test('with an Anthropic key present, every tier is byte-identical to before', () => { + withTempBrainHome({ chat_model: 'ollama:qwen3' }); + process.env.ANTHROPIC_API_KEY = 'sk-ant-test'; + for (const tier of ALL_TIERS) { + expect(resolveTierDefault(tier)).toBe(TIER_DEFAULTS[tier]); + } + }); + + test('with no key but a local chat_model, every tier resolves to the local model', () => { + withTempBrainHome({ chat_model: 'ollama:qwen3' }); + for (const tier of ALL_TIERS) { + expect(resolveTierDefault(tier)).toBe('ollama:qwen3'); + } + }); + + test('with no key and no chat_model, the Anthropic default stands', () => { + // Deliberate: the resulting error names the missing key honestly. Someone + // who has configured nothing has not chosen a local brain — they have an + // unfinished setup, and inventing an Ollama default for them would fail + // more confusingly (connection refused) than the truthful key error. + withTempBrainHome({}); + for (const tier of ALL_TIERS) { + expect(resolveTierDefault(tier)).toBe(TIER_DEFAULTS[tier]); + } + }); + + test('an anthropic chat_model with no key is not dressed up as a substitution', () => { + withTempBrainHome({ chat_model: 'anthropic:claude-sonnet-4-6' }); + expect(resolveTierDefault('reasoning')).toBe(TIER_DEFAULTS.reasoning); + }); + + test('an unreadable config falls back rather than throwing', () => { + withTempBrainHome(null); // no config.json at all + expect(() => resolveTierDefault('reasoning')).not.toThrow(); + expect(resolveTierDefault('reasoning')).toBe(TIER_DEFAULTS.reasoning); + }); + + test('a hosted non-Anthropic chat_model is honored too (not just local)', () => { + // The fix is about "no Anthropic credential", not about "local" — a brain + // on DeepSeek or xAI hit the exact same dead end. + withTempBrainHome({ chat_model: 'deepseek:deepseek-chat' }); + expect(resolveTierDefault('reasoning')).toBe('deepseek:deepseek-chat'); + }); +}); + +// ── 3. a local model survives the subagent tier gate ──────────────────────── + +describe('subagent tier accepts a local model', () => { + test('resolveModel does not fall back to Anthropic for a local subagent model', async () => { + withTempBrainHome({}); + const engine = { + getConfig: async (k: string) => (k === 'models.tier.subagent' ? 'ollama:qwen3' : null), + } as never; + const { resolveModel } = await import('../../src/core/model-config.ts'); + const got = await resolveModel(engine, { tier: 'subagent', fallback: 'anthropic:claude-sonnet-4-6' }); + expect(got).toBe('ollama:qwen3'); + }); + + test('the no-caching notice does not tell a free provider that Anthropic is cheaper', async () => { + withTempBrainHome({}); + const written: string[] = []; + const origWrite = process.stderr.write.bind(process.stderr); + process.stderr.write = ((s: string) => { written.push(String(s)); return true; }) as typeof process.stderr.write; + try { + const engine = { + getConfig: async (k: string) => (k === 'models.tier.subagent' ? 'ollama:qwen3' : null), + } as never; + const { resolveModel } = await import('../../src/core/model-config.ts'); + await resolveModel(engine, { tier: 'subagent', fallback: 'anthropic:claude-sonnet-4-6' }); + } finally { + process.stderr.write = origWrite; + } + const notice = written.join(''); + expect(notice).toContain('prompt caching'); + // Local inference is free at the margin — a cost warning is simply wrong, + // and it is what a deliberately-local brain would see on every process. + expect(notice).not.toContain('For lower cost'); + expect(notice).not.toContain('cost scales linearly'); + expect(notice).toContain('slower'); + }); +}); + +// ── 4. the new hosted providers ───────────────────────────────────────────── + +describe('vibecody provider delta — xai / cerebras / fireworks / sambanova', () => { + const NEW_IDS = ['xai', 'cerebras', 'fireworks', 'sambanova']; + + test('all four are registered', () => { + const ids = new Set(listRecipes().map(r => r.id)); + for (const id of NEW_IDS) { + expect(ids.has(id), `${id} not registered`).toBe(true); + } + }); + + test('all four are chat-capable and loop-capable', () => { + for (const id of NEW_IDS) { + const chat = getRecipe(id)!.touchpoints.chat; + expect(chat, `${id} must declare chat`).toBeDefined(); + expect(chat!.supports_tools).toBe(true); + expect(chat!.supports_subagent_loop).toBe(true); + expect(classifyCapabilities(`${id}:some-model`)).toBe('degraded:no_caching'); + } + }); + + test('all four require a key and pin a base URL', () => { + for (const id of NEW_IDS) { + const r = getRecipe(id)!; + expect(r.auth_env?.required?.length, `${id} must require a key`).toBeGreaterThan(0); + expect(r.base_url_default, `${id} must pin a base URL`).toBeTruthy(); + // createOpenAICompatible appends only the route, never the version + // segment — a base URL missing /v1 silently 404s every call. + expect(r.base_url_default!.endsWith('/v1')).toBe(true); + } + }); + + test('providers with unpublished per-token rates report unknown, never a guess', () => { + // Fabricating a rate is worse than admitting ignorance: it silently + // corrupts --max-usd pre-flights and est_cost_usd audit rows. + for (const id of ['cerebras', 'fireworks', 'sambanova']) { + const chat = getRecipe(id)!.touchpoints.chat!; + expect(chat.cost_per_1m_input_usd).toBeUndefined(); + expect(chat.cost_per_1m_output_usd).toBeUndefined(); + } + }); + + test('xai carries its published sub-200k tier rate', () => { + const chat = getRecipe('xai')!.touchpoints.chat!; + expect(chat.cost_per_1m_input_usd).toBeGreaterThan(0); + expect(chat.cost_per_1m_output_usd).toBeGreaterThan(chat.cost_per_1m_input_usd!); + }); + + test('fireworks is the only one of the four that can also embed', () => { + expect(getRecipe('fireworks')!.touchpoints.embedding).toBeDefined(); + for (const id of ['xai', 'cerebras', 'sambanova']) { + expect(getRecipe(id)!.touchpoints.embedding, `${id} declares no embedding model`).toBeUndefined(); + } + }); +}); diff --git a/test/config-set.test.ts b/test/config-set.test.ts index 076d98af81..3882aca2c8 100644 --- a/test/config-set.test.ts +++ b/test/config-set.test.ts @@ -14,6 +14,7 @@ import { KNOWN_CONFIG_KEYS, KNOWN_CONFIG_KEY_PREFIXES, isConfigTruthy } from '.. import { suggestNearest } from '../src/core/levenshtein.ts'; import { runConfig } from '../src/commands/config.ts'; import { checkSubagentCapability } from '../src/commands/doctor.ts'; +import { runAgent } from '../src/commands/agent.ts'; import { withEnv } from './helpers/with-env.ts'; import type { BrainEngine } from '../src/core/engine.ts'; @@ -211,6 +212,20 @@ describe('#2753 — the doctor-proposed gateway-loop command is accepted by `con return { engine, setCalls }; } + /** Capture `gbrain agent --help` output. Goes through the real `runAgent` + * dispatcher rather than a new test-only export, so the text asserted here + * is exactly the text a user sees. */ + async function captureAgentHelpAsync(): Promise { + const logs: string[] = []; + const logSpy = spyOn(console, 'log').mockImplementation((...a: unknown[]) => { logs.push(a.join(' ')); }); + try { + await runAgent({} as unknown as BrainEngine, ['--help']); + } finally { + logSpy.mockRestore(); + } + return logs.join('\n'); + } + /** Run `runConfig(engine, args)`, capturing console output + exit code * the way `config-get-plane.test.ts` does for the `get` subcommand. */ async function runConfigCapture( @@ -238,23 +253,25 @@ describe('#2753 — the doctor-proposed gateway-loop command is accepted by `con return { logs, errs, exit }; } - test('doctor-proposed command round-trips through `config set` without --force', async () => { - const check = await withEnv( - { GBRAIN_HOME: home, GBRAIN_CHAT_MODEL: undefined, ANTHROPIC_API_KEY: undefined }, - () => checkSubagentCapability(doctorStubEngine()), - ); - expect(check.status).toBe('warn'); - expect(check.message).toContain('agent.use_gateway_loop'); - - // Pull the exact backtick-quoted command out of the doctor message - // instead of hardcoding it, so the two call sites can't silently drift. - const match = check.message.match(/`(gbrain config set [^`]+)`/); - expect(match).not.toBeNull(); + test('the command we tell users to run round-trips through `config set` without --force', async () => { + // The recommendation used to come from the doctor, which warned whenever + // chat_model was non-Anthropic without a key. That is now a SUPPORTED + // configuration — non-Anthropic models auto-route to the gateway loop — + // so the doctor no longer emits it and this test no longer sources the + // command from there. + // + // The #2753 invariant is unchanged and still worth guarding: the exact + // command string gbrain prints to users must be accepted by `config set`. + // `gbrain agent --help` is now the surface that prints it, so the command + // is extracted from there, keeping the two call sites unable to drift. + const helpText = await captureAgentHelpAsync(); + const match = helpText.match(/(gbrain config set agent\.use_gateway_loop [^\s\n]+)/); + expect(match, 'agent --help must still document the gateway-loop key').not.toBeNull(); // Tokenize the WHOLE command and feed every argument through, rather than - // destructuring the first two and dropping the rest. If doctor ever starts - // recommending a trailing `--force`, that has to fail here — the entire - // point of #2753 is that the recommended command works without it. + // destructuring the first two and dropping the rest. If the help text ever + // starts recommending a trailing `--force`, that has to fail here — the + // entire point of #2753 is that the recommended command works without it. const tokens = match![1].trim().split(/\s+/); expect(tokens.slice(0, 3)).toEqual(['gbrain', 'config', 'set']); expect(tokens).not.toContain('--force'); @@ -269,6 +286,20 @@ describe('#2753 — the doctor-proposed gateway-loop command is accepted by `con expect(setCalls).toEqual([['agent.use_gateway_loop', 'true']]); expect(logs.join('\n')).toContain('Set agent.use_gateway_loop = true'); }); + + test('doctor does NOT warn about a non-Anthropic chat_model with no key', async () => { + // The inverse of the old assertion, pinned deliberately. A hosted + // non-Anthropic model auto-routes to the gateway loop, so warning here + // would tell the user to fix a configuration that is already correct. + // The stub engine has no probe-able local endpoint (chat_model is + // openai:gpt-5), so the check's local-reachability branch is skipped. + const check = await withEnv( + { GBRAIN_HOME: home, GBRAIN_CHAT_MODEL: undefined, ANTHROPIC_API_KEY: undefined }, + () => checkSubagentCapability(doctorStubEngine()), + ); + expect(check.status).toBe('ok'); + expect(check.message).not.toContain('will fail at job submission'); + }); }); describe('#2753 — doctor and the subagent worker share one truthiness set', () => { diff --git a/test/init-provider-picker.test.ts b/test/init-provider-picker.test.ts index c9e43fa90e..59a38afc0c 100644 --- a/test/init-provider-picker.test.ts +++ b/test/init-provider-picker.test.ts @@ -14,16 +14,36 @@ import { } from '../src/commands/init-provider-picker.ts'; describe('printSubagentAnthropicCaveat', () => { - test('writes the canonical D7 caveat lines', () => { + test('names the subagent surfaces the user is about to run on this provider', () => { let buf = ''; printSubagentAnthropicCaveat((s) => { buf += s; }); expect(buf).toContain('subagent features'); expect(buf).toContain('gbrain dream'); expect(buf).toContain('gbrain agent run'); expect(buf).toContain('gbrain autopilot'); - expect(buf).toContain('ANTHROPIC_API_KEY'); - // The caveat must clarify chat alone is fine without it. - expect(buf).toContain('Chat alone'); + }); + + test('does NOT claim subagent features require an Anthropic key', () => { + // This message used to say they "require ANTHROPIC_API_KEY regardless of + // which chat model you pick". That stopped being true when the subagent + // handler began auto-routing non-Anthropic models through the + // provider-agnostic tool loop — and it is shown to a user at the exact + // moment they deliberately chose a non-Anthropic provider, so a stale + // claim here tells them their choice doesn't count. + let buf = ''; + printSubagentAnthropicCaveat((s) => { buf += s; }); + expect(buf).not.toContain('require ANTHROPIC_API_KEY'); + expect(buf).toContain('no\n ANTHROPIC_API_KEY needed'); + }); + + test('states the two things that DO differ on a non-Anthropic provider', () => { + // Tool-calling quality and the absence of prompt caching are the real + // trade-offs. Dropping them would make the note pure reassurance. + let buf = ''; + printSubagentAnthropicCaveat((s) => { buf += s; }); + expect(buf).toContain('tool calling'); + expect(buf).toContain('prompt caching'); + expect(buf).toContain('gbrain doctor'); }); });