Support for alternative LLM backends (OpenAI-compatible / local vLLM)

[Emilien-Etadam]

Hi,

Thanks for open-sourcing Breeze.

Would you consider decoupling the AI Brain from Anthropic-only BYOK? Use cases:

• Self-hosted setups with local GPUs (vLLM, Ollama)
• Regulated environments where data can't leave the infra
• Experimentation with other models

Config surface could stay minimal:

• provider: anthropic (default) | openai-compatible
• endpoint URL, model name, API key

I realize this isn't a trivial swap given how the Agent SDK drives tool-use and the risk engine — a LiteLLM-style translation layer or a provider abstraction inside the Brain Connector both seem viable, you'll know better which fits.

Transparency: I'd be happy to test against a local vLLM instance, and potentially contribute a PoC behind a feature flag. Any code I'd submit would be AI-assisted (Claude Code) with me driving design and testing — flagging upfront so you know what to expect on review.

Two questions:

• Roadmap item, or explicitly out of scope?
• Is the Agent SDK's tool-use format a hard requirement for the risk engine?

Thanks!

[Emilien-Etadam]

Small correction to my message above — having looked at the code, the right names are aiGuardrails (not "risk engine") and the SDK surface is narrower than I implied.

So my real questions, more precisely:

1. Is provider-pluggability on the roadmap, or out of scope (e.g. commercial differentiation with the hosted Brain)?
2. If open: would you want the abstraction to sit above query() (replacing the call entirely per provider), or somewhere else?
3. Would a PoC behind a feature flag be welcome before any wider discussion?

Thanks.

[ToddHebebrand]

Thanks for the thoughtful write-up, and especially for being upfront about the AI-assisted contribution shape — that's the kind of transparency that makes review easier, not harder.

Honest answer: this is something I'd like to do, I'm just not sure when. It's not out of scope and it's not a deliberate Claude-only stance forever — it's more that the work to do it well is larger than it looks, and I don't have a timeline I'd commit to right now.

The reasons it's non-trivial, in case it's useful context:

• The Agent SDK's tool-use format is currently load-bearing for the risk engine and guardrails. We're not just calling a chat completion — the tool-call shape, the cost-tracking pipeline, and the streaming protocol all assume Anthropic semantics. A LiteLLM-style translation layer can paper over chat completion but tends to leak when tool-calling and structured outputs get involved.
• Prompt caching is wired in at multiple layers (system prompts, MCP tool definitions). vLLM and most OpenAI-compatible servers don't have an equivalent, so the cost story changes meaningfully.
• The safety / risk-engine evaluation runs are tuned per-model. Adding a second model family roughly doubles the eval surface.

If you want to drive a PoC, I'm open to reviewing it. Realistic shape:

• Provider abstraction inside the Brain Connector (not a top-level swap).
• MCP_LLM_PROVIDER=anthropic (default) | openai-compatible, gated by an env flag so it's off-by-default in production.
• Anthropic path stays the source of truth; OpenAI-compatible path is best-effort, no SLA, risk engine may degrade.
• Start with a read-only / non-tool-using path (just chat) before touching the tool-call surface — that way we can see the cost/eval tradeoffs before committing.
• No claims on the docs/marketing pages that "Breeze supports vLLM" until we have an answer for the eval/guardrails delta.

If you'd rather wait for it to come from us, that's fine too — just can't promise a date.

[Emilien-Etadam]

Thanks, that's a clear scope and the constraints make sense. Happy to work within it. Confirming what I've understood so I don't drift: - Flag: `MCP_LLM_PROVIDER=anthropic` (default) | `openai-compatible`, off in production. - Abstraction lives inside the Brain Connector, not above it. - First PR: chat-only, no tool-use surface. Best-effort, no SLA. - Anthropic path stays the source of truth, unchanged behavior when flag is off. - No docs / marketing changes. That's actually a tighter and more reviewable starting point than what I had in mind, so good for me. I'll start with a standalone validation of streaming chat against my local vLLM (Qwen3-32B-FP8) before touching Breeze, then open the draft PR. I'll ping you here once it's up. Thanks again for the detailed reasoning, the prompt caching and per-model eval points in particular are worth flagging in the PR description so reviewers see the tradeoffs explicitly.

…

On May 14 2026, at 8:55 pm, Todd Hebebrand ***@***.***> wrote: Thanks for the thoughtful write-up, and especially for being upfront about the AI-assisted contribution shape — that's the kind of transparency that makes review easier, not harder. Honest answer on roadmap stance: Claude-only is a deliberate design choice today, not a placeholder. The reasons aren't about lock-in so much as about what we lean on from the Anthropic SDK: The Agent SDK's tool-use format is currently load-bearing for the risk engine and guardrails. We're not just calling a chat completion — the tool-call shape, the cost-tracking pipeline, and the streaming protocol all assume Anthropic semantics. A LiteLLM-style translation layer can paper over the chat completion but tends to leak when tool-calling and structured outputs get involved. Prompt caching is wired in at multiple layers (system prompts, MCP tool definitions). vLLM and most OpenAI-compatible servers don't have an equivalent, so the cost story changes meaningfully. The safety / risk-engine evaluation runs are tuned per-model. Adding a second model family doubles the eval surface. That said, I'm open to a feature-flagged PoC if you want to drive it. Realistic shape: Provider abstraction inside the Brain Connector (not a top-level swap). MCP_LLM_PROVIDER=anthropic (default) | openai-compatible, gated by an env flag so it's off-by-default in production. Anthropic path stays the source of truth; OpenAI-compatible path is best-effort, no SLA, risk engine may degrade. No claims on the docs/marketing pages that "Breeze supports vLLM" until we have an answer for the eval/guardrails delta. If that scope works for you, open a draft PR and I'll review. I'd suggest starting with a read-only/non-tool-using path (just chat) before touching the tool-call surface — that way we can see the cost/eval tradeoffs before committing. If the scope ends up much bigger than that, this lands in "later, not now" territory. — Reply to this email directly, view it on GitHub (#505 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/ANO6LHOWIRWVXSP4EAXOGND42YJDLAVCNFSM6AAAAACYC2CTFWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMOJSGIYDOOA). You are receiving this because you authored the thread.

[saracmert]

Slightly off-topic but might be relevant here. Cloudflare posted about their Claude Managed Agents integration last week: https://blog.cloudflare.com/claude-managed-agents/

The bit I found interesting, Anthropic frames it as "decoupling the brain from the hands" the agent loop stays on Anthropic, but the execution environment runs on Cloudflare. Different axis than what we're talking about here, but the same underlying instinct (the LLM core and the tool/execution layer don't have to be the same vendor).

Doesn't change the PoC scope at all, just thought it was worth dropping here as background for the longer-term design question.

[ToddHebebrand]

Thanks @saracmert — that's a useful frame. The CF Managed Agents shape (Anthropic-hosted loop + customer-side execution) is closer to a delegation/sandboxing story than a model-swap story, but the underlying instinct you're pointing at is real: the LLM core and the tool-execution layer don't have to be the same vendor or even the same trust boundary. We'll bookmark it for the post-PoC design discussion if the OpenAI-compatible path lands cleanly — the question of "where does the tool surface live" gets a lot more interesting once you have more than one LLM family in the mix.

No change to the PoC scope discussed with @Emilien-Etadam above.

[Emilien-Etadam]

Draft PR up: #859

Chat-only OpenAI-compatible path behind MCP_LLM_PROVIDER, tested end-to-end
against a local vLLM (Qwen3.6). Limitations and a phase-2 sequencing question
are in the PR description. No rush on review, flagging it here since the
discussion is where we started.

[ToddHebebrand]

Closing the loop here since this discussion is where the work started: @Emilien-Etadam's PoC merged in #859 (814f422e, 2026-06-11) and is live on main.

What shipped, matching the scope we agreed in this thread:

• MCP_LLM_PROVIDER=anthropic (default) | openai-compatible, off unless explicitly set — Anthropic path stays the source of truth, unchanged when the flag is off.
• Provider abstraction lives inside the Brain Connector (apps/api/src/services/llm/openaiCompatibleProvider.ts + openaiSessionManager.ts), not above query().
• Chat-only, no tool-use surface — exactly the first-PR boundary we scoped. Verified end-to-end against local vLLM by the author.

Phase 2 (the tool-use surface) is intentionally not in scope yet — that's the harder piece given how the Agent SDK drives tool dispatch and how aiGuardrails tiers gate it, and it's the natural next conversation. Leaving this open as the home for that follow-up.

Thanks again @Emilien-Etadam for the careful scoping and the clean PoC.

[Emilien-Etadam]

Thanks @ToddHebebrand, glad the PoC landed cleanly.

Picking up the phase-2 thread, then. One new data point that might shape the approach: vLLM 0.23 now exposes an Anthropic-compatible /v1/messages endpoint, and I've verified end-to-end against my local setup (Qwen3.6 27b) that tool-calling works in streaming on both dialects, the OpenAI /v1/chat/completions path and the Anthropic /v1/messages path, including the tool-result round-trip.

That seems to open a door beyond the openai-compatible path: pointing the Agent SDK at a vLLM backend that speaks /v1/messages, rather than building a separate tool-dispatch surface. Given how the SDK and aiGuardrails are wired, how would you want to approach this?

Happy to help shape it whichever direction you think fits.

[ToddHebebrand]

Good call on the vLLM /v1/messages data point. That reframes phase 2.

Rather than build a separate tool-dispatch surface on the openai-compatible path, the cleaner route is to point the existing Agent SDK at an Anthropic-compatible base URL (vLLM /v1/messages, or LiteLLM's Anthropic passthrough). That reuses the real query() path, so it inherits full tool-use and the entire aiGuardrails stack (RBAC, tool tiering, rate limits, cost tracking) for free, none of which the chat-only provider has today.

Concretely it's a small surface. The SDK already honors ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN; today the only gap is that streamingSessionManager's subprocess env allowlist forwards the auth vars but not ANTHROPIC_BASE_URL. Add that plus a config entry and the SDK talks to your backend unmodified. The open questions are model-name handling (LiteLLM aliasing claude-sonnet-4-6 to your backend model is the tidy answer, with an optional ANTHROPIC_MODEL override for raw vLLM) and cost-tracking calibration for non-Anthropic pricing. It will be self-hosted only (hosted stays on Anthropic, fail-closed). Your existing openai-compatible provider stays as the fallback for endpoints that only speak the OpenAI dialect.

Tracked in #1412, with a design spec in the repo (docs/superpowers/specs/2026-06-15-ai-agent-anthropic-base-url.md). Thanks again for verifying the tool-calling round-trip end to end, that's the part that made this the obvious path.

[richardchen874-sys]

The caution around LiteLLM-style translation layers is fair. Tool calls, structured outputs, streaming, and prompt caching are exactly where a generic compatibility shim can start leaking provider-specific behavior.

One middle path might be a small provider abstraction that only standardizes what Breeze needs operationally:

• base URL + model profile
• tool-call capability flags
• cache support / no-cache fallback
• token and cost reporting
• per-task budget and downgrade policy

I'm building a private-beta OpenAI-compatible API aggregation layer focused on Chinese models like DeepSeek / Qwen / GLM-style providers, using official-provider keys. If you want to benchmark official-key Chinese models against Breeze’s agent workflow, I’d be happy to share some test credits.

Curious: is your biggest blocker the API shape, prompt caching semantics, or reliable tool-call behavior across providers?

[ToddHebebrand]

To answer the direct question: reliable tool-call behavior, by a wide margin.

API shape is the smallest problem — the vLLM /v1/messages development discussed above largely dissolves it. Caching matters for cost but degrades gracefully: a backend without cache support just costs more; nothing breaks.

Tool calling is different. Breeze's agent loop is unusually tool-heavy — a large registry of tools with strict schemas, multi-step chains (list devices → check patch state → create job), and guardrails that depend on the model emitting well-formed calls the loop can validate before execution. In an RMM, a malformed or hallucinated tool call isn't a chat glitch; it's an action against a customer fleet that gets blocked, and a model that degrades into prose instead of calls makes the whole assistant appear broken. That failure mode varies wildly across providers and even across quantizations of the same open model, which is why I'm wary of averaging over a compatibility shim: the provider abstraction you sketch (capability flags, cost reporting, budget/downgrade policy) is the right shape precisely because it makes those differences explicit instead of papering over them.

Appreciate the test-credit offer — I'll pass for now while phase 2 is still design-stage, but a benchmark of tool-call fidelity across those providers against a fixed agent-loop harness is exactly the evidence this thread needs when it gets there, so results from your aggregation work would be welcome data here regardless.

[richardchen874-sys]

@ToddHebebrand Understood. Tool-call reliability is the right gate for Breeze, and there is no reason to add a provider before that evidence exists.

When you revisit the comparison, I would keep the benchmark narrow: valid tool-call rate, schema-valid arguments without silent coercion, tool_choice adherence, time to first valid tool call, retry count, and whether malformed calls are surfaced or repaired. A final successful answer alone can hide several failed upstream attempts.

I will not push the test credit now. If I later have reproducible results for those exact dimensions across the OpenAI-compatible Chinese-model endpoints, I can post the model IDs, request shape, and pass/fail evidence here so you can decide whether it is worth evaluating.