(MOT-4358) feat(provider-groq): add the Groq provider worker - #712
Draft
rohitg00 wants to merge 12 commits into
Draft
(MOT-4358) feat(provider-groq): add the Groq provider worker#712rohitg00 wants to merge 12 commits into
rohitg00 wants to merge 12 commits into
Conversation
router::count_tokens resolves the model to its provider with the chat pipeline's routing and forwards to provider::<id>::count_tokens. A provider without a counter returns a typed no_token_counter error so callers fall back to their own estimate. provider-anthropic counts through the count_tokens metering endpoint (derived from the configured messages url, same wire builders as the stream path, no max_tokens or stream fields) and reports estimator "provider". provider-openai and provider-openai-codex count locally with tiktoken (cl100k for the gpt-3.5 and non-o gpt-4 families, o200k otherwise) and report estimator "tiktoken". Counting never runs the model and bills nothing.
…e provider scaffold Review cleanups, wire-identical (goldens unchanged in all three crates). The byte-identical openai and codex counters collapse into llm_router::provider_scaffold::tiktoken_count with the framing constants, encoder selection, and the shared test suite (including the cases one copy had dropped); each provider keeps a thin request adapter and the tiktoken dependency moves to the one crate. embed now detects a missing provider function with the same typed helper count_tokens uses instead of string matching.
…wn vocabulary Counting a DeepSeek model with tiktoken would be wrong in a way nobody could see: an OpenAI-compatible wire shape does not imply an OpenAI vocabulary, so the number would look authoritative while being off by whatever the two disagree about. DeepSeek publishes no metering endpoint but does publish its tokenizer, so the count is computed from that. `provider_scaffold::vocabulary_count` fetches a vocabulary once, caches it under `~/.iii/tokenizers/` behind an atomic rename so a killed process cannot poison the cache, and parses it once per process. Resolving it at runtime rather than compiling a table in is what lets a model announced tomorrow count correctly today. A cold cache with no network returns the typed `no_token_counter` error, leaving the caller on its own estimate rather than reporting a wrong number as exact. `chat_framing` carries the parts every local counter shares — which text a message contributes, what the framing costs, how a tool schema serializes — so tiktoken and vocabulary counting cannot drift apart, and a third tokenizer is a closure rather than a third copy of these rules. tokenizers is pinned with default features off: they pull onig (C) and esaxx (C++), which would need a cross C toolchain on all nine release targets. Measured against DeepSeek's own billed usage on a live rig: 8938 counted, 8938 billed. The heuristic it replaces was reporting a 8416-token system prompt as 5091.
Four providers, three different truths about who owns the tokenizer, so the seam is drawn where the difference actually is. Moonshot and llama.cpp meter a prompt themselves, so the count is simply asked for: Moonshot through its estimator endpoint, llama.cpp through the Anthropic-compatible count route it already speaks. llama.cpp is the one that could not have been solved any other way — the operator loads whichever GGUF they like, and no table compiled into this binary could know which vocabulary sits in memory right now. xAI splits the difference: it publishes a tokenizer but not a prompt meter, so xAI owns the vocabulary and this worker owns the chat framing. The request is tokenized in one call rather than one per row, which costs a separator token per join, and xAI's own FAQ notes the tokenizer can disagree with billing. Z.AI publishes neither, but GLM's vocabulary is public, so it counts the way DeepSeek does. Borrowing tiktoken here would have been worst of all: GLM's vocabulary disagrees with it most on the Chinese text these models are used for. `endpoint_count` carries what the metered providers share — a bounded timeout, a status check that keeps the upstream's own words, and pulling a number out of a reply whose shape nobody agrees on. `chat_framing` gains `frame`, splitting "what gets counted" from "how it adds up" so a remote tokenizer can batch a whole request into one call. Estimator strings now say which kind of answer a count is: `metered` when the upstream produced it, `tokenizer` when a real vocabulary did locally.
…not one segment up
Verifying kimi against Moonshot caught the bug: `chat/completions` is two
path segments, so cutting one built
`…/v1/chat/tokenizers/estimate-token-count`, which no upstream serves. xAI
had it too — both default to a `/v1/chat/completions` endpoint. Routes now
hang off the API base the same way each provider's discovery already
derives its models route, with tests naming the wrong URL so it stays out.
Both counting endpoints are now verified against real servers rather than
documentation. Moonshot answers `{"data":{"total_tokens":N}}`, and returns
9 tokens for kimi-k2.5 against 87 for kimi-k3 on a byte-identical body —
model-side overhead no local estimate could have known about, which is the
argument for metered counting in one number. llama.cpp answers
`{"input_tokens":N}` on its Anthropic-compatible route, verified against a
running llama-server; its URL derivation is now tested too, since that is
the half a golden cannot check.
Groq is an inference host rather than a model vendor, and that is the whole difference. Every provider before it serves one family, so one set of answers held for the provider as a whole. Here a Llama, a GPT-OSS and a Qwen model sit behind one endpoint, and three things stop being provider-wide facts: Counting picks the vocabulary per model. A single fixed one would be wrong for most of the catalog, and borrowing tiktoken for all of it would be wrong quietly — the number would read as authoritative while being off by whatever the vocabularies disagree about. A model no rule recognizes gets the typed `no_token_counter` instead of a guess, which leaves the caller exactly where it would have been without the function. Meta's repositories are gated behind a licence click a worker cannot perform, so Llama resolves through a public mirror of the identical tokenizer. Reasoning is per model: the GPT-OSS models take `reasoning_effort`, the Llama models do not reason at all, so the catalog marks it per row. Groq has no `thinking` object to enable first, and its ladder stops at `high`, so `xhigh` saturates rather than inventing a tier the API would reject. The catalog is mostly Groq's own: `GET /models` reports `context_window` and `active` per model, both taken live, so a window Groq raises arrives without a release and a model that cannot serve is never offered. Speech models share that listing and are dropped — the absence of a context window is what tells them apart — while a gateway that reports no windows at all is left alone, since requiring the field there would empty the catalog. Pricing comes from published third-party tracking: Groq's own pricing page renders client-side and ships no figures in the document. Noted in the code so nobody mistakes it for a vendor source. Stacked on MOT-4329, whose vocabulary scaffold this uses. No live verification yet — there is no Groq key on the rig, and everything else in this series was verified at the wire before shipping.
Verifying against the live API rewrote most of what the docs implied. `GET /models` turns out to carry the display name, the context window, the output ceiling, the modalities, the supported features and live per-token pricing. So the hand-kept table is gone: keeping a local price list beside a live one is strictly worse, because it goes stale in silence. What is left locally is the floor a sparse row falls back to, and it now claims no capability it has not been told about — a host serving other people's models has no provider-wide answer to "does this take tools", and `llama-3.1-8b-instant` and `allam-2-7b` genuinely disagree. Two things the docs got wrong and the wire did not: `whisper-large-v3` reports a context window like everything else (448), so the rule that dropped speech models by the absence of one dropped nothing and would have put Whisper in the model picker. Modality is what separates them, and that is what the filter reads now. `qwen/qwen3.6-27b` accepts images. The catalog declared no Groq model could, because at a single-family provider that was a provider-wide fact worth hardcoding. Here it is per model, read from `input_modalities`. Also live: a prompt over the per-minute token budget comes back as HTTP 413 carrying `code: rate_limit_exceeded`. The status alone reads as "too large for the model", which would send the router off to compact a prompt that was never too large — it was too large this minute. The envelope code wins over the status now, with the captured body as the test. Prices are scaled from per-token strings, and rounded: 0.00000079 times a million is 0.7899999999999999 in binary floating point, which was reaching the catalog verbatim. Verified on the rig: 11 chat models discovered from 15 rows, speech dropped, capabilities and pricing live, and per-model vocabularies counting through three different tokenizers with a typed refusal for the family that has none.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 55 skipped (no docs/).
Four for four. Nicely done. |
…ble that is gone The curated table was removed when the live listing turned out to carry per-token rates; the README kept claiming third-party tracking populates the catalog, which is now the opposite of what the code does.
…ools Reading `supports_tools: false` on groq/compound invites someone to treat it as a gap to route around. It is a system rather than a model: Groq runs it with web search and code execution of its own, so it declines function definitions from a caller by design. Also records that the listing is per-account, which is the argument for reading it instead of shipping a table that would offer models a key cannot reach.
rohitg00
marked this pull request as draft
August 5, 2026 14:04
… is not a rate limit Live behaviour caught this: an agent turn retried three times against `Limit 8000, Requested 39082` and could not have succeeded on any of them. Two different failures share the `rate_limit_exceeded` code. When the quota for the minute is merely spent, waiting is the fix and backoff is right. When a single request is larger than the entire per-minute allowance, waiting fixes nothing and the retries only burn the turn; the only thing that helps is a smaller prompt, which is what the upstream is asking for in the same sentence. Groq states both numbers, so they are compared and the classification follows the arithmetic rather than the code. A message with no such pair keeps the ordinary rate-limit reading.
… asked for Groq's per-minute token budget counts reserved output, not just the prompt. Measured against the live API: "hi" asking for llama-3.3-70b's full 32,768 ceiling is rejected on a 12,000 TPM key, the same prompt with the field omitted succeeds, and with 8,192 it succeeds too. The prompt was never the problem. So `max_completion_tokens` now rides only when a caller or the operator asked for a ceiling. Defaulting to one meant inventing a reservation, and on this provider an invented reservation is spent budget. Worth stating plainly in the README because the failure points at the wrong thing: a 7k prompt against a 12,000 TPM key leaves under 5k for output, so a caller reserving the model's advertised ceiling fails every time while the same conversation with a modest ceiling goes through.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
provider-groq, a Groq Chat Completions provider worker behind llm-router, following the same protocol as the other providers:stream,abort,refresh_models,on_router_ready, pluscount_tokens.Stacked on #685. The base is
feat/provider-token-counting, whose vocabulary scaffold this uses. Merge #685 to main first, then this.Why this one is shaped differently
Groq is an inference host, not a model vendor. Every provider before it serves one family, so a single answer held for the provider as a whole. Here a Llama, a GPT-OSS and a Qwen model sit behind one endpoint, and three things stop being provider-wide facts:
no_token_counterinstead of a guess, which leaves the caller exactly where it would have been without the function. Meta's repositories are gated behind a licence click a worker cannot perform, so Llama resolves through a public mirror of the identical tokenizer.qwen/qwen3.6-27btakes images; nothing else here does.The catalog is mostly Groq's own
GET /modelsturned out to carry the display name, context window, output ceiling, modalities, supported features and live per-token pricing. All of it is read from the listing rather than kept locally, so a window Groq raises or a price it changes arrives without a release. Only the floor for a sparse row lives in the worker, and it claims no capability it has not been told about.Keeping a local price table beside a live one would be strictly worse: it goes stale in silence.
What live verification changed
The worker was first built from the docs, then checked against the API. Three things only the wire showed:
whisper-large-v3reports a context window like every other model (448), so the rule that dropped speech models by the absence of one dropped nothing, and Whisper would have appeared in the model picker. Modality is what separates them.qwen/qwen3.6-27baccepts images, which the catalog had declared impossible for every Groq model.code: rate_limit_exceeded. Reading the status alone would send the router off to compact a prompt that was never too large for the model, it was too large that minute. The envelope code wins over the status now, with the captured body as the test.Prices are also rounded when scaled from per-token strings:
0.00000079times a million is0.7899999999999999in binary floating point, and that was reaching the catalog verbatim.Verified on a live rig
Catalog, from a real key:
router::count_tokensresolved three separate vocabularies (Llama, GPT-OSS, Qwen) and returned the typedno_token_counterforallam-2-7b, which has no published rule.Generation, on
qwen/qwen3.6-27b:Tool calling, same model:
So streaming, usage accounting, cost fill from live pricing, reasoning-token capture and the tool-call wire format are all exercised against the real API.
Error classification was exercised too, across both messages that share the
rate_limit_exceededcode:Rate limit reached ... try again in 1.5sbacks off and retries, whileRequest too large ... Limit 8000, Requested 23682fails once without retrying, because no wait makes a request larger than the whole per-minute allowance fit.Not verified
A long-form generation and a multi-step agent loop. The key available was rate limited at 8,000 tokens per minute, and Groq counts reserved output against that budget, so turns fit only with a small output ceiling configured (
providers.groq.max_tokens). Nothing about that is a property of the worker, but it does mean sustained agent work over many steps has not been run.Gates
cargo fmt --check,clippy -D warnings, 96 tests across unit, schema and integration suites, goldens generated for all five functions.