feat(models): OpenAI models on an Azure OpenAI resource, and a prompt-size query - #54
Merged
Conversation
…rows A prompt-cache entry is only worth writing if the NEXT request can read it, which requires its hashed prefix to be bytes that request still contains. When a caller appends per-request content after the durable conversation — a clock, a budget figure, anything derived from "now" — the unconditional last-message breakpoint lands on exactly the row the next request cannot reproduce, so every request writes a fresh entry and reads none. `CACHE_EXEMPT_KEY` is a neutral per-item key on the caller's own message dict (the `defer_loading` shape) marking a row as per-request. The marker then lands on the last durable row, which the next request contains by construction since durable rows only ever grow by append. Counted from the end and stopping at the first unmarked row: the exemption is about position, so a marked row with durable rows after it is not a tail. An all-exempt list writes no marker at all. Measured on Bedrock/Opus 5, single-step turns with tools present: marker on the volatile row: turn 1 write=8211 -> turn 2 write=8226 read=0 marker on the last durable row: turn 1 write=8425 -> turn 2 write=15 read=8425 With no exempt row the payload is byte-identical to before, so no existing caller changes behaviour. Twinned on the OAuth leg, which is not a subclass and has its own payload builder. The key never reaches the wire — both builders rebuild rows with only the wire-legal fields.
… prefix once
Four defects in the Responses-API adapter, all found by putting a real reasoning
deployment behind it. Each is a property of the API rather than of any one vendor's
hosting of it, so all four are fixed here on the shared adapter instead of in a
subclass.
The request forwarded a caller's whole message dict. `msg.copy()` plus a hand-pruned
`name` field meant every other annotation a caller had put on its own working object
went to the wire, and this endpoint answers an unrecognized per-item key with
`400 unknown_parameter: input[0].<key>` — measured against a live resource with
`kind`, the reasoner's routing tag. So an annotating caller could not talk to the
endpoint at all, and pruning offenders one at a time only defers the next 400 to the
next key someone adds. Rebuilt from the allow-list the input array actually accepts:
`role`, `content`, `type` — `type` kept because the array uses it to discriminate a
message from a function_call output, which the Anthropic adapter's narrower
`{role, content}` rebuild does not have to handle.
`cached_tokens` was dropped on the floor and `input_tokens` mapped straight onto
`UsageInfo.prompt_tokens`. The two conventions are inverses: this API reports the
cached figure as a SLICE of `input_tokens` (measured: `input_tokens: 3398` containing
`cached_tokens: 3395`, with `3398 + output 5 == total 3403`), while `prompt_tokens`
means the uncached REMAINDER with the cache counts beside it and `full_prompt_tokens`
summing all three. Mapped field-to-field, the cached prefix is counted twice and the
half of it that lands in the prompt figure is priced at the fresh-input rate — on that
measured call, 3,398 billable prompt tokens where 3 were owed. Subtracted once here,
clamped at zero: a vendor's slice cannot exceed the whole it came from, and a negative
prompt count would walk a spend ledger backwards.
`thinking_budget` reached this leg and did nothing. It is the deliberation knob every
caller in this stack sets, the Responses API takes `reasoning.effort` buckets instead,
and nothing mapped between them — so a caller's setting was inert and every call ran
at the provider default (`medium`), which is indistinguishable from the knob working.
Bucketed at 1024/4096/16384, following the Anthropic family's 1024 minimum; a
non-positive budget means thinking is off, which is not a request to think as little
as possible, so it omits the parameter rather than pinning `minimal`.
`provider` was the literal "openai" in the harmonized metadata and in the error path,
while the retry path beside it already reported the subclass's own id. A subclass
serving the same wire contract from a different host was therefore reported as
first-party OpenAI in usage metadata and in every failure — and `metadata.model` plus
`metadata.provider` are what a cost table is keyed on, where a miss prices at zero
silently.
A new `azure` provider, shaped like `bedrock`: a thin subclass over the adapter that already speaks the wire contract, plus the registry rows that make it addressable. On the v1 GA surface an Azure OpenAI resource serves the same Responses API as first-party OpenAI, so the subclass is 3 overrides and no request or response handling of its own. Three deltas, each measured against a live resource rather than inferred: * the base URL names a customer's own resource, so there is no literal to write. It is read from the environment and normalized, because the value an operator copies out of the portal is the project form `<host>/api/projects/<project>` — which does not serve `/openai/v1` at all. Normalization accepts the bare host, the project form and the fully-qualified base, idempotently. * auth is the `api-key` header. (A bearer also returns 200 on this resource, but the key-shaped credential belongs in the key-shaped header.) * the `model` field carries a DEPLOYMENT name, which is an operator's choice of string and is echoed back verbatim. No renormalization: the response's `model` is exactly what a cost table should be keyed on. Nothing about the legacy surface is carried: no `/deployments/<name>/` path segment and no pinned `api-version` query. Both are what the v1 API removed. `PROVIDER_BASE_URLS` is built once at import, which is fine for a vendor-wide endpoint and wrong for a per-resource one: a process that learns its resource after this module was imported — the normal order, since configuration is injected at startup and this module is imported by the package — would read that import-time snapshot forever, and for Azure the snapshot is typically the empty string, so every request would go nowhere while looking configured. `default_base_url()` re-resolves the two providers whose endpoint is not vendor-wide (Bedrock's carries a region, Azure's names a resource) and the config validator goes through it. A known provider with no resolvable endpoint now warns that its base URL is per-resource, which is a different failure from an unknown provider name and needs to say so — the fix is to supply an endpoint, not to correct a spelling. The error classifier's OpenAI branch is shared rather than duplicated, the way `bedrock` already shares Anthropic's: one error envelope, one classification. Without the share an Azure 429 matches no branch, classifies as UNKNOWN and comes back not retryable, so a rate-limited request is dropped instead of backed off. Prompt caching on this API takes no request parameter — it is implicit, and a `prompt_cache_breakpoint` is rejected as an unknown parameter in every position and in both cache modes. Verified by measurement: two identical calls report a write and then a read of the same 3,395 tokens with no cache field sent at all.
`minimal` is a 400 on Azure OpenAI — the endpoint's own reply enumerates the six efforts it takes (none / low / medium / high / xhigh / max) — so the smallest thinking budget could not reach that leg at all: every caller with a sub-1024 budget took a hard rejection instead of a cheap answer. The payload builder already had one such exception inline (codex does not serve `minimal` either). Two special cases in one conditional is where a hook belongs, so the substitution moves behind `_served_effort(effort, model_lower)` on the base adapter, keeping the codex mapping byte-identical, and the Azure subclass overrides it. Unconditional there rather than keyed on the model name: on that surface the name is an operator-chosen deployment label and says nothing about the generation underneath. `low` and not `none`, because a positive budget means "think a little" and `none` would answer a question nobody asked.
The adapter took the resource URL under two names — `base_url` and `endpoint` — and only the second was normalized. A caller handing over the bare resource host, or the portal's copyable project URL, as `base_url` therefore built a client that POSTs to `/responses` off the host root and 404s on every call; the same string arriving as `endpoint` worked. One idempotent normalizer now sees both, so the already-normalized `.../openai/v1` base a configured caller threads in is unchanged.
The model port had no way to learn a token count. Every consumer that needed one estimated it from characters, which is a property of the content rather than of the model — the same text can run 1.1 to 3.2 characters per token depending on what it is, so a bound computed that way is not an estimate with a wide error bar, it is a number with no relationship to the quantity it names. Anthropic's /v1/messages/count_tokens answers with the model's own tokenizer, is free of charge, is rate-limited independently of message creation, and does not participate in prompt caching, so a caller may count the exact payload it is about to send without perturbing anything. BaseAPIModel.acount_tokens returns None by default: a provider without a counting service must say so rather than be guessed for, and the caller needs to tell an absent count from a small one. The two Anthropic legs implement it on their async adapters, each through its OWN format_request_payload — the OAuth leg prepends a required system block and renames reserved tools, so a body assembled any other way would count a request nobody makes. Only the generation controls come off. Failure is None, never an exception: the caller is sizing something, not producing the turn's answer. A 404/405 says the route is absent and is remembered for the process; a credential-shaped refusal is deliberately not, because the OAuth token file has several writers and a refresh in flight looks exactly like a rejection for one request.
Measured against the live API: counting a message list whose last row is an
assistant reply ending in whitespace is refused —
invalid_request_error: messages: final assistant content cannot end with
trailing whitespace
The messages call almost never meets this rule, because something is appended
after the last assistant turn on every real request. A count meets it
constantly: counting a conversation means presenting its last row as final, a
settled conversation ends with an assistant reply by definition, and models end
replies with a newline. So the first caller of this endpoint would have found
its post-fold counts failing on exactly the conversations most likely to need
them, with nothing on the wire to say why.
The count payload now right-trims the final assistant message's last text block,
copying rather than mutating (those blocks can be the caller's own durable rows),
and drops the message when nothing is left of it. And a non-200 logs the
provider's own sentence rather than a bare status — this one answers in a line,
and a status alone cost a round trip to diagnose.
feat(models): a caller can ask the provider how big a prompt is
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the Azure OpenAI provider leg onto
main, together with the token-countingcall Spren's metering depends on. The branch is 10 ahead / 0 behind
main, so thisis a fast-forward.
Contents:
feat(models): OpenAI models on an Azure OpenAI resource— the Azure OpenAI adapter.feat(models): a caller can ask the provider how big a prompt is(feat(models): a caller can ask the provider how big a prompt is #53, already mergedinto this branch) — the count-tokens leg, plus both spellings of the endpoint going
through one normalizer.
feat(models): a caller can keep the cache breakpoint off per-request rows— the samecommit as feat(models): a caller can keep the cache breakpoint off per-request rows #50, which this subsumes.
fix(models): the OpenAI leg sends only wire keys, and counts a cached prefix once.Verified locally against this branch:
pytest tests/models→ 282 passed, 39 skipped(
mainis 201 passed, 39 skipped; the leg adds 81). The three collection errors intests/coordination/**are pre-existing onmainand environment-only (readlineis POSIX-only,
hypothesisnot installed).Spren pins this branch's tip today, off-main; merging puts the pin back on
main.