th-stream-5xx-retry: chat_stream retries retryable 5xx (0.16.2)#79
Merged
Conversation
|
…ng (bump 0.16.2) The streaming path (chat_stream — every chat turn) retried only transient transport errors on `.send()`, then bailed immediately on any non-success HTTP status. A transient gateway 5xx (groq/LiteLLM 502/503 during an incident or load blip) therefore propagated as an AGENT_ERROR and the chat widget rendered an EMPTY reply — while the non-streaming `chat()` path already retried these. Mirror chat()'s retry loop for the streaming request+status check, BEFORE any stream bytes are read (idempotent — no partial response emitted yet): retry `policy.retry_on_status` (429/5xx) with backoff + Retry-After, keep the transport retry, keep the request dump on final failure. Regression test serves 503→200+SSE from a throwaway local server and asserts the turn recovers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
brentrager
force-pushed
the
th-core-0162-stream-5xx-retry
branch
from
455cb13 to
f386329
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
chat_stream(the path EVERY chat turn takes) retried only transient transport errors on.send(), then bailed immediately on any non-success HTTP status. A transient gateway 5xx — groq/LiteLLM502/503during an incident or load blip — propagated as anAGENT_ERROR, and the chat widget rendered an empty reply. The non-streamingchat()path already retries these (policy.retry_on_status); the streaming path never did (a stale comment even claimed it "drop[s] into the existing status-code retry path" — that path didn't exist).Root-caused during a live prod incident: node-rotation wedged valkey → api-prime down → the LiteLLM gateway
Init:0/1, so gpt-oss-120b returned 502/503 on nearly every request → empty replies across the Chakra Transformation Posture agent.Fix
Mirror
chat()'s retry loop for the streaming request+status check, before any stream bytes are read (idempotent — no partial response emitted yet): retrypolicy.retry_on_status(429/5xx) with backoff +Retry-After, keep the transport-error retry, keep the request dump on final failure.Test
chat_stream_retries_retryable_5xx_then_succeeds— throwaway local server serves503then200+SSE; asserts the turn recovers and a retry occurred. No new deps.Bumps core
0.16.1 → 0.16.2.