fix(llm): retry rate-limited (429) batches instead of dropping them - #403
fix(llm): retry rate-limited (429) batches instead of dropping them#403keshprad wants to merge 18 commits into
Conversation
Add a bounded rate-limit retry policy (5s, 15s, 30s, 60s, 60s, honoring a numeric Retry-After up to 120s) to both the sync and async LLM batch retry loops, applied even for models with native provider retries. Exhausted retries record LLM_RATE_LIMIT_RETRIES_EXHAUSTED in the inspection ledger. Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-Authored-By: Keshav Pradeep <keshavp@nvidia.com>
rng1995
left a comment
rng1995
left a comment
There was a problem hiding this comment.
Requesting changes because supported provider exception shapes still bypass the intended rate-limit behavior. Anthropic retries but ignores the server's Retry-After, while Bedrock throttles are not classified as retryable and the batch is dropped immediately.
| the OpenAI-compatible endpoints reached through ``ChatOpenAI``); raw HTTP | ||
| clients surface it as a 429 status error instead. | ||
| """ | ||
| if type(exc).__name__ == "RateLimitError": |
There was a problem hiding this comment.
[P1] Handle the concrete rate-limit shapes from every supported provider. An actual anthropic.RateLimitError passes this class-name check, but _rate_limit_retry_after_seconds rejects it because it is not OpenAI's APIStatusError; Retry-After: 42 therefore becomes the fixed 5-second delay. Bedrock surfaces exhausted throttles as botocore.exceptions.ClientError with Error.Code=ThrottlingException and HTTP 429, which returns false here and drops the batch. Add provider-neutral status/header extraction (including botocore response metadata/code) and tests built from the real SDK exception types.
|
Powered by Codex: PR council review result. This is a triage signal, not a maintainer approval.
|
3 participants
Summary
Validation
make lint— passedmake format-check— passedpytest -q tests/nodes/test_llm_analyzer_base.py tests/test_inspection_ledger.py— 171 passed29b0dc8base.Attribution
Applied from the attached signed-off patch while preserving Devin AI as the commit author and the original Signed-off-by/Co-Authored-By trailers.