feat(agents): Harden the LLM call path against relay failure modes

[yebof]

Summary

Five changes to src/agents/base.py, each the codified fix for an observed production incident on the OPENAI_BASE_URL relay:

1. Stream OpenAI calls (stream=True + include_usage). The relay sits behind Cloudflare's ~120s proxy read timeout (HTTP 524), so any non-streamed generation over 120s could never succeed through it (2026-06-08/09: every long call 524'd and two mornings died with a funded failover key sitting idle). Streaming keeps bytes flowing; _LLM_HTTP_TIMEOUT becomes a per-chunk read timeout. A relay that ignores include_usage falls back to a loud chars/4 token estimate instead of 0/0.
2. Wall-clock retry deadline — 480s default, QUANT_AGENT_RETRY_DEADLINE_S override. The attempt budget alone doesn't bound time: 7 attempts × 120–380s collided with the wrapper's 1200s kill, so the Anthropic failover below the loop never fired in exactly the sustained-outage scenario it was built for. Past the deadline the primary is abandoned and failover runs while the session window still has room (480 + 300 failover < 1200).
3. Honor server retry-after hints on 429/524 (Retry-After header, body field, or message text), capped at 120s. Pure exponential jitter retried in 2–15s against a server that said "come back in 120s".
4. Per-provider in-flight semaphores around the HTTP call: OpenAI 3 (QUANT_AGENT_MAX_CONCURRENT_LLM), Anthropic 4 — independent, so failover never queues behind a wedged relay slot. The morning fan-out self-inflicted "Concurrency limit exceeded" 429 storms (175 occurrences in the 06-16..06-29 logs).
5. max_retries=0 on every SDK client — both SDKs default to 2 internal retries, silently tripling each agent-loop attempt and invalidating the retry-budget/deadline math. The agent loop is the single retry owner.

Degenerate responses now fail loudly instead of passing as clean no-signals: an empty 200 body raises LLMEmptyResponseError, a stream that ends without finish_reason raises LLMStreamInterruptedError (partial text discarded — a half-emitted PM decision parses like "no trades"). Both are retryable and reach the failover. Truncation-family finish reasons (max_tokens / length / insufficient_system_resource) are exempt: an empty body there is a legit ceiling hit surfaced via truncated=True, never a retry.

⚠️ Merge gate

scripts/relay_stream_smoke.py must PASS against the live relay before merging — it makes one real streamed gpt-5.5 call and checks content / finish_reason / usage. The relay was down (TLS handshake failure) throughout development, so the load-bearing assumption — the relay accepting stream=True + stream_options — is still unverified. If the relay 400s on stream_options, every production call would fail over to Anthropic.

Test plan

• Full suite: 1251 passed (was 1 failure: the old non-streamed OpenAI mock, fixed)
• 15 new tests: stream assembly + usage-chunk / estimate fallback, interrupt + empty-body guards on all three providers (incl. truncation exemptions), deadline-triggered failover, retry-after hint floor + cap, max_retries=0 on all four client constructions, semaphore leak check
• scripts/relay_stream_smoke.py against the live relay (blocked: relay down)
• Watch one live morning post-merge for FAILOVER / truncation / estimate warnings

🤖 Generated with Claude Code

https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR