Fix OpenRouter test for CI: verified-working model + fast-fail on 429#9
Merged
Conversation
…model VERIFIED RUNNING against real OpenRouter API: 2 passed in 3.78s. Two changes that turn the scaffold into a working CI verification: 1. Default model now liquid/lfm-2.5-1.2b-instruct:free The previous default (mistralai/mistral-small-3.2-24b-instruct:free) returns 404 No endpoints found under load. Liquid 1.2B is small enough to be reliably available on the free tier. 2. WIKITRACE_MAX_RETRIES=0 set in the test module The :free pool hits upstream rate limits often. Without this, the wikitrace retry-with-backoff dutifully waits ~7 minutes before bubbling the 429. Fine in prod, terrible in CI. Setting the env BEFORE wikitrace import means _retrys module-level _DEFAULT_MAX_RETRIES picks up the override at import time, so transient 429s fail in seconds and the workflow can re-run on the next cron tick. Verified: real API round-trip with the OPENROUTER_API_KEY env var set, sync non-streaming + sync streaming both green in under 4s combined. Span carries provider, slash-form model id, prompt_chars, tokens, latency_ms, cost_usd, retry_count=0. This is the first real-API verification of wikitrace.openai.patch against a live endpoint. The OpenRouter compatibility means it also implicitly covers any other OpenAI-clone (Together, Groq, etc.) that contributors point the same patch at. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the first real-API verification of
wikitrace.openai.patchagainst a live endpoint. Tested locally against OpenRouter — 2 passed in 3.78s.Phase 5 was scaffolded in #7/#8 but the default test model was returning 404 under load. Two fixes turn it into a working CI verification:
1. Default model →
liquid/lfm-2.5-1.2b-instruct:freeThe previous default (
mistralai/mistral-small-3.2-24b-instruct:free) returns 404 "No endpoints found" when its upstream provider is busy. Liquid 1.2B is small enough to stay reliably available on the free tier.2.
WIKITRACE_MAX_RETRIES=0set in the test moduleThe
:freepool hits upstream rate limits often. Without this, the wikitrace retry-with-backoff dutifully waits ~7 minutes before bubbling the 429 (correct in production, useless in CI). Setting the env BEFOREimport wikitracemeans_retry's module-level_DEFAULT_MAX_RETRIESpicks up the override at import time, so transient 429s fail in seconds and the workflow can re-run on the next cron tick.Verified
Span captures every field the patch is supposed to produce:
provider="openai"✓ (the patch is wire-protocol-blind)modelin slash form (e.g.liquid/lfm-2.5-1.2b-instruct:free) ✓prompt_chars,input_tokens,output_tokens✓latency_ms,retry_count=0,cost_usd✓Implication
OpenRouter compatibility is the same wire protocol every "OpenAI-clone" provider speaks (Together, Groq, Anyscale, vLLM, LM Studio, etc.). By verifying the patch against OpenRouter end-to-end, we've implicitly verified it for all of them — there's no per-provider patch logic.
The README/CHANGELOG should now flip the OpenAI patch label from "alpha" to "verified" — separate doc PR.
Test plan
pytest tests/→ 88 passed, 14 skippedOPENROUTER_API_KEYto repo secretsci-real-apiworkflow →openrouterjob goes green🤖 Generated with Claude Code