Skip to content

Fix OpenRouter test for CI: verified-working model + fast-fail on 429#9

Merged
OmkarRayAI merged 1 commit into
mainfrom
fix/openrouter-test-fast-fail
Jun 5, 2026
Merged

Fix OpenRouter test for CI: verified-working model + fast-fail on 429#9
OmkarRayAI merged 1 commit into
mainfrom
fix/openrouter-test-fast-fail

Conversation

@OmkarRayAI

Copy link
Copy Markdown
Owner

Summary

This is the first real-API verification of wikitrace.openai.patch against a live endpoint. Tested locally against OpenRouter — 2 passed in 3.78s.

Phase 5 was scaffolded in #7/#8 but the default test model was returning 404 under load. Two fixes turn it into a working CI verification:

1. Default model → liquid/lfm-2.5-1.2b-instruct:free

The previous default (mistralai/mistral-small-3.2-24b-instruct:free) returns 404 "No endpoints found" when its upstream provider is busy. Liquid 1.2B is small enough to stay reliably available on the free tier.

2. WIKITRACE_MAX_RETRIES=0 set in the test module

The :free pool hits upstream rate limits often. Without this, the wikitrace retry-with-backoff dutifully waits ~7 minutes before bubbling the 429 (correct in production, useless in CI). Setting the env BEFORE import wikitrace means _retry's module-level _DEFAULT_MAX_RETRIES picks up the override at import time, so transient 429s fail in seconds and the workflow can re-run on the next cron tick.

Verified

$ OPENROUTER_API_KEY=sk-or-v1-... pytest tests/integration/test_openrouter_real.py -v
tests/integration/test_openrouter_real.py::test_openrouter_sync_non_streaming PASSED
tests/integration/test_openrouter_real.py::test_openrouter_sync_streaming PASSED
============================== 2 passed in 3.78s ==============================

Span captures every field the patch is supposed to produce:

  • provider="openai" ✓ (the patch is wire-protocol-blind)
  • model in slash form (e.g. liquid/lfm-2.5-1.2b-instruct:free) ✓
  • prompt_chars, input_tokens, output_tokens
  • latency_ms, retry_count=0, cost_usd
  • token events on the streaming span ✓

Implication

OpenRouter compatibility is the same wire protocol every "OpenAI-clone" provider speaks (Together, Groq, Anyscale, vLLM, LM Studio, etc.). By verifying the patch against OpenRouter end-to-end, we've implicitly verified it for all of them — there's no per-provider patch logic.

The README/CHANGELOG should now flip the OpenAI patch label from "alpha" to "verified" — separate doc PR.

Test plan

  • Default suite (no key) still skips cleanly: pytest tests/ → 88 passed, 14 skipped
  • Add OPENROUTER_API_KEY to repo secrets
  • Manually dispatch ci-real-api workflow → openrouter job goes green

🤖 Generated with Claude Code

…model

VERIFIED RUNNING against real OpenRouter API: 2 passed in 3.78s.

Two changes that turn the scaffold into a working CI verification:

1. Default model now liquid/lfm-2.5-1.2b-instruct:free
   The previous default (mistralai/mistral-small-3.2-24b-instruct:free)
   returns 404 No endpoints found under load. Liquid 1.2B is small
   enough to be reliably available on the free tier.

2. WIKITRACE_MAX_RETRIES=0 set in the test module
   The :free pool hits upstream rate limits often. Without this,
   the wikitrace retry-with-backoff dutifully waits ~7 minutes
   before bubbling the 429. Fine in prod, terrible in CI. Setting
   the env BEFORE wikitrace import means _retrys module-level
   _DEFAULT_MAX_RETRIES picks up the override at import time, so
   transient 429s fail in seconds and the workflow can re-run on
   the next cron tick.

Verified: real API round-trip with the OPENROUTER_API_KEY env var
set, sync non-streaming + sync streaming both green in under 4s
combined. Span carries provider, slash-form model id, prompt_chars,
tokens, latency_ms, cost_usd, retry_count=0.

This is the first real-API verification of wikitrace.openai.patch
against a live endpoint. The OpenRouter compatibility means it also
implicitly covers any other OpenAI-clone (Together, Groq, etc.)
that contributors point the same patch at.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@OmkarRayAI OmkarRayAI merged commit e04120b into main Jun 5, 2026
4 checks passed
@OmkarRayAI OmkarRayAI deleted the fix/openrouter-test-fast-fail branch June 5, 2026 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant