Add OpenRouter real-API integration test (free-tier path)#8
Merged
Conversation
OpenRouter is OpenAI-protocol-compatible and offers free-tier models, so pointing the openai SDK at openrouter.ai/api/v1 verifies wikitrace.openai.patch over a real network without burning credits. Why on top of the OpenAI test: - Skip-default contributors usually have an OpenRouter key (free) before they have an OpenAI key (paid). Free-tier models like mistralai/mistral-small-3.2-24b-instruct:free pay nothing. - The OpenRouter <provider>/<model> id form exercises the recent pricing prefix-stripping fix end-to-end, which the bare OpenAI test does not. tests/integration/test_openrouter_real.py - Sync non-streaming + sync streaming - Asserts provider, model (slash form), prompt_chars, tokens, latency, retry_count, and that cost_usd is set (None or 0.0 is acceptable on :free models) - conftest gains an openrouter_key fixture; default model overrideable via WIKITRACE_OPENROUTER_TEST_MODEL env ci-real-api.yml - Third job openrouter, mirrors the openai/anthropic shape: reads OPENROUTER_API_KEY from repo secrets, exits 0 cleanly when absent - Gateable via vars.WIKITRACE_OPENROUTER_TESTS_ENABLED README - Tests section now lists OpenRouter first as the free-tier path Verified locally: pytest -q tests/ → 88 passed, 14 skipped (the 2 new OpenRouter tests skip cleanly without a key). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OpenRouter is OpenAI-protocol-compatible and offers free-tier models. Pointing the
openaiSDK atopenrouter.ai/api/v1verifieswikitrace.openai.patchover a real network — without burning OpenAI credits. Adds the most contributor-friendly real-API test path.Why on top of the existing OpenAI test
<provider>/<model>form. That exercises the recentwikitrace.pricingprefix-stripping fix end-to-end, which the bare OpenAI test does not.What
tests/integration/test_openrouter_real.py— sync non-streaming + sync streaming, asserts span shape (provider, slash-form model, tokens, latency, retry, cost_usd present)tests/integration/conftest.py—openrouter_keyfixture.github/workflows/ci-real-api.yml— third jobopenrouter, gated onOPENROUTER_API_KEYsecret +vars.WIKITRACE_OPENROUTER_TESTS_ENABLEDREADME.md— Tests section now lists OpenRouter first as the free-tier pathVerified locally
pytest -q tests/→ 88 passed, 14 skipped (the 2 new OpenRouter tests skip cleanly withoutOPENROUTER_API_KEY).To activate
Then trigger via Actions tab → "ci-real-api" → "Run workflow", or wait for the weekly cron.
Test plan
OPENROUTER_API_KEYrepo secret (free signup at openrouter.ai)ci-real-api; confirmopenrouterjob goes green🤖 Generated with Claude Code