Skip to content

Feat: Add models.dev pricing metadata pipeline#863

Merged
steipete merged 3 commits intosteipete:mainfrom
iam-brain:iam-brain/models-dev-pricing-pipeline
May 10, 2026
Merged

Feat: Add models.dev pricing metadata pipeline#863
steipete merged 3 commits intosteipete:mainfrom
iam-brain:iam-brain/models-dev-pricing-pipeline

Conversation

@iam-brain
Copy link
Copy Markdown
Contributor

@iam-brain iam-brain commented May 8, 2026

Model pricing metadata

CodexBar has an additive models.dev pricing pipeline for future cost lookup work. Existing hardcoded pricing remains unchanged for now.

Source and cache

  • Source API: https://models.dev/api.json
  • No API key is required.
  • Local cache: ~/Library/Caches/CodexBar/model-pricing/models-dev-v1.json
  • TTL: 24 hours

The pipeline lets future scanner code read the last valid cache synchronously with ModelsDevPricingPipeline.lookup and refresh stale metadata separately with ModelsDevPricingPipeline.refreshIfNeeded. If a refresh fails, the last valid cache remains usable.

Lookup rules

Pricing is scoped by provider id and model id. This prevents two providers with the same model id or display name from sharing pricing accidentally.

Planned local source mapping:

  • Codex/OpenAI logs: models.dev provider id openai
  • Claude logs: models.dev provider id anthropic
  • Vertex AI Claude logs: models.dev provider id google-vertex-anthropic

The first integration PR only adds the parser, client, cache, provider-scoped lookup, and tests. It does not route live cost calculations through models.dev yet.

Units

models.dev publishes costs as USD per 1M tokens. CodexBar converts those to USD per token in the metadata layer:

perToken = modelsDevCost / 1_000_000

When models.dev includes cost.context_over_200k, CodexBar parses those values as the above-200k-token pricing lane and converts them with the same per-1M-token rule.

Summary

  • add an additive models.dev pricing metadata parser/client/cache pipeline
  • support provider-scoped model lookup, cache TTL handling, and per-1M-token to per-token conversion
  • document the cache path and future lookup rules
  • fix Linux CLI builds by importing FoundationNetworking where Swift exposes URLSession networking types separately

Reasoning

  • reduce reliance on user-curated and logged model pricing, which is brittle as providers add aliases, change prices, and introduce threshold pricing
  • transition toward Anomaly's models.dev online catalog because it already does the cataloging work, tracks provider-scoped model metadata, and exposes a free API without requiring an API key
  • keep this additive first so CodexBar can validate the catalog/cache pipeline before switching existing cost calculations away from the fallback pricing tables

Scope

  • does not switch existing Codex/Claude cost calculations to models.dev
  • does not migrate or change existing fallback pricing tables
  • no API key required for models.dev

References

  • models.dev catalog API: https://models.dev/api.json

Validation

  • swift test --filter ModelsDevPricingTests
  • pnpm check
  • ./Scripts/compile_and_run.sh
  • Latest PR CI after rebase: Linux CLI x64 and arm64 checks pass.
  • Latest PR CI after rebase: macOS lint-build-test currently fails in existing Codex RPC fallback/baseline tests with timeout(method: "initialize"); the models.dev pricing suite passes locally.
  • swift test was also run earlier; it still fails on the pre-existing MistralUsageParserTests.parses dates from response expectation (expected month 11, got 10).

@iam-brain iam-brain marked this pull request as ready for review May 8, 2026 16:21
@iam-brain iam-brain marked this pull request as draft May 8, 2026 16:21
@iam-brain iam-brain force-pushed the iam-brain/models-dev-pricing-pipeline branch from 8020d71 to 28756e3 Compare May 8, 2026 16:27
@iam-brain iam-brain marked this pull request as ready for review May 8, 2026 16:44
@iam-brain
Copy link
Copy Markdown
Contributor Author

Current CI build failure seems to be caused by increased pressure from the new tests introduced for the feature.

@iam-brain iam-brain changed the title Add models.dev pricing metadata pipeline Feat: Add models.dev pricing metadata pipeline May 8, 2026
@steipete steipete force-pushed the iam-brain/models-dev-pricing-pipeline branch from 28756e3 to df87d1c Compare May 10, 2026 02:21
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df87d1cd99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

func containsProviderModels(from cachedCatalog: ModelsDevCatalog) -> Bool {
cachedCatalog.providers.allSatisfy { providerID, cachedProvider in
guard let provider = self.providers[ModelsDevProvider.normalizeProviderID(providerID)] else { return false }
return cachedProvider.models.keys.allSatisfy { provider.models.keys.contains($0) }
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate model continuity using model IDs, not map keys

refreshIfNeeded treats a fetched catalog as invalid when any cached model key is missing, but lookup itself supports key/ID divergence by matching ModelsDevModel.id in a second pass. Because containsProviderModels compares only cachedProvider.models.keys to provider.models.keys, a legitimate upstream key rename (same id, new map key) will be misclassified as a dropped model and block cache updates indefinitely, leaving pricing stale even though the model still exists.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

@steipete steipete left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed model pricing pipeline changes and local validation; CI is green. Thanks @iam-brain!

@steipete steipete merged commit 001561a into steipete:main May 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants