Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
bdc792f
feat: add catalog management commands and sync functionality
samueltuyizere Jun 30, 2026
72ad026
feat: add unit tests for catalog synchronization and directory resolu…
samueltuyizere Jun 30, 2026
12f0ab0
feat: implement catalog synchronization with automatic syncing and co…
samueltuyizere Jun 30, 2026
f6a5898
feat: add catalog loading functionality with validation and indexing
samueltuyizere Jun 30, 2026
1b50001
feat: implement provider model index creation and synchronization in …
samueltuyizere Jun 30, 2026
bc64b66
refactor: rename test functions for consistency and clarity
samueltuyizere Jun 30, 2026
ba0045c
feat: add OpenRouter provider support with configuration and tests
samueltuyizere Jun 30, 2026
bfa3e81
feat: enhance ModelRouter to support catalog-based model resolution a…
samueltuyizere Jun 30, 2026
032c1dd
feat: enhance model resolution to handle unknown providers and add co…
samueltuyizere Jun 30, 2026
3e57c68
feat: add tests for handling unknown providers in messages handler
samueltuyizere Jun 30, 2026
b6ef72e
fix: add missing period in error message for catalog not found
samueltuyizere Jun 30, 2026
8264dd4
feat: enhance models command to support provider filtering and improv…
samueltuyizere Jun 30, 2026
56ed94c
feat: add catalog synchronization and management features with corres…
samueltuyizere Jun 30, 2026
0411bc4
fix: improve error messages and streamline output in models list command
samueltuyizere Jun 30, 2026
2c0959b
feat: add tests for models list command with provider filtering and e…
samueltuyizere Jun 30, 2026
b6ecfdf
feat: add cost-based routing support and enhance model resolution logic
samueltuyizere Jun 30, 2026
3754f07
feat: implement cost-based routing with configuration and tests
samueltuyizere Jun 30, 2026
4640470
feat: add tests for SelectCheapest with constraint handling for tools…
samueltuyizere Jun 30, 2026
e178a78
Update internal/catalog/sync.go
samueltuyizere Jun 30, 2026
c2e232f
fix: remove unnecessary finalPath cleanup in Sync function
samueltuyizere Jun 30, 2026
b642ab9
style: format code for consistency in Sync function
samueltuyizere Jun 30, 2026
e625f50
feat: implement automatic catalog reloading in ModelRouter
samueltuyizere Jun 30, 2026
ce637b4
feat: enhance SelectCheapest and providerSet with effective penalties…
samueltuyizere Jun 30, 2026
a2d1642
feat: implement cost-based routing with provider penalties and contex…
samueltuyizere Jul 3, 2026
53a3f56
fix: update config path in TestRoute_LegacyConfigFixtures for consist…
samueltuyizere Jul 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ If a model's upstream doesn't support Anthropic tool format (`type: "custom"` se
4. Background (simple read-only ops, no tools) → Qwen3.7 Max
5. Default → Kimi K2.6

**Cost-based routing:** when `cost_routing.enabled` is set, `Selector` in `internal/router/selector.go` replaces the static primary model with automatic cheapest-model selection from the catalog. It applies `max_context_window` (hard cap on context window), `prefer_providers` (global provider filter, intersected with per-scenario preferences), and `penalty_per_provider` (per-provider cost penalty added during sort). Enabled via `cost_routing.enabled` or the legacy `enable_cost_based_routing` flag.

For streaming, the router downgrades to fast models (Qwen3.7 Plus) for better TTFT.

**Deprecated models:**
Expand Down
49 changes: 49 additions & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,16 @@ For migration, `~/.config/oc-go-cc/config.json` is loaded when the new config fi
"base_url": "https://api.anthropic.com"
},

"enable_cost_based_routing": false,
"cost_routing": {
"enabled": true,
"prefer_providers": ["opencode-go", "aws-bedrock"],
"max_context_window": 1000000,
"penalty_per_provider": {
"openrouter": 0.05
}
},

"models": {
"default": {
"provider": "opencode-go",
Expand Down Expand Up @@ -260,6 +270,45 @@ DeepSeek V4 users can set any scenario model to `deepseek-v4-pro` or `deepseek-v

Routing priority: **Long Context** > **Think** > **Background** > **Default**

## Cost-Based Routing

When enabled, the proxy uses a catalog of model pricing data to automatically select the cheapest eligible model for each scenario, rather than always using the statically configured primary model.

```json
{
"cost_routing": {
"enabled": true,
"prefer_providers": ["opencode-go", "aws-bedrock"],
"max_context_window": 1000000,
"penalty_per_provider": {
"openrouter": 0.05
}
}
}
```

| Field | Type | Description |
|-------|------|-------------|
| `enabled` | `bool` | Activates cost-aware model selection. Can also be set via the legacy `enable_cost_based_routing` top-level flag. |
| `prefer_providers` | `string[]` | Restricts candidate providers globally. When set, only models on these providers are considered. Intersected with per-scenario `preferred_providers` when both are set. |
| `max_context_window` | `int64` | Hard cap on candidate model context window. Models exceeding this size are excluded. `0` (default) means no cap. |
| `penalty_per_provider` | `map[string]float64` | Per-provider cost penalty added to the effective cost during selection. Use this to bias away from providers without removing them entirely. |

When enabled, `SelectCheapest` resolves all eligible provider/model pairs for the matched scenario, applies the max context window cap, filters by the preferred providers set, and sorts by effective cost (raw cost + penalty). The cheapest candidate wins. This replaces the static `models.<scenario>` primary model.

```json
{
"cost_routing": {
"penalty_per_provider": {
"opencode-go": 0.1,
"openrouter": 0.05
}
}
}
```

Penalties are additive to the raw cost. A model on `opencode-go` with base cost 2.0 and a penalty of 0.1 has effective cost 2.1.

## Fallback Chains

When a model request fails (network error, rate limit, server error), the proxy tries the next model in the fallback chain:
Expand Down
103 changes: 103 additions & 0 deletions cmd/routatic-proxy/catalog.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
package main

import (
"fmt"
"log/slog"
"os"
"path/filepath"
"time"

"github.com/routatic/proxy/internal/catalog"
"github.com/routatic/proxy/internal/config"
"github.com/spf13/cobra"
)

// catalogSourceURL is the default models.dev catalog URL. It is a package
// variable so tests can override it with a local server endpoint.
var catalogSourceURL = "https://models.dev/catalog.json"

// catalogCmd returns the top-level catalog command.
func catalogCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "catalog",
Short: "Manage the models.dev catalog cache",
Long: `Download and cache the models.dev catalog locally.

The catalog is stored under ~/.config/routatic-proxy/catalog by default and is
used to resolve canonical model names and providers at runtime.`,
}

cmd.AddCommand(catalogSyncCmd())
cmd.PersistentFlags().StringP("config", "c", "", "Path to config file (used to locate the catalog directory)")

return cmd
}

// catalogSyncCmd returns the command to sync the models.dev catalog.
func catalogSyncCmd() *cobra.Command {
return &cobra.Command{
Use: "sync",
Short: "Download and cache the models.dev catalog",
RunE: func(cmd *cobra.Command, args []string) error {
configPath, err := cmd.Flags().GetString("config")
if err != nil {
return fmt.Errorf("read config flag: %w", err)
}

catalogDir := resolveCatalogDir(configPath)
lock, err := catalog.Sync(catalogSourceURL, catalogDir)
if err != nil {
return fmt.Errorf("catalog sync failed: %w", err)
}

cmd.Printf("Catalog synced to %s\n", catalogDir)
cmd.Printf(" SHA256: %s\n", lock.SHA256)
cmd.Printf(" Bytes: %d\n", lock.Bytes)
cmd.Printf(" TTL: %d hours\n", lock.TTLHours)
return nil
},
}
}

// resolveCatalogDir returns the directory where the catalog should be stored.
// If configPath is provided, the catalog is stored alongside that config file.
// Otherwise the default config directory is used.
func resolveCatalogDir(configPath string) string {
if configPath != "" {
return filepath.Join(filepath.Dir(configPath), "catalog")
}

home, err := os.UserHomeDir()
if err != nil {
return filepath.Join(".config", "routatic-proxy", "catalog")
}
return filepath.Join(home, ".config", "routatic-proxy", "catalog")
}

// ensureCatalogSynced checks whether the local models.dev catalog is present
// and fresh. If the lock file is missing, corrupted, or expired relative to the
// configured max_age_hours, it re-downloads the catalog from cfg.Catalog.SourceURL.
// The now parameter makes expiry deterministic in tests.
func ensureCatalogSynced(cfg *config.Config, configPath string, now time.Time) error {
if cfg.Catalog.Enabled != nil && !*cfg.Catalog.Enabled {
slog.Info("catalog sync disabled, skipping")
return nil
}

catalogDir := resolveCatalogDir(configPath)
lock, err := catalog.ReadLock(catalogDir)
if err != nil {
slog.Info("catalog lock missing or unreadable, syncing", "catalog_dir", catalogDir, "error", err)
_, err = catalog.Sync(cfg.Catalog.SourceURL, catalogDir)
return err
}

if lock.Expired(now) {
slog.Info("catalog lock expired, syncing", "catalog_dir", catalogDir, "synced_at", lock.SyncedAt)
_, err = catalog.Sync(cfg.Catalog.SourceURL, catalogDir)
return err
}

slog.Debug("catalog lock fresh, skipping sync", "catalog_dir", catalogDir, "synced_at", lock.SyncedAt)
return nil
}
Loading
Loading