From c0cec255f4d0da601a942cade9f14a6185646c59 Mon Sep 17 00:00:00 2001 From: Arun Kumar Date: Wed, 15 Jul 2026 20:00:22 +0530 Subject: [PATCH] Refactor configuration around a single adapter interface (v0.2.0) Replaces the four hand-written caller lambdas (llm_caller, messages_caller, embedding_caller, classifier_caller) with one config.adapter integration point, restructures configuration into validated groups, and shrinks the Rails initializer template to a minimal working config. - Add adapter layer: built-in :ruby_llm, :open_ai, :anthropic adapters (lazily required with actionable errors), Adapters::Custom for any provider, and an internal Legacy adapter so 0.1.x caller lambdas keep working unchanged - Add Invoker: user lambdas only receive the keyword arguments their signature accepts, fixing the 0.1.6 regression where the documented ->(prompt, model:) signature crashed with unknown keyword: :tools - Group configuration (routing/cache/compression/history/conversation) with flat 0.1.x keys kept as delegating aliases, including per-call overrides - Add fail-fast validation: configure validates values, optimize validates cross-field consistency, LlmOptimizer.validate! checks everything at boot (replaces silent warn-and-disable) - Route the classifier through the adapter via routing.classifier_model - Wire the schema option through to adapters (declared but unused since 0.1.7) - Fix history summarization injecting [content, token_info] as message content; stop connecting to localhost Redis when redis_url is unset - Rewrite initializer template (~25-line working config + commented appendix) and README; bump version to 0.2.0 Co-Authored-By: Claude Fable 5 --- .rubocop.yml | 10 +- CHANGELOG.md | 32 +- Gemfile.lock | 4 +- README.md | 314 +++++++++--------- .../llm_optimizer/templates/initializer.rb | 185 ++++------- lib/llm_optimizer.rb | 18 +- lib/llm_optimizer/adapters.rb | 31 ++ lib/llm_optimizer/adapters/anthropic.rb | 57 ++++ lib/llm_optimizer/adapters/base.rb | 50 +++ lib/llm_optimizer/adapters/custom.rb | 36 ++ lib/llm_optimizer/adapters/legacy.rb | 40 +++ lib/llm_optimizer/adapters/open_ai.rb | 61 ++++ lib/llm_optimizer/adapters/ruby_llm.rb | 62 ++++ lib/llm_optimizer/configuration.rb | 284 ++++++++++++---- lib/llm_optimizer/invoker.rb | 27 ++ lib/llm_optimizer/model_router.rb | 22 +- lib/llm_optimizer/pipeline.rb | 31 +- lib/llm_optimizer/version.rb | 2 +- test/unit/test_adapters.rb | 128 +++++++ test/unit/test_configuration.rb | 112 ++++++- test/unit/test_model_router.rb | 29 ++ 21 files changed, 1173 insertions(+), 362 deletions(-) create mode 100644 lib/llm_optimizer/adapters.rb create mode 100644 lib/llm_optimizer/adapters/anthropic.rb create mode 100644 lib/llm_optimizer/adapters/base.rb create mode 100644 lib/llm_optimizer/adapters/custom.rb create mode 100644 lib/llm_optimizer/adapters/legacy.rb create mode 100644 lib/llm_optimizer/adapters/open_ai.rb create mode 100644 lib/llm_optimizer/adapters/ruby_llm.rb create mode 100644 lib/llm_optimizer/invoker.rb create mode 100644 test/unit/test_adapters.rb diff --git a/.rubocop.yml b/.rubocop.yml index c29f102..ded5e1b 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -21,10 +21,18 @@ Metrics/BlockLength: - "*.gemspec" - "test/**/*.rb" -# Test classes are naturally long +# Test classes are naturally long; Configuration is mostly declarative +# (grouped settings + flat compatibility aliases) Metrics/ClassLength: Exclude: - "test/**/*.rb" + - "lib/llm_optimizer/configuration.rb" + +# validate!/validate_values! raise on failure and return true (like AR's save!) +Naming/PredicateMethod: + AllowedMethods: + - validate! + - validate_values! # The optimize method is intentionally a pipeline — complexity is expected Metrics/MethodLength: diff --git a/CHANGELOG.md b/CHANGELOG.md index 8dbabe9..295edf7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.2.0] - 2026-07-15 + +Ease-of-use refactor: one adapter interface replaces the four caller lambdas, configuration is grouped and validated, and the Rails initializer shrinks to a minimal working config. 0.1.x initializers keep working via compatibility aliases. + +### Added +- **Adapter interface** (`config.adapter`) — the single integration point between the gem and your LLM provider. Built-in adapters: `:ruby_llm` (chat + embeddings across OpenAI/Anthropic/Gemini/Bedrock/Ollama), `:open_ai` (ruby-openai, chat + embeddings), `:anthropic` (official anthropic gem, chat only). Adapter gems are lazily required with an actionable error if missing +- `LlmOptimizer::Adapters::Custom` — wrap any provider with `chat:` / `embed:` lambdas +- **Grouped configuration** — `config.routing.*`, `config.cache.*`, `config.compression.*`, `config.history.*`, `config.conversation.*`; flat 0.1.x keys remain as aliases +- `config.routing.classifier_model` — enables the LLM routing classifier through the adapter (replaces hand-written `classifier_caller` lambdas) +- **Fail-fast validation** — `LlmOptimizer.configure` validates values immediately; each `optimize` call validates cross-field consistency; `LlmOptimizer.validate!` runs the full check at boot with actionable error messages +- **Defensive lambda invocation** (`LlmOptimizer::Invoker`) — user-supplied lambdas only receive the keyword arguments their signature accepts, so lambdas written against older signatures keep working +- `schema` config option is now actually passed through to adapters (it was declared but unused in 0.1.7) + +### Changed +- **Rails initializer template rewritten** — ~25-line minimal working config with advanced options in a commented appendix, instead of 120 lines of mixed provider examples +- Default complex model updated from `claude-3-5-sonnet-20241022` to `claude-sonnet-4-5` +- `embedding_model` default is now `nil` (the adapter picks its own default) and lives at `cache.embedding_model` +- `llm_caller`, `messages_caller`, `embedding_caller`, `classifier_caller` are deprecated; when set they are wrapped in an internal legacy adapter automatically +- Enabling the semantic cache without `redis_url` or embedding support now raises `ConfigurationError` instead of silently disabling the cache (`validate_configuration!` removed) +- Semantic cache lookup no longer connects to a default localhost Redis when `redis_url` is unset — it logs a warning and skips the cache + +### Fixed +- `llm_caller` lambdas written as `->(prompt, model:)` (the documented signature) crashed with `unknown keyword: :tools` since 0.1.6 — the gem passed `tools:` unconditionally; lambdas now only receive keywords they accept +- History summarization injected a `[content, token_info]` array as the summary message content instead of the summary string +- `tools` and `with_tools` are now true aliases of one value instead of two separate config keys +- Removed dead code: unused `tools_caller` config key and the never-assigned `@_current_llm_caller` + ## [0.1.7] - 2026-05-05 ### Added @@ -127,7 +154,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `OptimizeResult` struct with `response`, `model`, `model_tier`, `cache_status`, `original_tokens`, `compressed_tokens`, `latency_ms`, `messages` - Unit test suite covering all components with positive and negative scenarios using Minitest + Mocha -[Unreleased]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.5...HEAD +[Unreleased]: https://github.com/arunkumarry/llm_optimizer/compare/v0.2.0...HEAD +[0.2.0]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.7...v0.2.0 +[0.1.7]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.6...v0.1.7 +[0.1.6]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.5...v0.1.6 [0.1.5]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.4...v0.1.5 [0.1.4]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.3...v0.1.4 [0.1.3]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.2...v0.1.3 diff --git a/Gemfile.lock b/Gemfile.lock index f1cd3ec..d214789 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ PATH remote: . specs: - llm_optimizer (0.1.6) + llm_optimizer (0.2.0) logger (~> 1.6) msgpack (~> 1.7) redis (~> 5.0) @@ -107,7 +107,7 @@ CHECKSUMS json (2.19.3) sha256=289b0bb53052a1fa8c34ab33cc750b659ba14a5c45f3fcf4b18762dc67c78646 language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 - llm_optimizer (0.1.6) + llm_optimizer (0.2.0) logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 minitest (5.27.0) sha256=2d3b17f8a36fe7801c1adcffdbc38233b938eb0b4966e97a6739055a45fa77d5 mocha (2.8.2) sha256=1f77e729db47e72b4ef776461ce20caeec2572ffdf23365b0a03608fee8f4eee diff --git a/README.md b/README.md index 6018c41..e6968cc 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # llm_optimizer -A Smart Gateway for LLM API calls in Ruby and Rails applications. Reduces token usage and API costs through four composable optimizations all opt-in, all independently configurable. +A Smart Gateway for LLM API calls in Ruby and Rails applications. Reduces token usage and API costs through four composable optimizations — all opt-in, all independently configurable. ## How it works @@ -10,60 +10,130 @@ Every call to `LlmOptimizer.optimize` passes through an ordered pipeline: prompt → Compressor → ModelRouter → SemanticCache lookup → HistoryManager → LLM call → SemanticCache store → OptimizeResult ``` -Each stage is independently enabled via configuration flags. If any stage fails, the gem falls through to a raw LLM call your app never breaks because of the optimizer. +Each stage is independently enabled via configuration flags. If any stage fails at runtime, the gem falls through to a raw LLM call — your app never breaks because of the optimizer. (Misconfiguration, on the other hand, fails fast with an actionable error.) + +## Quick Start + +```ruby +LlmOptimizer.configure do |config| + config.adapter = :ruby_llm # how the gem talks to your LLM provider + + config.routing.simple_model = "gpt-4o-mini" + config.routing.complex_model = "claude-sonnet-4-5" + + config.cache.enabled = true # semantic caching + config.redis_url = ENV["REDIS_URL"] +end + +result = LlmOptimizer.optimize("What is Redis?") + +result.response # => "Redis is an in-memory data store..." +result.cache_status # => :hit or :miss +result.model_tier # => :simple or :complex +result.model # => "gpt-4o-mini" +result.original_tokens # => 5 +result.latency_ms # => 12.4 +``` + +That's the whole setup: pick an adapter, name two models, flip the flags you want. + +## Adapters + +An adapter is the single integration point between the gem and your LLM provider. Built-ins: + +| Adapter | Gem | Chat | Embeddings | +|---|---|---|---| +| `:ruby_llm` | [ruby_llm](https://rubygems.org/gems/ruby_llm) | OpenAI, Anthropic, Gemini, Bedrock, Ollama, … | ✅ | +| `:open_ai` | [ruby-openai](https://rubygems.org/gems/ruby-openai) | OpenAI | ✅ | +| `:anthropic` | [anthropic](https://rubygems.org/gems/anthropic) | Anthropic | ❌ (pair with `embedding_caller`) | + +The adapter gems are not dependencies of llm_optimizer — add the one you use to your own Gemfile. Symbols use environment-variable API keys; instantiate the class for options: + +```ruby +config.adapter = LlmOptimizer::Adapters::OpenAI.new( + access_token: Rails.application.credentials.openai_api_key, + embedding_model: "text-embedding-3-large" +) +``` + +Any other provider — or your existing service objects — via `Custom`: + +```ruby +config.adapter = LlmOptimizer::Adapters::Custom.new( + chat: ->(messages, model:, tools: nil) { + # messages is an Array of { role:, content: } hashes. + # Return a String, or { content:, input_tokens:, output_tokens: }. + MyLlmService.chat(messages, model: model) + }, + embed: ->(text) { MyEmbeddingService.embed(text) } # optional; needed for the cache +) +``` + +Lambdas are invoked defensively: the gem only passes the keyword arguments your lambda's signature accepts, so a plain `->(messages, model:)` keeps working as new keywords are added. ## Optimizations ### 1. Semantic Caching -Stores prompt embeddings in Redis. On subsequent calls, computes cosine similarity against stored embeddings. If similarity ≥ threshold, returns the cached response instantly no LLM call made. + +Stores prompt embeddings in Redis. On subsequent calls, computes cosine similarity against stored embeddings. If similarity ≥ threshold, returns the cached response instantly — no LLM call made. + +```ruby +config.cache.enabled = true +config.cache.threshold = 0.96 # cosine similarity cutoff +config.cache.ttl = 86_400 # seconds +config.cache.scope = nil # optional namespace (per user, tenant, ...) +config.redis_url = ENV["REDIS_URL"] +``` ### 2. Intelligent Model Routing Classifies each prompt and routes it to the appropriate model tier: -- **Simple** → cheaper/faster model (e.g. `llama3`, `gemini-2.5-flash-lite`) -- **Complex** → premium model (e.g. `claude-haiku-4-5-20251001`, `gemini-3.0-pro`) +- **Simple** → cheaper/faster model +- **Complex** → premium model Routing uses a three-layer decision chain: -1. **Explicit override** — if `route_to: :simple` or `:complex` is set, always use that -2. **Fast-path signals** — code blocks (` ``` `, `~~~`) and keywords (`analyze`, `refactor`, `debug`, `architect`, `explain in detail`) → instantly `:complex`, no LLM call -3. **LLM classifier** (optional) — for ambiguous prompts, calls a cheap model with a classification prompt; falls back to word-count heuristic if not configured or if the call fails +1. **Explicit override** — `config.routing.mode = :simple` / `:complex` (or per call: `route_to: :simple`) +2. **Fast-path signals** — code blocks and keywords (`analyze`, `refactor`, `debug`, `architect`, `explain in detail`) → instantly `:complex`, no LLM call +3. **LLM classifier** (optional) — for ambiguous prompts, asks a cheap model; falls back to a word-count heuristic if not configured or if the call fails + +```ruby +config.routing.classifier_model = "gpt-4o-mini" # enable the LLM classifier +``` This hybrid approach fixes the core weakness of pure heuristics: + - `"Fix this bug"` → 3 words but `:complex` via classifier - `"Explain Ruby blocks simply"` → long but `:simple` via classifier - `"analyze this code"` → keyword fast-path → `:complex` instantly (no classifier call) -Configure the classifier with any cheap model your app already uses: +If no classifier model is set, the router falls back to the word-count heuristic (< 20 words → `:simple`). + +### 3. Token Pruning + +Removes common English stop words from prompts before sending to the LLM. Preserves fenced code block content unchanged. ```ruby -config.classifier_caller = ->(prompt) { - RubyLLM.chat(model: "amazon.nova-micro-v1:0", provider: :bedrock, assume_model_exists: true) - .ask(prompt).content.strip.downcase -} +config.compression.enabled = true ``` -If `classifier_caller` is not set, the router falls back to the word-count heuristic (< 20 words → `:simple`). +### 4. Conversation History Sliding Window -### 3. Token Pruning -Removes common English stop words from prompts before sending to the LLM. Preserves fenced code block content unchanged. Typically reduces token count by 10–20%. +When a conversation history exceeds the configured token budget, summarizes the oldest messages using the simple model and replaces them with a single system summary message. -### 4. Conversation History Sliding Window -When a conversation history exceeds the configured token budget, summarizes the oldest messages using the simple model and replaces them with a single system summary message. Uses Redis to store for fast reetreival and summarizing. +```ruby +config.history.enabled = true +config.history.token_budget = 4000 +``` ## Installation -Add to your Gemfile: +Add to your Gemfile (plus the adapter gem you use): ```ruby gem "llm_optimizer" -``` - -Then run: - -```bash -bundle install +gem "ruby_llm" # or ruby-openai / anthropic ``` For Rails apps, generate the initializer: @@ -72,150 +142,71 @@ For Rails apps, generate the initializer: rails generate llm_optimizer:install ``` -This creates `config/initializers/llm_optimizer.rb` with all options pre-filled and commented. +This creates `config/initializers/llm_optimizer.rb` — a minimal working config with all advanced options documented in a commented appendix. -## Quick Start +## Configuration reference -```ruby -LlmOptimizer.configure do |config| - config.compress_prompt = true - config.use_semantic_cache = true - config.redis_url = ENV["REDIS_URL"] - - # Wire up your app's LLM client - config.llm_caller = ->(prompt, model:) { - # Use whatever LLM client your app already has - MyLlmService.chat(prompt, model: model) - } - - # Wire up your embeddings provider (required if use_semantic_cache: true) - config.embedding_caller = ->(text) { - MyEmbeddingService.embed(text) - } -end +| Key | Type | Default | Description | +|---|---|---|---| +| `adapter` | Symbol/Object | `nil` | `:ruby_llm`, `:open_ai`, `:anthropic`, or an adapter instance | +| `redis_url` | String | `nil` | Redis URL (required for cache and conversations) | +| `routing.mode` | Symbol | `:auto` | `:auto`, `:simple`, or `:complex` | +| `routing.simple_model` | String | `"gpt-4o-mini"` | Model for simple prompts | +| `routing.complex_model` | String | `"claude-sonnet-4-5"` | Model for complex prompts | +| `routing.classifier_model` | String | `nil` | When set, ambiguous prompts are LLM-classified | +| `cache.enabled` | Boolean | `false` | Redis-backed semantic cache | +| `cache.threshold` | Float | `0.96` | Minimum cosine similarity for a hit | +| `cache.ttl` | Integer | `86_400` | Cache entry TTL in seconds | +| `cache.scope` | String | `nil` | Namespace cache entries (per user, tenant, …) | +| `cache.embedding_model` | String | `nil` | Embedding model override (adapter default otherwise) | +| `compression.enabled` | Boolean | `false` | Strip stop words before sending to the LLM | +| `history.enabled` | Boolean | `false` | Summarize old messages over the token budget | +| `history.token_budget` | Integer | `4000` | Token limit before summarization | +| `conversation.system_prompt` | String | `nil` | Seeded when a new `conversation_id` is created | +| `conversation.ttl` | Integer | `86_400` | Conversation TTL in seconds (`0` = no expiry) | +| `tools` | Array | `nil` | Tool definitions, passed through to the adapter | +| `schema` | Hash | `nil` | JSON schema for structured output (adapter support varies) | +| `logger` | Logger | `Logger.new($stdout)` | Any Logger-compatible object | +| `debug_logging` | Boolean | `false` | Log full prompt and response at DEBUG level | +| `timeout_seconds` | Integer | `5` | Timeout for external API calls | -result = LlmOptimizer.optimize("What is Redis?") +Flat 0.1.x keys (`use_semantic_cache`, `compress_prompt`, `manage_history`, `route_to`, `similarity_threshold`, `token_budget`, `cache_ttl`, `cache_scope`, `simple_model`, `complex_model`, `system_prompt`, `conversation_ttl`, `with_tools`) still work as aliases of the grouped keys, as do the 0.1.x caller lambdas (`llm_caller`, `messages_caller`, `embedding_caller`, `classifier_caller`) — they are wrapped in a legacy adapter automatically. Both are deprecated in favor of the grouped keys and `config.adapter`. + +### Fail-fast validation -puts result.response # => "Redis is an in-memory data store..." -puts result.cache_status # => :hit or :miss -puts result.model_tier # => :simple or :complex -puts result.model # => "gemini-2.5-flash-lite" -puts result.original_tokens # => 5 -puts result.compressed_tokens # => 4 -puts result.latency_ms # => 12.4 +`LlmOptimizer.configure` validates values immediately (bad routing mode, out-of-range threshold). Add one line at the end of your initializer to also catch cross-field problems (no adapter, cache without Redis, cache without embedding support) at boot instead of at the first LLM call: + +```ruby +LlmOptimizer.validate! ``` -## Configuration +## Per-call configuration -### Rails initializer +Override global config for a single call using options or a block: ```ruby -# config/initializers/llm_optimizer.rb -require "llm_optimizer" +LlmOptimizer.optimize(prompt, route_to: :simple) -LlmOptimizer.configure do |config| - # --- Feature flags (all off by default) --- - config.compress_prompt = true # strip stop words before sending to LLM - config.use_semantic_cache = true # cache responses by vector similarity - config.manage_history = true # summarize old messages when over token budget - - # --- Model routing --- - config.route_to = :auto # :auto, :simple, or :complex - config.simple_model = "gemini-2.5-flash-lite" # used for simple prompts - config.complex_model = "claude-haiku-4-5-20251001" # used for complex prompts - - # --- Redis (required if use_semantic_cache: true) --- - config.redis_url = ENV["REDIS_URL"] - - # --- Token / cache settings --- - config.similarity_threshold = 0.96 # cosine similarity cutoff for cache hit - config.token_budget = 4000 # max tokens before history summarization - config.cache_ttl = 86400 # cache TTL in seconds (24h) - config.timeout_seconds = 5 # timeout for external API calls - - # --- Logging --- - config.logger = Rails.logger - config.debug_logging = Rails.env.development? # logs full prompt+response in dev - - # --- Wire up your app's LLM client --- - # Replace the body with however your app calls the LLM - config.llm_caller = ->(prompt, model:) { - model ||= "claude-haiku-4-5-20251001" - provider = if model.include?("claude") then :anthropic - elsif model.include?("gpt") then :openai - elsif model.include?("gemini") then :gemini - else :ollama - end - chat = RubyLLM.chat(model: model, provider: provider, assume_model_exists: true) - chat.ask(prompt).content - } - - # Embeddings caller — wire to your embeddings provider (required if use_semantic_cache: true) - config.embedding_caller = ->(text) { - response = RubyLLM.embed(text, provider: :gemini, model: 'gemini-embedding-001') - response.vectors - } - - # Classifier caller — optional, improves routing accuracy for ambiguous prompts - # Falls back to word-count heuristic if not set or if the call fails - config.classifier_caller = ->(prompt) { - RubyLLM.chat(model: "amazon.nova-micro-v1:0", provider: :bedrock, assume_model_exists: true) - .ask(prompt).content.strip.downcase - } - - # Messages caller - optional, handles converation summary and hostiry manager. - config.system_prompt = "You are a sarcastic comic person who gives witty responses in a non harmful way. If any serious question is asked, handle it in a calm way." - - config.messages_caller = ->(messages, model:) { - chat = RubyLLM.chat(model: model) - messages[0..-2].each { |m| chat.add_message(role: m[:role], content: m[:content]) } - response = chat.ask(messages.last[:content]) - response.content - } +LlmOptimizer.optimize(prompt) do |config| + config.routing.mode = :simple + config.compression.enabled = false end - ``` -### Configuration reference +## Conversations -| Key | Type | Default | Description | -|---|---|---|---| -| `compress_prompt` | Boolean | `false` | Strip stop words before sending to LLM | -| `use_semantic_cache` | Boolean | `false` | Enable Redis-backed semantic cache | -| `manage_history` | Boolean | `false` | Enable conversation history summarization | -| `route_to` | Symbol | `:auto` | `:auto`, `:simple`, or `:complex` | -| `simple_model` | String | `"gemini-2.5-flash-lite"` | Model for simple prompts | -| `complex_model` | String | `"claude-haiku-4-5-20251001"` | Model for complex prompts | -| `similarity_threshold` | Float | `0.96` | Minimum cosine similarity for cache hit | -| `token_budget` | Integer | `4000` | Token limit before history summarization | -| `cache_ttl` | Integer | `86400` | Cache entry TTL in seconds | -| `timeout_seconds` | Integer | `5` | Timeout for external API calls | -| `redis_url` | String | `nil` | Redis connection URL | -| `embedding_model` | String | `"gemini-embedding-001"` | Embedding model name (OpenAI fallback) | -| `logger` | Logger | `Logger.new($stdout)` | Any Logger-compatible object | -| `debug_logging` | Boolean | `false` | Log full prompt and response at DEBUG level | -| `llm_caller` | Lambda | `nil` | `(prompt, model:) -> String` | -| `embedding_caller` | Lambda | `nil` | `(text) -> Array` | -| `classifier_caller` | Lambda | `nil` | `(prompt) -> "simple" or "complex"` | -| `messages_caller` | Lambda | `nil` | `(messages, model:) -> String` — used when `conversation_id` is present; receives full history including current user turn | -| `system_prompt` | String | `nil` | Seeded as the first system message when a new conversation is created via `conversation_id` | -| `conversation_ttl` | Integer | `86400` | TTL in seconds for Redis-backed conversation history (`0` for no expiry) | -| `with_tools` | Array | `nil` | Tools (functions) available to the LLM; passed as `tools:` keyword to callers | - -## Per-call configuration - -Override global config for a single call using a block: +Pass a stable `conversation_id` and the gem loads history from Redis, calls the LLM with full context, and saves the updated history back: ```ruby -result = LlmOptimizer.optimize(prompt) do |config| - config.route_to = :simple - config.compress_prompt = false -end +LlmOptimizer.optimize("What did I just ask?", conversation_id: "user-42") +LlmOptimizer.clear_conversation("user-42") ``` +Or manage messages yourself and pass `messages:` — the result's `messages` field returns the final array (after any summarization) for the next turn. + ## OptimizeResult -Every call returns an `OptimizeResult` struct: +Every call returns an `OptimizeResult`: | Field | Type | Description | |---|---|---| @@ -224,11 +215,10 @@ Every call returns an `OptimizeResult` struct: | `model_tier` | Symbol | `:simple` or `:complex` | | `cache_status` | Symbol | `:hit` or `:miss` | | `original_tokens` | Integer | Estimated token count before compression | -| `compressed_tokens` | Integer | Estimated token count after compression (`nil` if not compressed) | +| `compressed_tokens` | Integer | Estimated tokens after compression (`nil` if not compressed) | +| `input_tokens` / `output_tokens` | Integer | Real usage when the adapter reports it | | `latency_ms` | Float | Total wall-clock time for the optimize call | -| `messages` | Array | Final messages array sent to the LLM, after history management and conversation hydration (`nil` on a cache hit) | - -The `messages` field reflects the actual array passed to `messages_caller` (or built from `conversation_id`), including any summarization applied by the history manager. You can pass it back as `options[:messages]` on the next call to continue a stateless conversation. +| `messages` | Array | Final messages array sent to the LLM (`nil` on a cache hit) | ## Resilience @@ -242,6 +232,17 @@ The `messages` field reflects the actual array passed to `messages_caller` (or b | Conversation load failure | Log warning, proceed without history | | Conversation save failure | Log warning, return result with pre-save messages | +## Upgrading from 0.1.x + +Existing initializers keep working — flat config keys and the caller lambdas are still accepted. To move to the new API: + +1. Replace `llm_caller` / `messages_caller` / `embedding_caller` with one `config.adapter` (built-in symbol or `Adapters::Custom`). +2. Replace `classifier_caller` with `config.routing.classifier_model`. +3. Optionally switch flat keys to their grouped equivalents (`use_semantic_cache` → `cache.enabled`, etc.). +4. Add `LlmOptimizer.validate!` at the end of the initializer. + +Note: enabling the cache without Redis/embedding support now raises a `ConfigurationError` instead of silently disabling the cache. + ## Development ```bash @@ -251,13 +252,8 @@ bundle exec rake rubocop # lint bundle exec rake # test + lint ``` -Generate the Rails initializer in a target app: - -```bash -rails generate llm_optimizer:install -``` - ## Contribution + See [CONTRIBUTING.md](https://github.com/arunkumarry/llm_optimizer/blob/main/CONTRIBUTING.md) ## License diff --git a/lib/generators/llm_optimizer/templates/initializer.rb b/lib/generators/llm_optimizer/templates/initializer.rb index 05b3741..27deaa8 100644 --- a/lib/generators/llm_optimizer/templates/initializer.rb +++ b/lib/generators/llm_optimizer/templates/initializer.rb @@ -1,123 +1,82 @@ # frozen_string_literal: true -# LlmOptimizer initializer -# Run `rails generate llm_optimizer:install` to regenerate this file. -# -# Docs: https://github.com/arunkumar/llm_optimizer +# LlmOptimizer — run `rails generate llm_optimizer:install` to regenerate. +# Docs: https://github.com/arunkumarry/llm_optimizer LlmOptimizer.configure do |config| - # --- Feature flags --- - # All optimizations are off by default. Enable what you need. - config.compress_prompt = false # strip stop words before sending to LLM - config.use_semantic_cache = false # cache responses by vector similarity in Redis - config.manage_history = false # summarize old messages when over token budget - - # --- Model routing --- - # :auto classifies each prompt; :simple or :complex forces a tier - config.route_to = :auto - config.simple_model = "gemini-1.5-flash" - config.complex_model = "claude-haiku-4-5" - - # --- Redis (required only if use_semantic_cache: true) --- - config.redis_url = ENV.fetch("REDIS_URL", nil) + # 1. Adapter — how the gem talks to your LLM provider. + # Built-in: :ruby_llm (recommended, covers OpenAI/Anthropic/Gemini/Bedrock/ + # Ollama via the ruby_llm gem), :open_ai (ruby-openai gem), :anthropic. + # Any other provider: see "Custom adapter" in the appendix below. + config.adapter = :ruby_llm - # --- Tuning --- - config.similarity_threshold = 0.96 # cosine similarity cutoff for a cache hit - config.token_budget = 4000 # token limit before history summarization kicks in - config.cache_ttl = 86_400 # cache entry TTL in seconds (default: 24h) - config.timeout_seconds = 5 # timeout for embedding / external API calls + # 2. Models — cheap tier for simple prompts, premium tier for complex ones. + config.routing.simple_model = "gpt-4o-mini" + config.routing.complex_model = "claude-sonnet-4-5" - # --- Tools --- - # config.with_tools = [] # Array of tool definitions (OpenAI/Anthropic format) + # 3. Optimizations — all off by default; enable what you need. + config.cache.enabled = false # semantic cache (needs redis_url) + config.compression.enabled = false # stop-word pruning before the LLM call + config.history.enabled = false # summarize old conversation messages - # --- Logging --- - config.logger = Rails.logger - config.debug_logging = Rails.env.development? + # Required when cache.enabled or conversation_id is used: + # config.redis_url = ENV["REDIS_URL"] - # --- LLM caller (required) --- - # Wire this up to however your app already calls the LLM. - # - # Example with ruby-openai: - # config.llm_caller = ->(prompt, model:) { - # OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"]) - # .chat(parameters: { model: model, messages: [{ role: "user", content: prompt }] }) - # .dig("choices", 0, "message", "content") - # } - # - # Example with a shared service object: - # config.llm_caller = ->(prompt, model:) { - # provider = if model.include?("claude") then :anthropic - # elsif model.include?("gpt") then :openai - # elsif model.include?("gemini") then :gemini - # elsif model.include?("nova") || model.include?("amazon") then :bedrock - # else :ollama - # end - # RubyLLM.chat(model: model, provider: provider, assume_model_exists: true) } - # end - # - config.llm_caller = lambda { |_prompt, **_kwargs| - raise NotImplementedError, "[llm_optimizer] llm_caller is not configured. " \ - "Edit config/initializers/llm_optimizer.rb and wire it to your LLM client." - } - - # --- Embeddings caller (optional) --- - # Only needed if use_semantic_cache: true. - # If omitted, falls back to OpenAI via ENV["OPENAI_API_KEY"]. - # - # Example: - # config.embedding_caller = ->(text) { EmbeddingService.embed(text) } - # - # --- Routing classifier (optional) --- - # When set, ambiguous prompts are classified by a cheap LLM instead of - # falling back to the word-count heuristic. Unambiguous signals (code blocks, - # keywords) still bypass the classifier for speed. - # - # Example: - # config.classifier_caller = ->(prompt) { - # RubyLLM.chat(model: "amazon.nova-micro-v1:0", assume_model_exists: true) - # .ask(prompt).content.strip.downcase - # } - # - # config.classifier_caller = nil - - # --- Messages caller (optional) --- - # Messages caller for history manager/conversation summary - Optional - # config.system_prompt = "You are a helpful person who gives responses in a non harmful way. " \ - # "If any serious question is asked, handle it in effectively." - # OpenAI implementation - - # config.messages_caller = ->(messages, model:, tools: nil) { - # parameters = { - # model: model, - # messages: messages.map { |m| { role: m[:role], content: m[:content] } } - # } - # parameters[:tools] = tools if tools&.any? - # - # response = $openai.chat(parameters: parameters) - # response.dig("choices", 0, "message", "content") - # } + config.logger = Rails.logger +end - # RubyLLM implementation - - # config.messages_caller = ->(messages, model:, tools: nil) { - # chat = RubyLLM.chat(model: model) - # chat.with_tools(*tools) if tools&.any? - # messages[0..-2].each { |m| chat.add_message(role: m[:role], content: m[:content]) } - # chat.ask(messages.last[:content]).content - # } +# Fails fast at boot with an actionable message if anything is missing: +LlmOptimizer.validate! - # Anthropic implementation - - # config.messages_caller = ->(messages, model:, tools: nil) { - # # Anthropic separates system messages from the messages array - # system_msg = messages.find { |m| m[:role] == "system" }&.dig(:content) - # chat_msgs = messages.reject { |m| m[:role] == "system" } - # .map { |m| { role: m[:role], content: m[:content] } } - # - # response = $anthropic.messages( - # model: model, - # max_tokens: 1024, - # system: system_msg, - # messages: chat_msgs, - # tools: tools - # ) - # response["content"].first["text"] - # } -end +# --------------------------------------------------------------------------- +# Appendix — advanced options +# --------------------------------------------------------------------------- +# +# Custom adapter (any provider — you supply the lambdas): +# +# config.adapter = LlmOptimizer::Adapters::Custom.new( +# chat: ->(messages, model:, tools: nil) { +# # messages is an Array of { role:, content: } hashes. +# # Return a String, or { content:, input_tokens:, output_tokens: }. +# MyLlmService.chat(messages, model: model) +# }, +# embed: ->(text) { MyEmbeddingService.embed(text) } # needed for the cache +# ) +# +# Adapter with options (instead of the symbol shorthand): +# +# config.adapter = LlmOptimizer::Adapters::OpenAI.new( +# access_token: Rails.application.credentials.openai_api_key, +# embedding_model: "text-embedding-3-large" +# ) +# +# Routing: +# +# config.routing.mode = :auto # :auto | :simple | :complex +# config.routing.classifier_model = "gpt-4o-mini" # LLM-classify ambiguous +# # prompts (default: word-count +# # heuristic, no LLM call) +# +# Semantic cache tuning: +# +# config.cache.threshold = 0.96 # cosine similarity cutoff for a hit +# config.cache.ttl = 86_400 # seconds +# config.cache.scope = nil # namespace hits (e.g. per user/tenant) +# +# Conversations (LlmOptimizer.optimize(prompt, conversation_id: "abc")): +# +# config.conversation.system_prompt = "You are a helpful assistant." +# config.conversation.ttl = 86_400 # 0 = no expiry +# +# History summarization: +# +# config.history.token_budget = 4000 # summarize once history exceeds this +# +# Tools / structured output (passed through to the adapter): +# +# config.tools = [ ... ] # provider-format tool definitions +# config.schema = { ... } # JSON schema for structured output +# +# Debugging: +# +# config.debug_logging = Rails.env.development? # logs full prompt + response diff --git a/lib/llm_optimizer.rb b/lib/llm_optimizer.rb index ff7468c..8fbad41 100644 --- a/lib/llm_optimizer.rb +++ b/lib/llm_optimizer.rb @@ -1,6 +1,8 @@ # frozen_string_literal: true require_relative "llm_optimizer/version" +require_relative "llm_optimizer/invoker" +require_relative "llm_optimizer/adapters" require_relative "llm_optimizer/configuration" require_relative "llm_optimizer/optimize_result" require_relative "llm_optimizer/compressor" @@ -27,17 +29,14 @@ def self.configure temp = Configuration.new yield temp configuration.merge!(temp) - validate_configuration!(configuration) + configuration.validate_values! + configuration end - def self.validate_configuration!(config) - return unless config.use_semantic_cache && config.embedding_caller.nil? - - config.logger.warn( - "[llm_optimizer] use_semantic_cache is true but no embedding_caller is configured. " \ - "Semantic caching will be skipped. Set config.embedding_caller to enable it." - ) - config.use_semantic_cache = false + # Full configuration check — call from the end of your initializer to fail + # fast at boot with an actionable message instead of at the first LLM call. + def self.validate! + configuration.validate! end def self.configuration @@ -90,6 +89,7 @@ def self.optimize(prompt, options = {}, &) call_config = build_call_config(options, &) conversation_id = options[:conversation_id] validate_conversation_options!(conversation_id, options, call_config) + call_config.validate! original_prompt = prompt original_tokens = Compressor.new.estimate_tokens(prompt) diff --git a/lib/llm_optimizer/adapters.rb b/lib/llm_optimizer/adapters.rb new file mode 100644 index 0000000..dd9db1c --- /dev/null +++ b/lib/llm_optimizer/adapters.rb @@ -0,0 +1,31 @@ +# frozen_string_literal: true + +require_relative "adapters/base" +require_relative "adapters/custom" +require_relative "adapters/legacy" +require_relative "adapters/ruby_llm" +require_relative "adapters/open_ai" +require_relative "adapters/anthropic" + +module LlmOptimizer + module Adapters + REGISTRY = { + ruby_llm: "RubyLLM", + open_ai: "OpenAI", + openai: "OpenAI", + anthropic: "Anthropic" + }.freeze + + def self.resolve(id, **) + class_name = REGISTRY[id.to_sym] + unless class_name + raise ConfigurationError, + "Unknown adapter #{id.inspect}. Built-in adapters: :ruby_llm, :open_ai, :anthropic. " \ + "For any other provider, pass an adapter instance " \ + "(e.g. LlmOptimizer::Adapters::Custom.new(chat: ->(messages, model:) { ... }))." + end + + const_get(class_name).new(**) + end + end +end diff --git a/lib/llm_optimizer/adapters/anthropic.rb b/lib/llm_optimizer/adapters/anthropic.rb new file mode 100644 index 0000000..053c339 --- /dev/null +++ b/lib/llm_optimizer/adapters/anthropic.rb @@ -0,0 +1,57 @@ +# frozen_string_literal: true + +module LlmOptimizer + module Adapters + # Adapter for the official anthropic gem (https://rubygems.org/gems/anthropic). + # + # config.adapter = :anthropic # uses ENV["ANTHROPIC_API_KEY"] + # # or with options / your own client: + # config.adapter = LlmOptimizer::Adapters::Anthropic.new(api_key: "...", max_tokens: 2048) + # config.adapter = LlmOptimizer::Adapters::Anthropic.new(client: $anthropic) + # + # Anthropic has no embeddings API, so semantic caching needs a separate + # config.embedding_caller (e.g. Voyage AI or OpenAI embeddings). + class Anthropic < Base + DEFAULT_MAX_TOKENS = 4096 + + def initialize(client: nil, api_key: nil, max_tokens: DEFAULT_MAX_TOKENS) + @client = client + @api_key = api_key + @max_tokens = max_tokens + super() + end + + def chat(messages, model:, tools: nil, schema: nil) # rubocop:disable Lint/UnusedMethodArgument + system_parts = messages.select { |m| message_role(m) == "system" }.map { |m| message_content(m) } + chat_messages = messages.reject { |m| message_role(m) == "system" } + .map { |m| { role: message_role(m), content: message_content(m) } } + + parameters = { model: model, max_tokens: @max_tokens, messages: chat_messages } + parameters[:system] = system_parts.join("\n\n") unless system_parts.empty? + parameters[:tools] = tools if tools && !tools.empty? + + response = client.messages.create(**parameters) + + { content: extract_text(response), + input_tokens: response.usage&.input_tokens, + output_tokens: response.usage&.output_tokens } + end + + private + + def extract_text(response) + response.content + .select { |block| block.respond_to?(:type) && block.type.to_s == "text" } + .map(&:text) + .join + end + + def client + @client ||= begin + require_dependency!("anthropic") + ::Anthropic::Client.new(api_key: @api_key || ENV.fetch("ANTHROPIC_API_KEY", nil)) + end + end + end + end +end diff --git a/lib/llm_optimizer/adapters/base.rb b/lib/llm_optimizer/adapters/base.rb new file mode 100644 index 0000000..fa88802 --- /dev/null +++ b/lib/llm_optimizer/adapters/base.rb @@ -0,0 +1,50 @@ +# frozen_string_literal: true + +module LlmOptimizer + module Adapters + # Interface every adapter implements. + # + # chat(messages, model:, tools: nil, schema: nil) + # messages — Array of { role:, content: } hashes + # (roles: "system", "user", "assistant") + # returns — String, or a Hash with :content plus optional + # :input_tokens, :output_tokens, :cached_tokens + # + # embed(text) + # returns — Array embedding vector + # (only required when supports_embedding? is true) + class Base + def chat(_messages, model:, tools: nil, schema: nil) + raise NotImplementedError, "#{self.class} must implement #chat" + end + + def embed(_text) + raise EmbeddingError, + "#{self.class} does not support embeddings. Use an adapter that does, " \ + "or set config.embedding_caller = ->(text) { ... }." + end + + def supports_embedding? + false + end + + private + + def require_dependency!(gem_name, require_path = gem_name) + require require_path + rescue LoadError + raise ConfigurationError, + "The #{self.class.name.split("::").last} adapter needs the #{gem_name} gem. " \ + "Add `gem \"#{gem_name}\"` to your Gemfile and run `bundle install`." + end + + def message_role(message) + (message[:role] || message["role"]).to_s + end + + def message_content(message) + message[:content] || message["content"] + end + end + end +end diff --git a/lib/llm_optimizer/adapters/custom.rb b/lib/llm_optimizer/adapters/custom.rb new file mode 100644 index 0000000..f3c7de1 --- /dev/null +++ b/lib/llm_optimizer/adapters/custom.rb @@ -0,0 +1,36 @@ +# frozen_string_literal: true + +module LlmOptimizer + module Adapters + # Escape hatch for providers without a built-in adapter. Wraps plain + # lambdas and invokes them defensively — only the keyword arguments a + # lambda's signature accepts are passed, so a simple + # ->(messages, model:) { ... } works fine. + # + # config.adapter = LlmOptimizer::Adapters::Custom.new( + # chat: ->(messages, model:, tools: nil, schema: nil) { ... }, # -> String or {content:, ...} + # embed: ->(text) { ... } # -> Array (optional) + # ) + class Custom < Base + def initialize(chat:, embed: nil) + @chat = chat + @embed = embed + super() + end + + def chat(messages, model:, tools: nil, schema: nil) + Invoker.call(@chat, messages, model: model, tools: tools, schema: schema) + end + + def embed(text) + return super unless @embed + + Invoker.call(@embed, text) + end + + def supports_embedding? + !@embed.nil? + end + end + end +end diff --git a/lib/llm_optimizer/adapters/legacy.rb b/lib/llm_optimizer/adapters/legacy.rb new file mode 100644 index 0000000..cd599cb --- /dev/null +++ b/lib/llm_optimizer/adapters/legacy.rb @@ -0,0 +1,40 @@ +# frozen_string_literal: true + +module LlmOptimizer + module Adapters + # Internal adapter built automatically from the deprecated 0.1.x caller + # lambdas (llm_caller / messages_caller / embedding_caller) so old + # initializers keep working. Prefer config.adapter going forward. + class Legacy < Base + def initialize(llm_caller: nil, messages_caller: nil, embedding_caller: nil) + @llm_caller = llm_caller + @messages_caller = messages_caller + @embedding_caller = embedding_caller + super() + end + + def chat(messages, model:, tools: nil, schema: nil) + if @messages_caller && messages.length > 1 + Invoker.call(@messages_caller, messages, model: model, tools: tools, schema: schema) + else + unless @llm_caller + raise ConfigurationError, + "No llm_caller configured. Set config.adapter (e.g. :ruby_llm) or config.llm_caller." + end + + Invoker.call(@llm_caller, message_content(messages.last), model: model, tools: tools, schema: schema) + end + end + + def embed(text) + return super unless @embedding_caller + + Invoker.call(@embedding_caller, text) + end + + def supports_embedding? + !@embedding_caller.nil? + end + end + end +end diff --git a/lib/llm_optimizer/adapters/open_ai.rb b/lib/llm_optimizer/adapters/open_ai.rb new file mode 100644 index 0000000..f428470 --- /dev/null +++ b/lib/llm_optimizer/adapters/open_ai.rb @@ -0,0 +1,61 @@ +# frozen_string_literal: true + +module LlmOptimizer + module Adapters + # Adapter for the ruby-openai gem (https://rubygems.org/gems/ruby-openai). + # + # config.adapter = :open_ai # uses ENV["OPENAI_API_KEY"] + # # or with options / your own client: + # config.adapter = LlmOptimizer::Adapters::OpenAI.new( + # access_token: Rails.application.credentials.openai_api_key, + # embedding_model: "text-embedding-3-large" + # ) + # config.adapter = LlmOptimizer::Adapters::OpenAI.new(client: $openai) + class OpenAI < Base + DEFAULT_EMBEDDING_MODEL = "text-embedding-3-small" + + def initialize(client: nil, access_token: nil, embedding_model: nil, **client_options) + @client = client + @access_token = access_token + @client_options = client_options + @embedding_model = embedding_model || DEFAULT_EMBEDDING_MODEL + super() + end + + def chat(messages, model:, tools: nil, schema: nil) + parameters = { + model: model, + messages: messages.map { |m| { role: message_role(m), content: message_content(m) } } + } + parameters[:tools] = tools if tools && !tools.empty? + parameters[:response_format] = { type: "json_schema", json_schema: schema } if schema + + response = client.chat(parameters: parameters) + + { content: response.dig("choices", 0, "message", "content"), + input_tokens: response.dig("usage", "prompt_tokens"), + output_tokens: response.dig("usage", "completion_tokens") } + end + + def embed(text) + response = client.embeddings(parameters: { model: @embedding_model, input: text }) + response.dig("data", 0, "embedding") + end + + def supports_embedding? + true + end + + private + + def client + @client ||= begin + require_dependency!("ruby-openai", "openai") + options = @client_options.dup + options[:access_token] = @access_token || ENV.fetch("OPENAI_API_KEY", nil) + ::OpenAI::Client.new(**options) + end + end + end + end +end diff --git a/lib/llm_optimizer/adapters/ruby_llm.rb b/lib/llm_optimizer/adapters/ruby_llm.rb new file mode 100644 index 0000000..ea69fae --- /dev/null +++ b/lib/llm_optimizer/adapters/ruby_llm.rb @@ -0,0 +1,62 @@ +# frozen_string_literal: true + +module LlmOptimizer + module Adapters + # Adapter for the ruby_llm gem (https://rubygems.org/gems/ruby_llm). + # Covers OpenAI, Anthropic, Gemini, Bedrock, Ollama and more through a + # single dependency — the recommended default. + # + # config.adapter = :ruby_llm + # # or with options: + # config.adapter = LlmOptimizer::Adapters::RubyLLM.new( + # provider: :bedrock, assume_model_exists: true, + # embedding_model: "text-embedding-3-small" + # ) + class RubyLLM < Base + def initialize(provider: nil, assume_model_exists: nil, embedding_model: nil) + @provider = provider + @assume_model_exists = assume_model_exists + @embedding_model = embedding_model + super() + end + + def chat(messages, model:, tools: nil, schema: nil) + require_dependency!("ruby_llm") + + chat = ::RubyLLM.chat(**chat_options(model)) + chat = chat.with_tools(*tools) if tools && !tools.empty? + chat = chat.with_schema(schema) if schema + + *history, current = messages + history.each { |m| chat.add_message(role: message_role(m).to_sym, content: message_content(m)) } + response = chat.ask(message_content(current)) + + { content: response.content, + input_tokens: response.respond_to?(:input_tokens) ? response.input_tokens : nil, + output_tokens: response.respond_to?(:output_tokens) ? response.output_tokens : nil } + end + + def embed(text) + require_dependency!("ruby_llm") + + options = {} + options[:model] = @embedding_model if @embedding_model + ::RubyLLM.embed(text, **options).vectors + end + + def supports_embedding? + true + end + + private + + def chat_options(model) + options = {} + options[:model] = model if model + options[:provider] = @provider if @provider + options[:assume_model_exists] = @assume_model_exists unless @assume_model_exists.nil? + options + end + end + end +end diff --git a/lib/llm_optimizer/configuration.rb b/lib/llm_optimizer/configuration.rb index 4529eb3..60a7f68 100644 --- a/lib/llm_optimizer/configuration.rb +++ b/lib/llm_optimizer/configuration.rb @@ -4,70 +4,182 @@ module LlmOptimizer class Configuration - KNOWN_KEYS = %i[ - use_semantic_cache - compress_prompt - manage_history - route_to - similarity_threshold - token_budget - redis_url - embedding_model - simple_model - complex_model - logger - debug_logging - timeout_seconds - cache_ttl - llm_caller - embedding_caller - classifier_caller - messages_caller - system_prompt - conversation_ttl - cache_scope - tools - with_tools - tools_caller - schema + # Base class for nested option groups (config.routing, config.cache, ...). + # Tracks explicitly set keys so merge! only copies what the user touched. + class Group + class << self + def settings + @settings ||= {} + end + + def setting(name, default: nil) + settings[name] = default + define_method(name) { @values.key?(name) ? @values[name] : self.class.settings[name] } + define_method(:"#{name}=") do |value| + @explicitly_set << name + @values[name] = value + end + end + end + + attr_reader :explicitly_set + + def initialize + @values = {} + @explicitly_set = Set.new + end + + def merge!(other) + other.explicitly_set.each { |key| public_send(:"#{key}=", other.public_send(key)) } + self + end + end + + class Routing < Group + MODES = %i[auto simple complex].freeze + + setting :mode, default: :auto + setting :simple_model, default: "gpt-4o-mini" + setting :complex_model, default: "claude-sonnet-4-5" + setting :classifier_model # when set, ambiguous prompts are classified by this model + end + + class Cache < Group + setting :enabled, default: false + setting :threshold, default: 0.96 + setting :ttl, default: 86_400 + setting :scope + setting :embedding_model # nil lets the adapter pick its own default + end + + class Compression < Group + setting :enabled, default: false + end + + class History < Group + setting :enabled, default: false + setting :token_budget, default: 4000 + end + + class Conversation < Group + setting :ttl, default: 86_400 + setting :system_prompt + end + + GROUPS = { + routing: Routing, + cache: Cache, + compression: Compression, + history: History, + conversation: Conversation + }.freeze + + # Flat keys kept for 0.1.x compatibility and terse per-call overrides, + # e.g. LlmOptimizer.optimize(prompt, route_to: :simple). + ALIASES = { + route_to: %i[routing mode], + simple_model: %i[routing simple_model], + complex_model: %i[routing complex_model], + classifier_model: %i[routing classifier_model], + use_semantic_cache: %i[cache enabled], + similarity_threshold: %i[cache threshold], + cache_ttl: %i[cache ttl], + cache_scope: %i[cache scope], + embedding_model: %i[cache embedding_model], + compress_prompt: %i[compression enabled], + manage_history: %i[history enabled], + token_budget: %i[history token_budget], + conversation_ttl: %i[conversation ttl], + system_prompt: %i[conversation system_prompt] + }.freeze + + TOP_LEVEL_KEYS = %i[ + adapter redis_url logger debug_logging timeout_seconds tools schema + llm_caller messages_caller embedding_caller classifier_caller ].freeze - # Define readers for all known keys (setters below track explicit sets) - KNOWN_KEYS.each { |key| define_method(key) { instance_variable_get(:"@#{key}") } } + TOP_LEVEL_DEFAULTS = { + debug_logging: false, + timeout_seconds: 5 + }.freeze + + # Every key accepted as a per-call option on LlmOptimizer.optimize. + KNOWN_KEYS = (TOP_LEVEL_KEYS + ALIASES.keys + %i[with_tools]).freeze + + attr_reader :explicitly_set def initialize @explicitly_set = Set.new + @values = TOP_LEVEL_DEFAULTS.merge(logger: Logger.new($stdout)) + @groups = GROUPS.transform_values(&:new) + end - @use_semantic_cache = false - @compress_prompt = false - @manage_history = false - @route_to = :auto - @similarity_threshold = 0.96 - @token_budget = 4000 - @redis_url = nil - @embedding_model = "text-embedding-3-small" - @simple_model = "gpt-4o-mini" - @complex_model = "claude-3-5-sonnet-20241022" - @logger = Logger.new($stdout) - @debug_logging = false - @timeout_seconds = 5 - @cache_ttl = 86_400 - @llm_caller = nil - @embedding_caller = nil - @classifier_caller = nil - @conversation_ttl = 86_400 - @system_prompt = nil - @with_tools = nil - end - - # Copies only explicitly set keys from other_config without resetting unmentioned keys. - def merge!(other_config) - other_config.instance_variable_get(:@explicitly_set).each do |key| - public_send(:"#{key}=", other_config.public_send(key)) + GROUPS.each_key do |name| + define_method(name) { @groups[name] } + end + + TOP_LEVEL_KEYS.each do |key| + define_method(key) { @values[key] } + define_method(:"#{key}=") do |value| + @explicitly_set << key + @values[key] = value end + end + + ALIASES.each do |alias_name, (group, attr)| + define_method(alias_name) { @groups[group].public_send(attr) } + define_method(:"#{alias_name}=") { |value| @groups[group].public_send(:"#{attr}=", value) } + end + + # `with_tools` is an alias of `tools`. + def with_tools + tools + end + + def with_tools=(value) + self.tools = value + end + + # Copies only explicitly set keys from other_config without resetting + # unmentioned keys. + def merge!(other_config) + other_config.explicitly_set.each { |key| public_send(:"#{key}=", other_config.public_send(key)) } + GROUPS.each_key { |name| @groups[name].merge!(other_config.public_send(name)) } self end + # The adapter actually used for LLM calls: an explicit adapter instance, + # a registry symbol (:ruby_llm, :open_ai, :anthropic), or — when only the + # deprecated 0.1.x caller lambdas are set — a Legacy adapter wrapping them. + def resolved_adapter + case adapter + when nil then legacy_adapter + when Symbol, String then Adapters.resolve(adapter, **symbol_adapter_options) + else adapter + end + end + + def embedding_support? + !embedding_caller.nil? || !!resolved_adapter&.supports_embedding? + end + + # Cheap value-level checks, run on every `LlmOptimizer.configure`. + def validate_values! + errors = value_errors + raise ConfigurationError, format_errors(errors) unless errors.empty? + + true + end + + # Full cross-field validation. Run automatically on each optimize call; + # call `LlmOptimizer.validate!` from your initializer to fail fast at boot. + def validate! + errors = value_errors + cross_field_errors + raise ConfigurationError, format_errors(errors) unless errors.empty? + + true + end + def method_missing(name, *args, &) key = name.to_s.chomp("=").to_sym raise ConfigurationError, "Unknown configuration key: #{key}" unless KNOWN_KEYS.include?(key) @@ -80,12 +192,66 @@ def respond_to_missing?(name, include_private = false) KNOWN_KEYS.include?(key) || super end - # Override generated attr_accessor setters to track explicitly set keys. - KNOWN_KEYS.each do |key| - define_method(:"#{key}=") do |value| - @explicitly_set << key - instance_variable_set(:"@#{key}", value) + private + + def value_errors + errors = [] + unless Routing::MODES.include?(routing.mode) + errors << "routing.mode must be one of :auto, :simple, :complex (got #{routing.mode.inspect})" + end + unless cache.threshold.is_a?(Numeric) && cache.threshold.between?(0, 1) + errors << "cache.threshold must be a number between 0 and 1 (got #{cache.threshold.inspect})" end + errors + end + + def cross_field_errors + errors = [] + errors.concat(adapter_errors) + errors.concat(cache_errors) + errors + rescue ConfigurationError => e + errors << e.message + errors + end + + def adapter_errors + return [] if resolved_adapter + + ["no adapter configured. Set config.adapter = :ruby_llm (or :open_ai, :anthropic), " \ + "or pass your own: config.adapter = LlmOptimizer::Adapters::Custom.new(chat: ->(messages, model:) { ... })"] + end + + def cache_errors + return [] unless cache.enabled + + errors = [] + unless redis_url + errors << "cache.enabled is true but redis_url is not set. " \ + "Set config.redis_url (e.g. ENV[\"REDIS_URL\"]) or disable the cache." + end + unless embedding_support? + errors << "cache.enabled is true but the configured adapter does not support embeddings. " \ + "Use an adapter with embeddings (:ruby_llm, :open_ai) or set config.embedding_caller." + end + errors + end + + def format_errors(errors) + "Invalid LlmOptimizer configuration:\n- #{errors.join("\n- ")}" + end + + def symbol_adapter_options + return {} unless cache.explicitly_set.include?(:embedding_model) + + { embedding_model: cache.embedding_model } + end + + def legacy_adapter + return nil unless llm_caller || messages_caller || embedding_caller + + Adapters::Legacy.new(llm_caller: llm_caller, messages_caller: messages_caller, + embedding_caller: embedding_caller) end end end diff --git a/lib/llm_optimizer/invoker.rb b/lib/llm_optimizer/invoker.rb new file mode 100644 index 0000000..f708f07 --- /dev/null +++ b/lib/llm_optimizer/invoker.rb @@ -0,0 +1,27 @@ +# frozen_string_literal: true + +module LlmOptimizer + # Invokes user-supplied callables (lambdas, procs, or any object responding + # to #call), passing only the keyword arguments the callable's signature + # accepts. This lets the gem add new keywords (tools:, schema:, ...) over + # time without breaking lambdas written against an older signature. + module Invoker + module_function + + def call(callable, *, **kwargs) + params = parameters_for(callable) + return callable.call(*, **kwargs) if params.nil? || params.any? { |type, _name| type == :keyrest } + + accepted = params.filter_map { |type, name| name if %i[key keyreq].include?(type) } + callable.call(*, **kwargs.slice(*accepted)) + end + + def parameters_for(callable) + return callable.parameters if callable.respond_to?(:parameters) + + callable.method(:call).parameters + rescue NameError + nil + end + end +end diff --git a/lib/llm_optimizer/model_router.rb b/lib/llm_optimizer/model_router.rb index cec0f3b..c55d3f9 100644 --- a/lib/llm_optimizer/model_router.rb +++ b/lib/llm_optimizer/model_router.rb @@ -26,7 +26,8 @@ def initialize(config) def route(prompt) # Explicit override — always - return @config.route_to if %i[simple complex].include?(@config.route_to) + mode = @config.routing.mode + return mode if %i[simple complex].include?(mode) # Unambiguous fast-path signals (no LLM call needed) return :complex if CODE_BLOCK_RE.match?(prompt) @@ -36,7 +37,7 @@ def route(prompt) return :complex if COMPLEX_PHRASES.any? { |ph| lower.include?(ph) } # LLM classifier for ambiguous prompts - if @config.classifier_caller + if classifier_configured? result = classify_with_llm(prompt) return result if result end @@ -47,10 +48,13 @@ def route(prompt) private + def classifier_configured? + !@config.classifier_caller.nil? || !@config.routing.classifier_model.nil? + end + def classify_with_llm(prompt) classifier_prompt = format(CLASSIFIER_PROMPT, prompt: prompt) - response = @config.classifier_caller.call(classifier_prompt) - normalized = response.to_s.strip.downcase + normalized = classifier_response(classifier_prompt).to_s.strip.downcase # Check for word boundary match to handle responses like # "simple." / "**simple**" / "the answer is simple" @@ -61,5 +65,15 @@ def classify_with_llm(prompt) rescue StandardError nil # classifier failure — fall through to heuristic end + + def classifier_response(classifier_prompt) + return @config.classifier_caller.call(classifier_prompt) if @config.classifier_caller + + result = @config.resolved_adapter&.chat( + [{ role: "user", content: classifier_prompt }], + model: @config.routing.classifier_model + ) + result.is_a?(Hash) ? result[:content] : result + end end end diff --git a/lib/llm_optimizer/pipeline.rb b/lib/llm_optimizer/pipeline.rb index 901916f..57c4f2f 100644 --- a/lib/llm_optimizer/pipeline.rb +++ b/lib/llm_optimizer/pipeline.rb @@ -59,7 +59,7 @@ def load_conversation(conversation_id, options, config) def apply_history_manager(messages, config) return messages unless config.manage_history && messages - llm_caller = ->(p, model:) { raw_llm_call(p, model: model, config: config) } + llm_caller = ->(p, model:) { raw_llm_call(p, model: model, config: config).first } history_mgr = HistoryManager.new( llm_caller: llm_caller, simple_model: config.simple_model, @@ -96,15 +96,15 @@ def fallback_result(original_prompt, original_tokens, options, start) end def raw_llm_call(prompt, model:, messages: nil, config: nil) - tools = config&.with_tools || config&.tools - result = if messages && !messages.empty? && config&.messages_caller - config.messages_caller.call(messages + [{ role: "user", content: prompt }], model: model, tools: tools) - else - llm = config&.llm_caller || @_current_llm_caller - raise ConfigurationError, "No llm_caller configured." unless llm + adapter = config&.resolved_adapter + unless adapter + raise ConfigurationError, + "No adapter configured. Set config.adapter = :ruby_llm (or :open_ai, :anthropic), " \ + "or pass your own via LlmOptimizer::Adapters::Custom." + end - llm.call(prompt, model: model, tools: tools) - end + full_messages = Array(messages) + [{ role: "user", content: prompt }] + result = adapter.chat(full_messages, model: model, tools: config.tools, schema: config.schema) if result.is_a?(Hash) [result[:content], result] @@ -113,6 +113,12 @@ def raw_llm_call(prompt, model:, messages: nil, config: nil) end end + def compute_embedding(prompt, config) + return config.embedding_caller.call(prompt) if config.embedding_caller + + config.resolved_adapter.embed(prompt) + end + def elapsed_ms(start) ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000).round(2) end @@ -138,7 +144,12 @@ def semantic_cache_lookup(prompt, model, model_tier, original_tokens, compressed_tokens, original_prompt, start, config) return [nil, nil] unless config.use_semantic_cache - embedding = config.embedding_caller.call(prompt) + unless config.redis_url + config.logger.warn("[llm_optimizer] semantic cache enabled but redis_url is not set — skipping cache") + return [nil, nil] + end + + embedding = compute_embedding(prompt, config) cache = SemanticCache.new(build_redis(config.redis_url), threshold: config.similarity_threshold, ttl: config.cache_ttl, diff --git a/lib/llm_optimizer/version.rb b/lib/llm_optimizer/version.rb index 11e9dab..7f7932a 100644 --- a/lib/llm_optimizer/version.rb +++ b/lib/llm_optimizer/version.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true module LlmOptimizer - VERSION = "0.1.7" + VERSION = "0.2.0" end diff --git a/test/unit/test_adapters.rb b/test/unit/test_adapters.rb new file mode 100644 index 0000000..154c91d --- /dev/null +++ b/test/unit/test_adapters.rb @@ -0,0 +1,128 @@ +# frozen_string_literal: true + +require_relative "../test_helper" + +class TestInvoker < Minitest::Test + def test_passes_only_accepted_keywords + received = nil + callable = ->(prompt, model:) { received = [prompt, model] } + LlmOptimizer::Invoker.call(callable, "hi", model: "m1", tools: [1], schema: {}) + assert_equal %w[hi m1], received + end + + def test_passes_all_keywords_to_keyrest + received = nil + callable = ->(_prompt, **kwargs) { received = kwargs } + LlmOptimizer::Invoker.call(callable, "hi", model: "m1", tools: [1]) + assert_equal({ model: "m1", tools: [1] }, received) + end + + def test_passes_optional_keywords_when_accepted + received = nil + callable = ->(_prompt, model:, tools: nil) { received = { model: model, tools: tools } } + LlmOptimizer::Invoker.call(callable, "hi", model: "m1", tools: [1], schema: {}) + assert_equal({ model: "m1", tools: [1] }, received) + end + + def test_works_with_positional_only_callable + received = nil + callable = ->(text) { received = text } + LlmOptimizer::Invoker.call(callable, "hello") + assert_equal "hello", received + end + + def test_works_with_call_objects + obj = Object.new + obj.define_singleton_method(:call) { |prompt, model:| "#{prompt}-#{model}" } + result = LlmOptimizer::Invoker.call(obj, "hi", model: "m1", tools: [1]) + assert_equal "hi-m1", result + end +end + +class TestCustomAdapter < Minitest::Test + def test_chat_receives_messages_and_model + received = nil + adapter = LlmOptimizer::Adapters::Custom.new( + chat: ->(messages, model:) { received = [messages, model] and "ok" } + ) + messages = [{ role: "user", content: "hi" }] + adapter.chat(messages, model: "m1", tools: [1]) + assert_equal [messages, "m1"], received + end + + def test_supports_embedding_only_with_embed_lambda + without_embed = LlmOptimizer::Adapters::Custom.new(chat: ->(_m, model:) { model }) + with_embed = LlmOptimizer::Adapters::Custom.new(chat: ->(_m, model:) { model }, embed: ->(_t) { [1.0] }) + refute without_embed.supports_embedding? + assert with_embed.supports_embedding? + end + + def test_embed_without_lambda_raises_embedding_error + adapter = LlmOptimizer::Adapters::Custom.new(chat: ->(_m, model:) { model }) + assert_raises(LlmOptimizer::EmbeddingError) { adapter.embed("text") } + end +end + +class TestLegacyAdapter < Minitest::Test + def test_single_message_uses_llm_caller_with_prompt_string + received = nil + adapter = LlmOptimizer::Adapters::Legacy.new(llm_caller: ->(prompt, model:) { received = [prompt, model] }) + adapter.chat([{ role: "user", content: "hi" }], model: "m1") + assert_equal %w[hi m1], received + end + + def test_history_prefers_messages_caller + received = nil + adapter = LlmOptimizer::Adapters::Legacy.new( + llm_caller: ->(_p, model:) { model }, + messages_caller: ->(messages, model:) { received = [messages.length, model] and "ok" } + ) + adapter.chat([{ role: "user", content: "a" }, { role: "user", content: "b" }], model: "m1") + assert_equal [2, "m1"], received + end + + def test_history_without_messages_caller_falls_back_to_llm_caller + received = nil + adapter = LlmOptimizer::Adapters::Legacy.new(llm_caller: ->(prompt, model:) { received = [prompt, model] }) + adapter.chat([{ role: "user", content: "a" }, { role: "user", content: "current" }], model: "m1") + assert_equal %w[current m1], received + end + + def test_no_llm_caller_raises_configuration_error + adapter = LlmOptimizer::Adapters::Legacy.new + assert_raises(LlmOptimizer::ConfigurationError) do + adapter.chat([{ role: "user", content: "hi" }], model: "m1") + end + end + + def test_strict_arity_lambda_does_not_receive_unknown_keywords + # Regression: 0.1.6 passed tools: unconditionally, crashing lambdas + # written against the documented ->(prompt, model:) signature. + adapter = LlmOptimizer::Adapters::Legacy.new(llm_caller: ->(prompt, model:) { "#{prompt}-#{model}" }) + result = adapter.chat([{ role: "user", content: "hi" }], model: "m1", tools: [{ name: "t" }], schema: {}) + assert_equal "hi-m1", result + end +end + +class TestAdapterRegistry < Minitest::Test + def test_unknown_adapter_raises_with_available_list + err = assert_raises(LlmOptimizer::ConfigurationError) { LlmOptimizer::Adapters.resolve(:nope) } + assert_includes err.message, ":ruby_llm" + end + + unless defined?(::RubyLLM) + def test_missing_gem_raises_actionable_error + adapter = LlmOptimizer::Adapters.resolve(:ruby_llm) + err = assert_raises(LlmOptimizer::ConfigurationError) do + adapter.chat([{ role: "user", content: "hi" }], model: "m1") + end + assert_includes err.message, "ruby_llm" + assert_includes err.message, "Gemfile" + end + end + + def test_anthropic_adapter_does_not_support_embeddings + adapter = LlmOptimizer::Adapters.resolve(:anthropic) + refute adapter.supports_embedding? + end +end diff --git a/test/unit/test_configuration.rb b/test/unit/test_configuration.rb index 7958e14..cec548a 100644 --- a/test/unit/test_configuration.rb +++ b/test/unit/test_configuration.rb @@ -33,8 +33,8 @@ def test_default_token_budget assert_equal 4000, LlmOptimizer::Configuration.new.token_budget end - def test_default_embedding_model - assert_equal "text-embedding-3-small", LlmOptimizer::Configuration.new.embedding_model + def test_default_embedding_model_is_nil + assert_nil LlmOptimizer::Configuration.new.embedding_model end def test_default_simple_model @@ -42,7 +42,7 @@ def test_default_simple_model end def test_default_complex_model - assert_equal "claude-3-5-sonnet-20241022", LlmOptimizer::Configuration.new.complex_model + assert_equal "claude-sonnet-4-5", LlmOptimizer::Configuration.new.complex_model end def test_default_debug_logging_is_false @@ -171,4 +171,110 @@ def test_reset_configuration_restores_defaults LlmOptimizer.reset_configuration! assert_equal 4000, LlmOptimizer.configuration.token_budget end + + # Grouped settings and flat aliases + + def test_grouped_setting_readable_via_flat_alias + config = LlmOptimizer::Configuration.new + config.cache.enabled = true + assert_equal true, config.use_semantic_cache + end + + def test_flat_alias_writes_into_group + config = LlmOptimizer::Configuration.new + config.route_to = :simple + assert_equal :simple, config.routing.mode + end + + def test_with_tools_aliases_tools + config = LlmOptimizer::Configuration.new + config.with_tools = [{ name: "t" }] + assert_equal [{ name: "t" }], config.tools + end + + def test_merge_copies_group_settings + base = LlmOptimizer::Configuration.new + other = LlmOptimizer::Configuration.new + other.cache.threshold = 0.5 + base.merge!(other) + assert_in_delta 0.5, base.cache.threshold + end + + def test_merge_does_not_reset_group_settings + base = LlmOptimizer::Configuration.new + base.routing.simple_model = "my-model" + base.merge!(LlmOptimizer::Configuration.new) + assert_equal "my-model", base.routing.simple_model + end + + # Adapter resolution + + def test_resolved_adapter_nil_without_adapter_or_callers + assert_nil LlmOptimizer::Configuration.new.resolved_adapter + end + + def test_resolved_adapter_wraps_legacy_callers + config = LlmOptimizer::Configuration.new + config.llm_caller = ->(_p, **_k) { "hi" } + assert_instance_of LlmOptimizer::Adapters::Legacy, config.resolved_adapter + end + + def test_resolved_adapter_resolves_symbols + config = LlmOptimizer::Configuration.new + config.adapter = :anthropic + assert_instance_of LlmOptimizer::Adapters::Anthropic, config.resolved_adapter + end + + def test_resolved_adapter_returns_instance_untouched + adapter = LlmOptimizer::Adapters::Custom.new(chat: ->(_m, model:) { model }) + config = LlmOptimizer::Configuration.new + config.adapter = adapter + assert_same adapter, config.resolved_adapter + end + + # Validation + + def test_configure_raises_on_invalid_route_mode + assert_raises(LlmOptimizer::ConfigurationError) do + LlmOptimizer.configure { |c| c.route_to = :fastest } + end + end + + def test_configure_raises_on_out_of_range_threshold + assert_raises(LlmOptimizer::ConfigurationError) do + LlmOptimizer.configure { |c| c.similarity_threshold = 1.5 } + end + end + + def test_validate_raises_without_adapter + err = assert_raises(LlmOptimizer::ConfigurationError) { LlmOptimizer.validate! } + assert_includes err.message, "adapter" + end + + def test_validate_raises_when_cache_enabled_without_redis_url + config = LlmOptimizer::Configuration.new + config.llm_caller = ->(_p, **_k) { "hi" } + config.embedding_caller = ->(_t) { [1.0] } + config.cache.enabled = true + err = assert_raises(LlmOptimizer::ConfigurationError) { config.validate! } + assert_includes err.message, "redis_url" + end + + def test_validate_raises_when_cache_enabled_without_embedding_support + config = LlmOptimizer::Configuration.new + config.llm_caller = ->(_p, **_k) { "hi" } + config.redis_url = "redis://localhost:6379" + config.cache.enabled = true + err = assert_raises(LlmOptimizer::ConfigurationError) { config.validate! } + assert_includes err.message, "embedding" + end + + def test_validate_passes_with_complete_cache_config + config = LlmOptimizer::Configuration.new + config.llm_caller = ->(_p, **_k) { "hi" } + config.embedding_caller = ->(_t) { [1.0] } + config.redis_url = "redis://localhost:6379" + config.cache.enabled = true + assert config.validate! + end end diff --git a/test/unit/test_model_router.rb b/test/unit/test_model_router.rb index 5a07245..17ee037 100644 --- a/test/unit/test_model_router.rb +++ b/test/unit/test_model_router.rb @@ -161,4 +161,33 @@ def test_classifier_returns_complex r = LlmOptimizer::ModelRouter.new(cfg) assert_equal :complex, r.route("Fix this bug") end + + def test_classifier_model_routes_through_adapter + used_model = nil + cfg = LlmOptimizer::Configuration.new + cfg.adapter = LlmOptimizer::Adapters::Custom.new( + chat: lambda { |_messages, model:| + used_model = model + "complex" + } + ) + cfg.routing.classifier_model = "cheap-model" + r = LlmOptimizer::ModelRouter.new(cfg) + assert_equal :complex, r.route("Fix this bug") + assert_equal "cheap-model", used_model + end + + def test_no_classifier_model_skips_adapter_and_uses_heuristic + called = false + cfg = LlmOptimizer::Configuration.new + cfg.adapter = LlmOptimizer::Adapters::Custom.new( + chat: lambda { |_messages, model:| + called = true + model + } + ) + r = LlmOptimizer::ModelRouter.new(cfg) + assert_equal :simple, r.route("What time is it?") + refute called, "adapter should not be called without routing.classifier_model" + end end