Release 0.12.0

[jakub-walaszczyk]

Release v0.12.0

Summary

This release makes ai4rag genuinely provider-agnostic. Model access is now built on a plain OpenAI-compatible client — the shipped integration targets OpenShift AI Models-as-a-Service (MaaS), which serves listing, chat, and embeddings from a single endpoint — replacing the OGX-specific stack entirely. The vector-store layer is rebuilt on direct backend clients for Chroma, Milvus, and PostgreSQL/pgvector, configured through typed config dataclasses and with proper connection lifecycle and concurrency handling. Evaluation gains an optional RAGAS evaluator alongside the existing LLM-as-a-Judge path, selectable per optimization run.

Changes

Added

• Vector store — direct backend clients for Chroma, Milvus, and PostgreSQL/pgvector (ChromaVectorStore, MilvusVectorStore, PGVectorStore), each selected via a typed, frozen config dataclass (ChromaConfig, MilvusConfig, PGVectorConfig) passed as a single vector_store_config.
• Vector store — reranker module implementing RRF and weighted fusion for hybrid search.
• Vector store — BaseVectorStore now supports close() and the context-manager protocol; each optimization trial scopes its store in a with block, so connections and pools are no longer leaked per trial.
• Vector store (pgvector) — connection pooling via psycopg_pool.ConnectionPool with a configurable PGVectorConfig.pool_max_size (default 10); AI4RAGExperiment sizes the pool from inference_max_threads so it tracks real query concurrency.
• Evaluator — optional RagasEvaluator (with RAGAS adapter classes AI4RAGRagasLLM / AI4RAGRagasEmbeddings) enabling RAGAS-based metrics; ragas is now a regular dependency.
• RAG optimization component — llm_judge_mode selector (base / ragas / all / none) on run_rag_optimization() to choose which LLM-as-a-Judge evaluators run.
• Evaluator — build_aggregate_metric() shared helper on BaseEvaluator for constructing aggregate metric payloads.
• Search space preparation — build_search_space_report() and serialize_model() in ai4rag.search_space.prepare, co-locating the model↔spec round-trip (serialize_model() is the write mirror of the model restore path).
• Model access — create_maas_client() and shared model discovery/restore helpers get_foundation_models() / get_embedding_models() in ai4rag.search_space.prepare.models, accepting either bare model ids (discovery) or serialized report specs (restore).
• Assets generator — get_vector_store_config() / get_vector_store_env_vars() factories that build a backend config from a provider discriminator and expose each backend's required environment variables for documentation.
• Dependencies — added openai as the model-access SDK (replacing ogx-client), plus chromadb, pymilvus, pgvector, and psycopg[binary,pool] for the direct vector-store clients.

Changed

• Model provider — replaced the OGX integration with any OpenAI-compatible endpoint; the shipped integration targets OpenShift AI Models-as-a-Service (MaaS), which serves listing, chat, and embeddings from a single endpoint.
• Vector store — get_vector_store() and AI4RAGExperiment now take a single vector_store_config and dispatch on config.provider, replacing the vector_store_type string plus the OGX vector_io provider id.
• Vector store — collection-name resolution centralized in BaseVectorStore, enforcing a mandatory ai4rag prefix as the cross-backend isolation guard.
• Vector store — hybrid-search reranking parameter renamed impact_factor → k.
• Search space — default vector_store_type changed from ogx to milvus; the default Chroma search space no longer includes the window retrieval method.
• Search space preparation — renamed prepare_search_space_with_ogx to prepare_search_space_with_maas, now accepting an openai.OpenAI client. Because MaaS models.list() carries no metadata (model type, embedding dimension, context length), the payload must declare foundation and embedding model IDs explicitly; embedding dimension and context length are auto-detected at construction time.
• Model ids — model ids are used verbatim, exactly as models.list() reports them (including any / characters); there is no more model-specific URL derivation or id stripping.
• Client factory — replaced create_ogx_client with create_maas_client, a single client that serves listing, chat, and embeddings for every model at the one MaaS endpoint.
• Notebook templates — renamed the generated ogx_{indexing,inference} templates to maas_{indexing,inference}, each building a single OpenAI client from MAAS_BASE_URL / MAAS_API_KEY and reusing it for every model; the inference notebook now also rebuilds the pattern's detected generation language and passes it to OpenAIFoundationModel, so answers keep the benchmark's language.
• Experiment / evaluator — metrics and optimization_metric now require RAGMetric instances selected from Metrics and reject bare metric-name strings, which are ambiguous now that a name (e.g. faithfulness) is shared across the unitxt and RAGAS evaluators.
• Model helpers — model-instantiation helpers moved to ai4rag.search_space.prepare.models, removing the components↔search_space coupling.
• Search space report — model pre-selection decoupled from report building into an explicit ModelsPreSelector step; SearchSpaceReport slimmed to the search-space dict and no longer carries selected_models or a per-model base_url, and pattern.json no longer carries base_url.
• Leaderboard — aggregate scores are keyed by a collision-free key (unitxt and custom metrics keep their bare name; other evaluators are prefixed, e.g. ragas_faithfulness), so colliding metric names each get their own column instead of overwriting one another.

Fixed

• Vector store (Milvus) — forced consistency_level="Strong" on vector/hybrid search so a query immediately following an add_documents() upsert can no longer race Milvus's default bounded-staleness read and return zero hits against a collection that does contain matching data.
• Vector store (pgvector) — corrected inner_product scoring: the <#> operator returns the negative inner product, so the score is now derived by negation (cosine/l2/l1 keep 1/dist), fixing an inverted ranking.
• Vector store (pgvector) — guarded lazy index creation with double-checked locking (plus a UniqueViolation fallback) so concurrent search threads no longer race on CREATE INDEX.
• Experiment — an optimization metric that is produced but unscored (None mean) is now recorded as a failed — not fatal — iteration; a genuinely absent metric still raises a RAGExperimentError with an evaluator-qualified message.
• Components — added vector_db_secret_name to the indexing pipeline params.
• Core — ensure_ascii=False when JSON-dumping documents that may reach the end user, preserving non-ASCII characters.
• Benchmark data — reject BenchmarkData records with zero correct answers, preventing a downstream unitxt TokenOverlap crash on max() of an empty iterable.
• Experiment — benchmark JSON is now read with an explicit UTF-8 encoding.

Removed

• OGX — removed all OGX support: the ogx-client dependency, OGXFoundationModel, OGXEmbeddingModel, OGXVectorStore, OGXModelParameters, OGXEmbeddingParams, create_ogx_client, the ogx_utils module, the ogx_inference_base_url helper, and the OGX_CLIENT_BASE_URL / OGX_CLIENT_API_KEY environment variables (replaced by MAAS_BASE_URL / MAAS_API_KEY).
• Assets generator — removed the OGX-only pattern_builder and prompt_filters modules and the build_pattern_json export; indexing-spec enrichment is now inlined.
• Search space preparation — prepare_search_space_report() and the search_space_preparation module removed from ai4rag.components.optimization; build a search space with prepare_search_space_with_maas(), then call build_search_space_report() from ai4rag.search_space.prepare.
• Experiment — EvaluationResult no longer carries a rag_pattern field; a trial's vector store is closed once the trial finishes, so read pattern_name / scores from EvaluationResult instead of calling .generate() on a previously returned pattern.
• Dependencies — removed langchain-chroma; Chroma is now used directly via chromadb.
• Samples — removed the outdated samples/run_ai4rag.ipynb notebook.

Migration notes

This is a breaking release. Upgrading from 0.11.x requires the following changes:

1. Environment variables — replace OGX_CLIENT_BASE_URL / OGX_CLIENT_API_KEY with MAAS_BASE_URL / MAAS_API_KEY. MAAS_BASE_URL must be the full OpenAI-compatible endpoint, used verbatim (e.g. https://<host>/v1).
2. Model access — replace create_ogx_client(...) with create_maas_client(base_url, api_key). All OGX model classes (OGXFoundationModel, OGXEmbeddingModel, and their parameter types) are gone; use OpenAIFoundationModel / OpenAIEmbeddingModel. Model ids are now used verbatim as reported by models.list() — do not strip path segments.
3. Search space preparation — replace prepare_search_space_with_ogx(...) with prepare_search_space_with_maas(...), passing an openai.OpenAI client and explicit foundation and embedding model IDs (MaaS models.list() returns no metadata). To produce a report, call build_search_space_report() from ai4rag.search_space.prepare — prepare_search_space_report() no longer exists in ai4rag.components.optimization. Model pre-selection is now a separate ModelsPreSelector step, and SearchSpaceReport no longer carries selected_models.
4. Vector store — replace the vector_store_type string (and any OGX vector_io provider id / client argument) with a single vector_store_config built from ChromaConfig, MilvusConfig, or PGVectorConfig. The ogx vector-store backend is removed and the default is now milvus. If you tune hybrid search, rename the reranking parameter impact_factor → k.
5. Results — EvaluationResult no longer exposes rag_pattern. Read pattern_name and scores from EvaluationResult instead of calling .generate() on a previously returned pattern; the trial's vector store is closed once the trial finishes.
6. Metrics API — pass RAGMetric instances selected from Metrics to metrics / optimization_metric; bare metric-name strings are no longer accepted, because a name such as faithfulness is now shared across the unitxt and RAGAS evaluators.
7. Dependencies — ogx-client and langchain-chroma are removed. The new runtime dependencies (openai, chromadb, pymilvus, pgvector, psycopg[binary,pool], ragas) are installed automatically; re-sync your environment (uv sync).

Checklist

• __version__ in ai4rag/__init__.py updated to 0.12.0
• docs/about/changelog.md updated
• All tests pass (pytest)
• Docs build successfully