feat(evaluator): add optional RAGAS evaluator

[piotrhm]

Description

Adds RAGAS as an optional, independent evaluator alongside the in-house LLM judge, exposing four RAG metrics (faithfulness, answer relevancy, context precision, context recall) that run through the pipeline's already-configured foundation and embedding models.

Motivation

The project only offered its in-house LLM-as-a-Judge metrics. RAGAS is a widely-used, standardized RAG evaluation library, and supporting it gives users an independent, comparable set of metrics without having to stand up a separate evaluation stack. Wrapping our own model abstractions means RAGAS reuses the endpoint the rest of the pipeline is already configured with instead of opening its own OpenAI/LangChain client.

Changes

• New RagasEvaluator ai4rag/evaluator/ragas_evaluator.py - scores RAG metrics via the RAGAS library and returns them in the shared EvaluationMetricsResult shape, with bootstrap confidence intervals computed the same way as the other evaluators. Failed/slow samples yield NaN → None instead of aborting the whole pattern evaluation.
• RAGAS adapters ai4rag/evaluator/ragas_adapters.py thin BaseRagasLLM / BaseRagasEmbeddings wrappers delegating to BaseFoundationModel.chat and BaseEmbeddingModel. All ragas/langchain imports are local so importing ai4rag.evaluator never requires the optional extra.
• New metrics ai4rag/evaluator/metric.py - RAGAS_FAITHFULNESS, RAGAS_ANSWER_RELEVANCY, RAGAS_CONTEXT_PRECISION, RAGAS_CONTEXT_RECALL, plus a ragas evaluator literal.
• Wiring — run_rag_optimization gains a ragas_enabled: bool = False flag that appends a RagasEvaluator; AI4RAGExperiment adds the four RAGAS metrics to its default set when a ragas evaluator is present; RagasEvaluator exported from ai4rag.evaluator.
• Dependencies — new optional ragas extra: ragas>=0.3,<0.5 with langchain-community<0.4.2 (pinned because ragas 0.4.3 imports a Vertex AI module langchain-community 0.4.2 removed).
• Tuned run config — RunConfig(timeout=1200, max_workers=4) to avoid overwhelming small evaluating models and to prevent spurious thread-pool TimeoutErrors.

Testing

Added tests/unit/ai4rag/evaluator/test_ragas_evaluator.py covering the evaluator.

[piotrhm]

Related pipelines-components PR -> opendatahub-io/pipelines-components#204

[piotrhm]

RAG Evaluation Metrics: Legacy vs. RAGAS

The experiment

We ran the documents-rag-optimization-pipeline across 7 benchmark datasets
spanning three corpora families and multiple formats/sizes:

Corpus	Datasets
Enterprise RAG-Bench	github-50, confluence-50
Open RAG-Bench (arXiv)	arxiv-10-txt, arxiv-100-txt, arxiv-100-md
HTML-RAG NQ	nq-val-10, nq-val-50

For each dataset the pipeline performed HPO over the RAG design space (chunking, embeddings,
retrieval, generation), producing 10 candidate patterns per dataset = 70 configurations total.
Every configuration was scored by two independent metric families:

• Legacy metrics (current selection criteria): faithfulness, answer correctness,
  context correctness, overall score.
• RAGAS metrics (proposed addition): faithfulness, answer relevancy, context precision,
  context recall.

We then measured how well the two families agree — per dataset (Spearman rank correlation,
n=10 each) and pooled (n=70) — to determine whether RAGAS is redundant (measures the same
thing) or complementary (adds new signal).

---

Summary

The legacy and RAGAS metric families do not agree, and where they diverge the legacy metrics
are the less trustworthy of the two. Adding RAGAS is warranted — it supplies discriminating
signal precisely where the current metrics fail.

Three findings drive the recommendation:

1. Low, unstable correlation → the metrics are complementary, not redundant.
   Same-named pairs (e.g. legacy faithfulness vs. RAGAS faithfulness) correlate only
   +0.37 pooled, and swing from +0.85 to −0.18 across datasets. If RAGAS merely restated
   the existing metrics, correlations would be uniformly high. They are not — RAGAS captures
   information the legacy metrics miss.

2. The legacy "context correctness" metric is unreliable. It is saturated at a constant
   1.0 on arxiv-10-txt (zero variance — it cannot rank anything) and near-ceiling on the other
   open/NQ datasets. On the enterprise data where it does vary, it inversely correlates with
   RAGAS context precision on github-50 (ρ = −0.84). A metric that is either flat or backwards
   cannot be a sound optimization target. RAGAS context precision/recall provide the variance
   needed to actually discriminate retrieval quality.

3. The choice of metric family changes which RAG configuration "wins" on most datasets.
   Because the families rank patterns differently, optimizing on legacy scores alone risks
   selecting a suboptimal RAG design. A second, independent measurement reduces that risk.

---

Best configuration per dataset (actual scores)

The best pattern per dataset (selected by legacy overall score), with both metric
families side by side. Legacy context correctness (L.ctxCorr) is pinned at 1.000
on every non-enterprise dataset — a visible symptom of the saturation problem.

dataset	best pattern	L.overall	L.faith	L.ansCorr	L.ctxCorr	R.faith	R.ansRel	R.ctxPrec	R.ctxRecall
arxiv-10-txt	Pattern7	0.839	0.711	0.462	1.000	0.850	0.931	0.915	1.000
arxiv-100-txt	Pattern9	0.835	0.751	0.633	1.000	0.755	0.870	0.841	0.994
arxiv-100-md	Pattern9	0.828	0.718	0.641	1.000	0.753	0.866	0.850	0.971
nq-val-10	Pattern1	0.782	0.715	0.465	1.000	0.867	0.863	0.675	0.891
nq-val-50	Pattern3	0.825	0.668	0.781	1.000	0.883	0.703	0.767	0.975
confluence-50	Pattern7	0.742	0.570	0.481	0.828	0.764	0.719	0.854	0.979
github-50	Pattern1	0.690	0.533	0.433	0.704	0.684	0.626	0.896	0.951

L. = legacy metric, R. = RAGAS metric. All winning configs share bge-m3 embeddings,
redhataillama-31-8b-instruct generation, and recursive chunking.

Legacy overall vs. RAGAS faithfulness for the best pattern of each dataset. On arxiv-100-txt/md
the two families disagree by ~7–8 points in opposite directions; the apparent "winner" depends on
which family you trust.

---

Supporting data

Agreement between "matched" metric pairs (Spearman ρ, n=10 per dataset)

dataset	faith ↔ ragas_faith	ctx_corr ↔ ragas_ctx_prec	overall ↔ ragas_faith
arxiv-10-txt	+0.25	n/a (flat)	+0.50
arxiv-100-txt	−0.04	+0.75	+0.67
arxiv-100-md	+0.49	+0.57	+0.78
nq-val-10	−0.18	+0.70	+0.13
nq-val-50	+0.85	+0.79	+0.89
confluence-50	+0.70	+0.41	+0.71
github-50	+0.22	−0.84	+0.49
Pooled (n=70)	+0.37	+0.16	+0.56

If the two families were interchangeable, every cell would be ≈ +0.9. Instead they range from
strongly positive to negative — evidence of complementary signal, not duplication.

The legacy context metric has no discriminating power on most datasets

Legacy context correctness across all 10 patterns per dataset. It sits at the 1.0 ceiling on the
open/NQ datasets (no ability to rank) and only spreads out on confluence-50 and github-50.

dataset	distinct legacy context_correctness values (10 patterns)
arxiv-10-txt	[1.0] — constant, cannot rank
arxiv-100-txt	[0.92, 0.96, 1.0] — ceiling-bound
arxiv-100-md	[0.94, 0.98, 1.0] — ceiling-bound
nq-val-10	[0.9, 1.0] — near-binary
confluence-50	[0.70 … 0.85] — healthy spread
github-50	[0.68 … 0.77] — healthy spread

The legacy context metric only varies meaningfully on the two enterprise datasets — and that is
exactly where it disagrees with RAGAS (github ρ = −0.84). Everywhere else it is saturated and blind.

Most reliable cross-family signal

overall ↔ RAGAS faithfulness is positive on all 7 datasets (+0.13 to +0.89, pooled +0.56) —
RAGAS faithfulness is the single best independent corroborator of the current aggregate score,
making it a natural first RAGAS metric to adopt.

---

Recommendation

Adopt RAGAS as a parallel, complementary evaluation family — not a replacement. Concretely:

• Add RAGAS faithfulness + answer relevancy as reported metrics immediately (relevancy is the
  strongest driver of the current overall, ρ up to +0.87).
• Replace or supplement legacy "context correctness" with RAGAS context precision/recall, given
  the legacy metric's ceiling saturation and sign inversion.
• Flag configurations where the two families disagree for human review before promoting a
  "best" RAG pattern.

---

Basis: 7 datasets × 10 patterns = 70 configurations, from the latest 10-pattern
documents-rag-optimization-pipeline run per dataset (enterprise runs completed 2026-08-10).
Correlations are Spearman rank, computed within-dataset to remove cross-corpus difficulty confounds.

[jakub-walaszczyk]

Overall very strong PR. There are some suggestions, that seem to address LLM-generated content. Once we address them we may ship it with coming v0.12.0.

2 additional comments:

• you should also introduce documentation update
• make sure docstrings are in place even for private methods (this is for developers in the first place)

[jakub-walaszczyk]

Minor comments to resolve and we are getting closer and closer to the merge. The solution for milvus returning empty contexts will be merged soon 👀