RAGEval: Offline RAG Evaluation Toolkit for Faithfulness, Retrieval Metrics, and Hallucination Checks

Keywords: RAG evaluation, LLM evaluation, retrieval metrics, faithfulness, hallucination checking, context precision, context recall, answer relevance, groundedness, RAGAS alternative

Point RAGEval at a set of (question, retrieved_contexts, answer) records. It returns a scorecard: faithfulness, context precision, context recall, answer relevance, and a freshness signal. It also names the weakest step in the pipeline.

What is RAGEval?

RAGEval is a small Python toolkit for RAG evaluation. You already have questions, the chunks a retriever returned, and the answers a model generated. RAGEval scores those triples. It does not retrieve, and it does not generate. It tells you whether the retriever missed evidence, whether the generator invented claims, and whether the contexts are stale.

Piece	Role
Deterministic metrics	Token and n-gram overlap. No API key. This is the offline scorecard.
LLMJudge	Optional Anthropic tool-use loop (claude-opus-5) for faithfulness and answer relevance. Falls back to the overlap proxies with no key.
evaluate	Runs selected metrics, aggregates means, names the weakest metric.
CLI	rageval run and rageval demo print the scorecard and write a JSON report.

Direct answer

How do I evaluate a RAG pipeline without an API key? Write one JSONL row per example with question, retrieved_contexts, answer, and optional ground_truth. Run rageval run records.jsonl. RAGEval scores faithfulness (answer vs contexts), context precision and recall, answer relevance (question keywords in the answer), and freshness from context timestamps. No embeddings, no network.

What does a low faithfulness score mean? The answer uses tokens and n-grams that do not appear in the retrieved chunks. That is a groundedness / hallucination signal: the generator is adding claims the context does not support. The optional judge asks the same question at claim level and still degrades to this proxy if the key is missing.

Where is the pipeline weak? The scorecard prints Weakest: plus a one-line diagnosis. Low context_recall is a retriever miss. Low context_precision is unrelated chunks. Low faithfulness is hallucination. Low answer_relevance is an off-topic answer. Low freshness is a stale corpus; re-rank with freshness-aware-rag or recrawl.

Why RAGEval?

Most RAG eval stacks start with an LLM judge and an embedding model. That is useful, and it is also slow, billed, and hard to run in CI. RAGEval starts the other way: a pure overlap scorecard that a unit test can pin, then an opt-in judge for the two judgment metrics.

	Embedding + LLM eval suites (RAGAS and similar)	RAGEval
Needs a key for the usual path	Yes	No
Faithfulness	LLM-as-judge	Token/n-gram overlap; optional judge
Context precision / recall	LLM and/or embeddings	Question-keyword coverage / gold-token recall
Answer relevance	LLM or embeddings	Question keyword coverage; optional judge
Freshness / recency	Not a built-in	Exponential decay from context timestamps
CI without network	Extra work	Default
Adding a metric	Framework plugin	One file and @register

Tradeoffs: overlap cannot see paraphrases. "Paris" and "the French capital" will not match. If you need semantic similarity, use an embedding suite or turn on --judge. RAGEval will not replace a full LLM eval board. It will give you a number you can recompute offline.

How it works

Everything below is what the code does, not a wish list.

1. Input. An EvalRecord is a question, a list of retrieved contexts, a generated answer, and optional gold text. Contexts may be strings or {text, timestamp, source} objects. Load them from JSONL, a JSON array, or CSV (rageval.io.load_records).
2. Faithfulness. Unigram precision of the answer against the concatenated contexts, mixed with bigram precision (weights 0.6 / 0.4). Empty answer scores 1.0 (no unsupported claims). Non-empty answer with empty retrieval scores 0.0.
3. Context precision. For each chunk, the fraction of question content keywords found in that chunk (stopwords dropped). Extra words in a long on-topic chunk do not hurt. The record score is the mean of those chunk scores. The reason line also reports hit-rate at hit_threshold (default 0.1). Gold text is not used here.
4. Context recall. Unigram recall of the gold tokens in the concatenated contexts. Skipped when ground_truth is missing.
5. Answer relevance. Set recall of question content tokens (stopwords dropped) in the answer. "What is the capital of France?" is scored on capital and france.
6. Freshness. For each dated chunk, exp(-age_hours / half_life_hours) with default half-life 168 hours. Mean over dated chunks. Skipped when no timestamps are present. Same decay idea as freshness-aware-rag.
7. Aggregate. evaluate builds a Scorecard: per-record results, per-metric means (skipped records excluded), an unweighted overall, and weakest plus a diagnosis string.
8. Optional judge. If ANTHROPIC_API_KEY is set and you pass --judge (or use_judge=True), LLMJudge starts a tool-use loop (get_record, get_proxy_scores, submit_judgment) on claude-opus-5. A refusal stop reason, a missing SDK, a missing key, --no-judge, or any exception falls back to the overlap proxies.

records.jsonl
    │
    ▼
tokenize ──► faithfulness, precision, recall, relevance
timestamps ─► freshness
    │
    ▼
optional LLMJudge (faithfulness + answer relevance)
    │
    ▼
Scorecard (per record, means, weakest, JSON report)

Install

Python 3.10 or newer.

git clone https://github.com/pandeyvishwas51-oss/rageval.git
cd rageval
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest -q

Runtime dependency: anthropic (imported only if you set a key). pydantic is an optional extra (pip install -e ".[pydantic]"); the toolkit itself uses dataclasses. Deterministic metrics, the demo, and --no-judge do not call the network.

Quick start

Sample demo (no keys)

rageval demo
rageval --now 2026-08-21T12:00:00Z --out report.json demo
rageval --json --metrics faithfulness,freshness run examples/sample.jsonl

examples/sample.jsonl covers a grounded Paris fact, a Nile hallucination, unrelated retrieval for photosynthesis, a Mars recall miss, a 2018 library report (stale), an off-topic Mars answer, and a 2026 library opening (fresh).

Python, metrics only

from datetime import datetime, timezone
from rageval import EvalRecord, evaluate, EvalConfig

now = datetime(2026, 8, 21, 12, 0, tzinfo=timezone.utc)
records = [
    EvalRecord(
        id="q1",
        question="What is the capital of France?",
        retrieved_contexts=["Paris is the capital and most populous city of France."],
        answer="Paris is the capital of France.",
        ground_truth="Paris is the capital of France.",
    )
]
card = evaluate(records, config=EvalConfig(now=now))
print(card.means)
print(card.weakest, card.diagnosis)
print(card.format_text())

One metric

from rageval import faithfulness, EvalRecord

rec = EvalRecord(
    question="How long is the Nile?",
    retrieved_contexts=["Many sources give the Nile a length of about 6650 kilometres."],
    answer="The Nile is 12000 kilometres long and flows through Spain.",
)
print(faithfulness(rec).score, faithfulness(rec).reason)

Optional judge

from rageval import LLMJudge, evaluate

card = evaluate(records, use_judge=True)  # degrades without a key
# or
results = LLMJudge(force_proxy=True).judge(records[0])

Set ANTHROPIC_API_KEY if you want the model loop. Use --no-judge (the default) on the CLI to stay on proxies even when a key is present.

JSONL shape

{
  "id": "q1",
  "question": "What is the capital of France?",
  "retrieved_contexts": [
    {"text": "Paris is the capital of France.", "timestamp": "2026-08-20T09:00:00Z"}
  ],
  "answer": "Paris is the capital of France.",
  "ground_truth": "Paris is the capital of France."
}

CSV columns: id,question,answer,ground_truth,retrieved_contexts,timestamps. Separate multiple chunks with |||.

Add a metric

Create one module under src/rageval/metrics/, decorate a pure function, import it from metrics/__init__.py:

from rageval.metrics.base import register
from rageval.models import EvalConfig, EvalRecord, MetricResult

@register("my_metric")
def my_metric(record: EvalRecord, config: EvalConfig | None = None) -> MetricResult:
    return MetricResult(name="my_metric", score=1.0, reason="stub")

FAQ

What Python versions are supported?

Python 3.10+. Tests run on 3.10, 3.11, 3.12, and 3.13 in CI.

Do I need an Anthropic key?

No. The overlap metrics, freshness, sample demo, and JSON report run without one. The key is only for the optional tool-use loop.

Which model does the optional loop use?

claude-opus-5, constant rageval.DEFAULT_MODEL. A refusal stop reason is treated as a fallback to the overlap proxies, not as a crash.

How is faithfulness different from answer relevance?

Faithfulness asks whether the answer is supported by the retrieved chunks (groundedness / hallucination). Answer relevance asks whether the answer addresses the question. An answer can be faithful and still off-topic, or on-topic and still invented.

What if I have no ground truth?

context_recall is skipped. context_precision scores chunks against the question instead. Faithfulness, answer relevance, and freshness do not need gold.

Why did freshness skip a record?

None of the retrieved chunks had a timestamp. Pass ISO-8601 strings or unix seconds on each context. Undated chunks in a mixed record are ignored; dated ones still score.

Does overlap catch paraphrases?

No. That is the point of the optional judge. The offline path is a lower bound you can run in CI.

Are tests offline?

Yes. CI unsets ANTHROPIC_API_KEY. The Anthropic layer is mocked or skipped. Hypothesis checks that every metric stays inside [0, 1] or is skipped.

How do I run tests?

pip install -e ".[dev]"
pytest -q

License

MIT. Free for commercial and personal use.

Related: freshness-aware-rag re-ranks retrieved chunks by recency and source authority before you generate. RAGEval scores the result after you generate.