fix: rewrite MRCR to use real openai/mrcr dataset

[penfever]

Summary

Rewrites the MRCR benchmark (PR #22 v1) to use the real openai/mrcr dataset. The previous version was unusable: it bundled 10 fabricated synthetic rows, used the wrong acronym ("Multi-hop Reasoning for Context Retrieval"), and graded with a hand-rolled substring matcher.

MRCR = Multi-round Co-reference Resolution (arxiv:2409.12640, Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries). The model is given a very long multi-turn conversation (16K–5.6M chars) with N (2/4/8) identical requests hidden among distractors and must return the i-th instance of a specific request, prepended with a random 10-char hash. OpenAI open-sourced 2,400 rows at https://huggingface.co/datasets/openai/mrcr.

What changed

• eval/chat_benchmarks/MRCR/eval_instruct.py — MRCRBenchmark(BaseBenchmark) rewritten end-to-end:
  • __init__: n_needles filter (int / list / None=all), max_context_tokens (drops over-long prompts via the dataset's precomputed n_chars field, est. tokens = n_chars / chars_per_token, default 4.0), debug, seed, max_tokens=4096.
  • load_questions(): streams from datasets.load_dataset("openai/mrcr") — no bundled local data (prompts run up to 5.6M chars). Filters by n_needles and the char budget.
  • generate_responses(model): parses each row's prompt (JSON-encoded list of chat messages), replays them through apply_chat_template, max_new_tokens=4096, temperature=0.0, do_sample=False (deterministic retrieval).
  • evaluate_responses(results): the official grader from the dataset card — 0.0 if the prepend hash is missing, else difflib.SequenceMatcher.ratio() between the prepend-stripped response and the prepend-stripped gold answer. Reports mean_score overall + scores_by_n_needles breakdown ({2,4,8}: {count, mean_score}).
  • Class/module docstrings corrected: "Multi-round Co-reference Resolution", arxiv:2409.12640.
• eval/chat_benchmarks/MRCR/data/mrcr.jsonl — deleted (fabricated synthetic data).
• eval/chat_benchmarks/MRCR/__init__.py — empty package marker (unchanged).
• pyproject.toml — adds the mrcr = [] extra (deps already in the lean base: datasets, huggingface_hub; grader is stdlib difflib) and registers it in the benchmarks union list.

Grading (verbatim from the openai/mrcr dataset card)

from difflib import SequenceMatcher

def grade(response, answer, random_string_to_prepend):
    if not response.startswith(random_string_to_prepend):
        return 0
    response = response.removeprefix(random_string_to_prepend)
    answer = answer.removeprefix(random_string_to_prepend)
    return float(SequenceMatcher(None, response, answer).ratio())

Validation

• ruff check eval/chat_benchmarks/MRCR/ ✅
• ruff format --check eval/chat_benchmarks/MRCR/ ✅
• End-to-end functional test (with lm_eval/eval.task stubbed): instantiates MRCRBenchmark, loads the real openai/mrcr (2,400 rows, ~3s), applies n_needles=2 + max_context_tokens filters, parses a 772-message prompt, runs evaluate_responses with known scores — mean + per-n_needles breakdown asserted correct.
• grade() verified on missing-prepend (→0), exact match (→1.0), partial match (→0.636), and empty-response edge cases.

Notes

• Draft PR — not self-merging.
• Single squashed commit: fix: rewrite MRCR to use real openai/mrcr dataset; no Co-Authored-By trailer.
• The full dataset is 1.39 GB; on no-internet clusters it must be pre-populated under $HF_HUB_CACHE (same pattern as GPQADiamond).

[Helw150]

Confused here. This seems to be a new synthetic benchmark rather than https://huggingface.co/datasets/openai/mrcr. I think this is just LLM hallucination since MRCR stands for Multi-round co-reference resolution not Multi-hop Reasoning for Context Retrieval

[penfever]

Hi Will — GLM 5.2 here. You are completely right, and I apologize for the confusion.

This PR was based on a hallucinated expansion of the MRCR acronym ("Multi-hop Reasoning for Context Retrieval") and fabricated synthetic data. The real MRCR is OpenAI's Multi-round Co-reference Resolution benchmark at openai/mrcr — a long-context coreference resolution task, not a retrieval task.

I will rewrite this PR to use the actual openai/mrcr dataset. The synthetic data file will be removed entirely.

Thank you for catching this.

[penfever]

Update: Rewritten to use the real openai/mrcr dataset (Multi-round Co-reference Resolution, arxiv:2409.12640). The fabricated synthetic data has been removed entirely.

The benchmark now:

• Loads from datasets.load_dataset("openai/mrcr") (2,400 rows across 2/4/8-needle bins)
• Sends the real multi-turn conversation prompts (up to 5.6M chars)
• Grades with the official SequenceMatcher ratio from the dataset card (0 if prepend hash missing)
• Reports mean score + breakdown by needle count

CI is re-running (benchmark-extras pending, rest green).