Skip to content

README.md

Latest commit

 

History

History
104 lines (78 loc) · 4.32 KB

README.md

File metadata and controls

104 lines (78 loc) · 4.32 KB
Raw

Benchmark Data

Language / 语言: English | 中文

The benchmark in this directory is based on SciReplicate-Bench (SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers, arXiv:2504.00255) — 36 NLP papers from 2024, 100 algorithmic reproduction tasks.

What is / is not versioned

To keep the repository lightweight, only minimal inputs are version-controlled. The original paper PDFs and codebases must be obtained separately.

File Description Versioned
Data.json Full task definitions: task descriptions, latex_code, reference implementations, dependency/API annotations, test cases (test_case_code) ✅
download_papers.py Script for downloading paper PDFs/text and cloning repositories ✅
<paper>/paper.txt Full paper text (parsed from PDF, UTF-8) ✅
<paper>/metadata.json Paper metadata: title / arxiv_url / github / project_path ✅
<paper>/paper.pdf Paper PDF ❌ fetch separately
<paper>/codebase/ Original paper codebase ❌ fetch separately

Pipeline intermediates (step1~4*, graph*.json/html, communities.json, combined_graph.json, etc.) and venv/ environments are also not versioned — they are regenerated at runtime.

Directory layout

data/
  Data.json
  download_papers.py
  <paper_id>-<safe_title>/
    paper.txt        ← versioned
    metadata.json    ← versioned
    paper.pdf        ← fetch separately
    codebase/        ← fetch separately (original repo)

<paper_id> is a two-digit number (00–35) matching the paper_id field in Data.json / metadata.json.

How to obtain paper PDFs and codebases

Each paper's download URLs are already recorded in its metadata.json:

  • arxiv_url — source for the paper PDF
  • github — original codebase repository

Option A: from SciReplicate-Bench (recommended)

The official SciReplicate-Bench repository ships all papers and codebases under its Benchmark/ directory. After cloning it, copy each paper's codebase to <paper>/codebase/ and its PDF to <paper>/paper.pdf.

Option B: fetch per paper from metadata

For any single paper:

cd data/<paper_dir>
# Clone the codebase (github field)
git clone --depth 1 \
  "$(python3 -c "import json; print(json.load(open('metadata.json'))['github'])")" \
  codebase
# Download the PDF (arxiv_url field)
curl -L -o paper.pdf \
  "$(python3 -c "import json; u=json.load(open('metadata.json'))['arxiv_url']; print(u if u.endswith('.pdf') else u.replace('/abs/','/pdf/')+'.pdf')")"

Note: download_papers.py was used to batch-build the dataset and requires an external papers_summary.json manifest. For re-fetching individual papers, Option B above is more straightforward — all URLs are available in metadata.json or Data.json.

Data.json schema

Data.json is a list; each element represents one paper:

Field Description
paper_id Paper index
paper_details {title, url}
repo_original_url Original codebase URL
project_path / latex_code_path Relative paths within SciReplicate-Bench
task_details[] List of tasks for this paper

Each task_details entry:

Field Description
task_id Task index
latex_code Algorithm snippet from the paper (LaTeX)
Description Natural-language task description
completion_path / namespace / ori_python_file File and namespace of the target function
signature_position / body_position Line ranges of function signature / body
ReferenceCode_With_Comments Reference implementation with inline comments
Arguments / Return / dependency / external_APIs Interface and dependency annotations
test_case_code Evaluation test case: {insert_pos, indent, code}

Citation

If you use this benchmark, please cite the original work:

@article{scireplicate2025,
  title   = {SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers},
  journal = {arXiv preprint arXiv:2504.00255},
  year    = {2025},
  url     = {https://github.com/xyzCS/SciReplicate-Bench}
}