Skip to content

README.md

Latest commit

 

History

History
246 lines (175 loc) · 9.32 KB

README.md

File metadata and controls

246 lines (175 loc) · 9.32 KB
Raw

PGSPD

Paper-Guided Semantic Program Decomposition — automatically aligns a research paper with its codebase and extracts self-contained, independently importable module code packages.

Given a paper (paper.txt) and the corresponding repository (codebase/), PGSPD:

  1. Extracts a structured algorithmic module specification from the paper
  2. Builds a multi-source call graph over the codebase and clusters it
  3. Locates each paper module to a concrete entry function and extracts its dependency closure via BFS on the graph
  4. Materialises the closure files deterministically into a mirror directory structure ready for independent import

Language / 语言: English | 中文

Table of Contents

Pipeline Overview

paper.txt + codebase/
    │
    ├─ Step 1 ──→ Paper module spec extraction (LLM structured parsing)
    │             Output: step1_spec.json
    │
    ├─ Step 2 ──→ Code graph construction (PyCG + Astroid + pyan3 + AST + Leiden clustering)
    │             Output: graph.json / communities.json / graph.html
    │
    ├─ Step 3 ──→ Module localisation + closure extraction
    │             (OpenHands Agent locates entry → BFS extracts dependency closure)
    │             Output: step3_matches.json / step3_alignment.json / step3/*.md / graph_alignment.html
    │
    └─ Step 4 ──→ Closure materialisation (deterministic, no LLM:
                  graph-closure files + import fixpoint → mirror directory structure)
                  Output: <task_id>/ (mirrored source) / <task_id>/closure_meta.json

Step 5 (assembly & end-to-end verification) is planned but not yet enabled in this release.

Requirements

  • Python >= 3.11
  • openhands: required for Step 3 to drive the Agent that locates module entry points on the graph + source
  • pyan3 / astroid: Step 2 graph-building dependencies (installed via pip install -e .)

Step 4 is purely deterministic file I/O — no virtual environment creation, no tests, no LLM/Agent calls, no extra runtime dependencies.

Installation

From the project root:

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -U pip
pip install -e .

Optional development dependencies:

pip install -e .[dev]

LLM Configuration

PGSPD reads LLM settings from environment variables (prefix PGSPD_) and supports any OpenAI-compatible endpoint:

export PGSPD_LLM_BASE_URL="https://your-openai-compatible-endpoint/v1"
export PGSPD_LLM_API_KEY="your_api_key"
export PGSPD_LLM_MODEL="deepseek-v3"

These can also be overridden temporarily via CLI flags (see Usage).

Additional configurable options:

Environment variable Default Description
PGSPD_EXPLANATION_LANGUAGE English Language for LLM-generated module explanations (English, Chinese / 中文, …)
PGSPD_MATCH_TOP_K 20 Number of candidate nodes passed to the LLM in Step 3
PGSPD_STEP3_SEARCH_TOP_K 20 Max results returned by the search_graph tool in Step 3

Data Directory Convention

data/
  Data.json              ← Benchmark task definitions (ground-truth tasks + test cases)
  download_papers.py     ← Script for downloading papers / cloning repos
  <paper_name>/
    paper.txt            ← Full paper text (UTF-8 plain text)       [versioned in repo]
    metadata.json        ← Paper metadata (title / arXiv / GitHub)  [versioned in repo]
    paper.pdf            ← Paper PDF                                 [not versioned — fetch separately]
    codebase/            ← Original code repository                  [not versioned — fetch separately]

⚠️ To keep the repository lightweight, only the minimal inputs are versioned (paper.txt / metadata.json / Data.json). paper.pdf and codebase/ come from SciReplicate-Bench — see data/README.md for how to obtain them.

Usage

Option A: CLI (recommended)

After installation, the pgspd command is available:

pgspd data/pinn/paper.txt data/pinn/codebase --output data/pinn

Full options:

pgspd <paper_txt> <repo_path> \
  --output ./output_dir \    # output directory (default: ./pgspd_output)
  --step 4 \                 # stop after this step (1–4, default: 4)
  --model deepseek-v3 \      # LLM model name
  --base-url https://... \   # LLM API base URL
  --api-key your_key \       # LLM API key
  --verbose                  # verbose logging

Run only the first two steps (graph construction):

pgspd data/pinn/paper.txt data/pinn/codebase --output data/pinn --step 2

Option B: main.py batch runner

Edit the PAPERS list in main.py, then run:

python main.py

Each paper is processed in sequence following the data/<paper_name>/ convention.

Output Files

Using --output data/pinn as an example:

data/pinn/
  step1_spec.json              ← Paper module spec (PaperSpec JSON)
  step1_spec_zh.json           ← Chinese translation of module descriptions

  graph.json                   ← Code call graph (NetworkX node-link format)
  communities.json             ← Leiden community partition (with paper-module annotations)
  GRAPH_REPORT.md              ← Human-readable codebase structure summary
  graph.html                   ← D3.js interactive call graph visualisation ★

  step3_matches.json           ← Paper module ↔ code entry function alignment (with BFS closure nodes)
  step3_alignment.json         ← Alignment summary and diagnostics (including bfs_config)
  step3_diagnostics.json       ← Per-module match source and closure size diagnostics
  step3_subgraph_stats.json    ← Per-module closure file count / LOC / byte size
  step3/
    <ModuleName>.md            ← Detailed alignment explanation per module (LLM-generated)
  graph_alignment.html         ← Interactive module–node alignment visualisation ★

  <task_id>/                   ← Step 4 closure package (mirrors original repo structure)
    <relative/path>/*.py       ← Source files in the closure (including __init__.py completions)
    closure_meta.json          ← Closure manifest: per-node provenance
                                  (graph_edge / import_scan) and summary stats
  static_blindspots.jsonl      ← Graph BFS blind spots (deps recovered by import_scan)

  pipeline_result.json         ← Full pipeline run summary

Baseline (No-Graph)

nograph/ provides the ablation baseline (Task B): it skips the Step 2/3/4 graph-based pipeline (code graph + community guidance + BFS closure) and instead uses a single OpenHands Agent that reads source code directly, traces dependencies manually, and copies the relevant files. It reuses the step1_spec.json produced by Step 1 and its output format is identical to the graph-based Step 4, enabling a fair head-to-head comparison.

import asyncio, pathlib
from nograph.nograph_runner import run_nograph
from pgspd.config import PGSPDConfig

asyncio.run(run_nograph(
    spec_json=pathlib.Path("data/15-RAPTOR.../step1_spec.json"),
    repo_path=pathlib.Path("data/15-RAPTOR.../codebase"),
    output_dir=pathlib.Path("nograph_output"),
    task_id="15-5",
    config=PGSPDConfig(),
))

Benchmark Data

Evaluation uses SciReplicate-Bench (36 NLP papers from 2024, 100 algorithmic reproduction tasks).

  • Only lightweight inputs are versioned in this repo: data/Data.json (task / test-case definitions) plus each paper's paper.txt and metadata.json.
  • paper.pdf and codebase/ are not versioned — see data/README.md for how to fetch them from SciReplicate-Bench or the original arXiv / GitHub sources.

FAQ

Q: pyan3 not found
Make sure your virtual environment is activated and pip install -e . has completed successfully.

Q: Step 4 materialised files are incomplete / missing dependencies
Step 4 first materialises the BFS closure from Step 3, then runs an import fixpoint scan to recover any intra-repo dependencies the graph missed. Both sources are recorded in closure_meta.json under provenance; dependencies recovered solely by import_scan are also written to static_blindspots.jsonl for graph-coverage debugging.

Q: A module is unmatched in Step 3 (entry_fqn is empty)
The Agent could not locate an entry point for that module; MatchedModule.match_source will be unmatched and Step 4 will produce no closure for it. Check step3_workspace/diagnostics/agent_log.txt and step3_diagnostics.json to diagnose the cause.

Q: LLM API errors
Verify that PGSPD_LLM_BASE_URL, PGSPD_LLM_API_KEY, and PGSPD_LLM_MODEL match your provider's configuration.

Q: Using a different LLM (GPT-4o, Claude, local Ollama, …)
PGSPD uses an OpenAI-compatible interface — just set PGSPD_LLM_BASE_URL and PGSPD_LLM_MODEL to match your provider.

Version

Current version: 1.0.2