Scientific Method Mining from arXiv

This project builds an end-to-end NLP pipeline to automatically identify and rank scientific methods that are reused across research domains such as chemistry, materials science, and AI.

The system mines arXiv abstracts, canonicalizes method mentions using embeddings, and produces a curated, scored list of real cross-domain methods.

---

Motivation

Scientific ideas often migrate across disciplines, but tracking this manually is slow and subjective. This project explores whether NLP and embedding-based canonicalization can automatically surface which established methods are reused across domains.

---

Pipeline Overview

1. Data ingestion

   • Fetches recent arXiv abstracts via the arXiv API
   • Domains: chemistry, materials science, AI/ML

2. Method extraction

   • Extracts candidate method mentions from titles and abstracts
   • Aggressive filtering to remove prose, measurements, and noise

3. Canonicalization

   • Uses SBERT embeddings to cluster method variants
   • Collapses spelling and naming differences into canonical methods

4. Method filtering

   • Rule-based ontology to retain only real scientific and ML methods
   • Removes hardware terms, observables, datasets, and section artifacts

5. Scoring

   • Ranks methods by:
     • Cross-domain reuse
     • Domain-specific importance (specificity prior)
   • Temporal weighting is minimal due to recent-data bias

6. Visualization

   • Streamlit dashboard for interactive exploration

---

Output

The final output is a ranked list of real scientific methods: