arXiv Semantic Search

A semantic search engine over arXiv papers, with a vector index built from scratch — hand-written k-means clustering and IVF (inverted-file) approximate nearest-neighbour search — benchmarked against the Qdrant vector database.

Type a plain-English question and get back the most relevant papers, ranked by meaning rather than keywords.

Demo

📄 Read the full technical writeup →

Headline result

My from-scratch IVF index reaches 88% recall at ~7× the speed of brute-force search (nprobe=3, 1000 papers), and is competitive with Qdrant at this scale.

How it works

1. Embed — each paper's abstract is turned into a 384-dimensional vector using all-MiniLM-L6-v2.
2. Index — vectors are grouped into clusters with a hand-written k-means; IVF search only scans the clusters nearest the query, skipping most of the data.
3. Search — a query is embedded and compared (cosine similarity) against candidate vectors; the closest are returned.
4. Serve — a FastAPI backend exposes a /search endpoint; a small web page provides the search box.

Benchmark: brute force vs IVF vs Qdrant

Measured on 1000 arXiv abstracts, 30 clusters, top-10 retrieval:

Method	ms/search	Recall	Speedup
Brute force	13.4	1.00	1.0×
IVF (nprobe=1)	0.7	0.52	20.5×
IVF (nprobe=3)	2.0	0.88	6.9×
IVF (nprobe=10)	5.3	0.96	2.5×
Qdrant	3.5	1.00	3.8×

The nprobe parameter trades speed for accuracy: searching more clusters raises recall but costs time. The knee of the curve sits around nprobe=3. I validated correctness by confirming my search returns identical results to Qdrant.

Running it

# 1. Install dependencies
pip install -r requirements.txt

# 2. Build the embeddings index (see Data note below)
python src/build_index_file.py

# 3. Run the search API
uvicorn src.api:app --reload

# 4. Open web/index.html in your browser

Data: the arXiv metadata dump is available on Kaggle. It is not committed to the repo due to size.

What I learned

• How embeddings turn text into vectors where semantic similarity becomes geometric proximity.
• Why brute-force search doesn't scale, and how approximate nearest-neighbour indexes trade a little recall for large speedups.
• How to measure retrieval quality (recall@k) and validate a from-scratch implementation against a production tool.

Tech

Python · sentence-transformers · NumPy · DuckDB · FastAPI · Qdrant · matplotlib