An end-to-end research paper recommender system for the NLP domain — scrape → quality-gate → annotate → embed → recommend → evaluate → serve, built as a reproducible Moon monorepo and deployed on Kubernetes.
Given a paper (or a free-text query) ArXivFlow returns the most relevant papers from a curated corpus, combining dense semantic embeddings, lexical (BM25) retrieval, citation-graph overlap, and NER entity overlap — fused with Reciprocal Rank Fusion and evaluated honestly against two independent ground truths.
arXiv API + Semantic Scholar API (crucible)
→ Kubernetes Jobs scrape into a shared PVC
→ Data-quality gate (crucible) 1000 → 992 clean papers
→ Label Studio + model-assisted NER (annotate + lsml)
→ Vector embeddings in ChromaDB (papervec) 4 encoder models × 992 vectors
→ Recommender: dense + BM25 + citation + entity, RRF fusion (compass)
→ Evaluation: tag-overlap + citation-overlap proxies + LLM-judge gold (compass)
→ Perturbation / robustness analysis (compass)
→ FastAPI backend + Next.js frontend (api + client)
- Scrape 1,000 NLP papers (arXiv
cs.CL/ ACM I.2.7 / MSC 68T50) from the ArXiv API and enrich them with influential-citation counts and reference graphs from the Semantic Scholar API. - Quality-gate the raw dataset (completeness, uniqueness, consistency, validity) down to a clean 992-paper corpus before anything is annotated or embedded.
- Annotate abstracts with 7-label Named Entity Recognition in Label Studio, assisted by a custom ML backend and cross-checked against GLiNER, phi-4, Qwen2.5-14B and Claude Opus 4.8.
- Embed every paper with domain-specific scientific encoders (SPECTER2, SciNCL, BGE-large, Qwen3-8B) into ChromaDB (HNSW, cosine).
- Recommend with a multi-signal recommender fused by Reciprocal Rank Fusion, and evaluate it against two independent proxy ground truths plus an LLM-judge gold set.
- Serve the corpus, similarity neighbours and a 2D embedding atlas through a FastAPI backend and a Next.js frontend.
The workspace is orchestrated by Moon (task runner + project graph), with tool
versions pinned by proto and Python dependencies managed by uv. Each package /
app owns its own moon.yml, pyproject.toml and Dockerfile.tera.
| Project | Kind | Role |
|---|---|---|
crucible |
package | Scrape arXiv + Semantic Scholar, enrich metadata, pre-annotation QC |
annotate |
package | NER annotation (Label Studio / GLiNER / LM) + inter-annotator agreement |
lsml |
package | Label Studio ML backend serving GLiNER annotation suggestions |
papervec |
package | Generate SPECTER2 (+ alt encoders) embeddings into ChromaDB |
compass |
package | Framework-agnostic recommender logic, evaluation & robustness harness |
api |
app | FastAPI backend — listings, cosine neighbours, embedding atlas |
client |
app | Next.js (App Router) + SCSS frontend |
Run any task from the workspace root:
moon run <project>:<task> # e.g. moon run crucible:dockerize, moon run compass:evaluate.
├── apps/
│ ├── api/ # FastAPI backend (serves listings + SPECTER2 neighbours + atlas)
│ └── client/ # Next.js + SCSS frontend
├── packages/
│ ├── crucible/ # scraping + enrichment + pre-annotation QC
│ ├── annotate/ # NER annotation + IAA
│ ├── lsml/ # Label Studio ML backend (GLiNER)
│ ├── papervec/ # SPECTER2 / alt-encoder embeddings → ChromaDB
│ └── compass/ # recommender + evaluation + perturbation/robustness
├── data/ # shared artifacts (raw dataset, annotations, embeddings, eval reports)
├── k8s/ # cluster-wide manifests (PVC + PVC inspector)
├── scripts/ # workspace helpers
├── .moon/ # Moon workspace + toolchain config
├── .prototools # pinned tool versions (moon, proto, uv)
├── PLAN.md # work-package plan / lab notebook
├── SOURCES.md # presentation fact sheet (every headline number, sourced from the repo)
└── README.md # this fileShared artifacts live in data/ and flow one way between packages — crucible writes the cleaned
dataset, papervec writes the embeddings, and compass / api only ever read them.
All heavy workloads run as Jobs / Deployments on the cluster, mounting a shared
arxivflow-pvc (5Gi, ReadWriteMany) so each stage hands artifacts to the next through the PVC.
GPU workloads target NVIDIA L4 nodes (24GB, Ada Lovelace) via nodeSelector.
kubectl apply -f k8s/pvc.yml # shared 5Gi RWX volume
kubectl apply -f k8s/inspect-pvc.yml # long-lived inspector pod for kubectl cp in/outPer-package cluster usage (image build, secrets, jobs, teardown) is documented in each package README.
Images are published to Docker Hub under dashaikh10/.