Find the research gaps the literature hasn't connected yet.
Give Aporia a topic. It fetches a paper corpus, embeds and clusters it, builds the citation graph, and surfaces the under-explored space between clusters — pairs of related research areas that few papers have actually bridged, ranked by how promising the gap looks.
Aporia is a full research-gap-discovery pipeline wrapped in an institutional-looking web app. Given a topic, it:
- Fetches a paper corpus from Semantic Scholar (with an arXiv fallback), cached locally so repeat runs never re-hit the API.
- Embeds each paper with SPECTER2 (the proximity-tuned scientific-paper encoder), caching every embedding per-paper so a paper is only ever encoded once.
- Filters off-topic papers out by semantic similarity to the topic itself.
- Clusters the corpus (UMAP → HDBSCAN) into research sub-areas, each labelled with TF-IDF top terms plus a human-readable Gemini-generated name.
- Builds a directed citation graph across the corpus.
- Scores every pair of clusters for "gap-ness" — citation-sparse but semantically related pairs that also show open-problem language and a publication-time lag.
- Validates the signal with an honest GraphSAGE link-prediction GNN (gated on held-out AUC — it returns nothing rather than a fake number when it can't learn).
- Explains the top gaps with a Gemini-generated research question, testable hypothesis, and suggested method.
The result is a ranked list of research gaps, an interactive citation graph, and per-cluster publication-growth charts — all rendered in a dark "Ink Observatory" editorial theme (Cormorant Garamond + Inter).
- No API keys required to run the core pipeline. Semantic Scholar's public API and arXiv both work unauthenticated. Gemini features (advice, hypotheses, cluster names) are optional and degrade gracefully when no key is set.
- Cache-first. Fetched corpora and computed embeddings persist locally, so re-running a topic is near-instant and never rate-limited.
- Honest signals. Every score is normalized, explained in-app, and — for the GNN — refuses to report a number it can't back up.
The public deployment runs Showcase mode only. Live queries are disabled there. To run a real, live query, run the project locally (see Installation).
The live deployment (Render backend + Vercel frontend) serves Showcase mode by
default — real, pre-saved results from previously completed pipeline runs
(data/showcase/*.json), rendered through the exact same UI as a live run, with no wait
and no compute.
Live queries (POST /api/query, the Live query tab) are turned off on the public
deployment because the full pipeline loads SPECTER2, UMAP/HDBSCAN, and a GraphSAGE GNN
into memory simultaneously during a run, which does not fit a free-tier RAM budget —
running it there would very likely crash the whole backend. This is a deliberate choice
to keep the demo free rather than pay for a larger instance. The gate is
ENABLE_LIVE_QUERY (see Environment variables); against the
public URL, POST /api/query returns 503 by design.
Aporia's frontend is a six-module app (top nav). Every module renders through the same shared results components (gap list, citation graph, growth chart), so a showcase fixture, a live run, and a re-opened past run all look and behave identically.
| Module | What it does |
|---|---|
| Showcase | Pre-saved results from real completed runs — instant, no live compute. The public default. |
| Live query | Submit a topic, watch the pipeline run stage-by-stage, get ranked gaps. (Local only.) |
| Search | Semantic (cosine) search across every paper ever cached, across all topics. |
| Compare | Honest cross-topic semantic overlap between two already-run topics, cluster-by-cluster. |
| History | Re-open any past run's full results into the shared graph / growth / gap views. |
| How it works | A scrollytelling walkthrough of the whole pipeline, stage by stage. |
Plus a header System dialog (cache/job stats, clear the paper cache) and a light/dark theme toggle.
aporia
├── core/ # the pipeline, one package per stage
│ ├── ingestion/ # Semantic Scholar + arXiv clients, corpus cache
│ │ ├── semantic_scholar.py
│ │ ├── arxiv_client.py
│ │ └── corpus_cache.py # cache-first SQLite store (kills repeat 429s)
│ ├── embedding/ # SPECTER2 encoder + LanceDB vector cache
│ │ ├── specter_encoder.py
│ │ └── vector_store.py
│ ├── clustering/ # UMAP reduce → HDBSCAN → TF-IDF labels
│ │ ├── umap_reducer.py
│ │ ├── hdbscan_clusterer.py
│ │ └── cluster_labeler.py
│ ├── graph/ # directed citation graph + density metrics
│ ├── gap_detection/ # the four gap signals + the combiner
│ │ ├── citation_sparsity.py
│ │ ├── semantic_proximity.py
│ │ ├── temporal_lag.py
│ │ ├── future_work_miner.py
│ │ └── gap_scorer.py
│ ├── gnn/ # GraphSAGE link-prediction (impact scores)
│ ├── analysis/ # cross-topic comparison (Compare module)
│ ├── validation/ # offline backtest (gap signal vs. real citations)
│ └── llm/ # Gemini: advice, hypotheses, cluster names, topic fix
│
├── backend/ # FastAPI app wrapping the pipeline
│ ├── main.py # app + CORS
│ ├── routes.py # HTTP endpoints
│ ├── pipeline_runner.py # runs the full pipeline as an async job
│ ├── db.py # SQLite-backed job tracking
│ └── schemas.py # Pydantic request/response models
│
├── frontend/ # Vite + React + TypeScript
│ ├── public/
│ │ └── readme/
│ └── src/
│ ├── api/client.ts # the ONLY module that talks HTTP
│ ├── hooks/ # useJobPoller, useGraphData
│ ├── components/ # GapList, GraphViewer, GrowthChart, CompareModule, …
│ └── types/api.ts # mirrors backend/schemas.py
│
├── scripts/ # numbered CLI pipeline stages (01_…07_)
├── tests/ # pytest suite for the signal math
├── docker/ # backend + frontend Dockerfiles
├── data/ # gitignored, except data/showcase/*.json fixtures
├── render.yaml # Render deploy blueprint
├── requirements.txt
└── docker-compose.yml
Data flow:
Semantic Scholar / arXiv
│ (cache-first)
▼
corpus cache ──► SPECTER2 embeddings (LanceDB cache)
│
▼
relevance filter
│
▼
UMAP → HDBSCAN clusters ──► citation graph
│ │
└──────────┬─────────────┘
▼
four gap signals → gap_scorer → ranked gaps
│ │
▼ ▼
GraphSAGE impact score Gemini hypotheses
│
▼
FastAPI job API ──► React frontend
The pipeline can be run two ways: as the numbered CLI scripts in scripts/ (one
stage at a time, reading/writing data/{slug}/), or as a single async HTTP job through
the FastAPI backend. Both share the same embedding and corpus caches, so work done one
way is reused by the other.
- Python 3.11+
- Node.js 18+
- ~2–4 GB disk for model weights (SPECTER2) and per-topic caches
- The first live run downloads SPECTER2 (~440 MB) from Hugging Face; it's cached afterward.
git clone https://github.com/yourusername/aporia.git
cd aporiapip install -r requirements.txt
pip install torch==2.12.1 --index-url https://download.pytorch.org/whl/cpu
python -m spacy download en_core_web_sm
torchneeds the CPU wheel index explicitly (Aporia is CPU-only — no CUDA anywhere). spaCy'sen_core_web_smmodel is a separate download, not a pip dependency; the future-work miner fails at import without it.
cd frontend
npm install
cd ..Backend (starts the HTTP API):
uvicorn backend.main:app --reloadFrontend (separate terminal):
cd frontend
npm run dev # proxies /api/* to http://127.0.0.1:8000Open http://localhost:5173.
Or bring up both services together with Docker:
docker-compose up --buildEither through the Live query tab in the UI, or over HTTP:
curl -X POST http://127.0.0.1:8000/api/query -H "Content-Type: application/json" \
-d '{"topic": "Adversarial Robustness in Deep Learning", "limit": 200}'
# -> {"job_id": "<uuid>"}
curl http://127.0.0.1:8000/api/status/<uuid> # poll until "done" or "failed"
curl http://127.0.0.1:8000/api/results/<uuid> # 202 while running, else the resultThe first run of a topic fetches and embeds the corpus (a minute or two, depending on
limit); re-runs of the same topic are served almost entirely from cache.
A "gap" in Aporia is a relationship between two clusters — never a single paper or a single cluster. For every pair of clusters that's semantically related enough to be worth comparing, four independent, individually-normalized signals are combined into one score:
| Signal | What it measures |
|---|---|
| Citation sparsity | How few citations actually cross between the two clusters (1 − cross-cluster density). The core gap signal. |
| Semantic proximity | How related the two clusters are, by cosine similarity of their (mean-centered) SPECTER2 centroids. |
| Future-work density | Fraction of papers on both sides whose abstracts contain "open problem" / "under-explored" language (regex-mined, verbatim). |
| Temporal lag | How far apart the two clusters sit in average publication year. |
| Size asymmetry | How lopsided the two clusters are in paper count. |
Combined with fixed, non-tunable weights:
gap_score = 0.40 × (citation_sparsity × semantic_similarity) # the core gap signal
+ 0.25 × future_work_density # direct "open problem" evidence
+ 0.20 × temporal_lag
+ 0.15 × size_asymmetry
Pairs below a semantic-similarity floor are dropped entirely — a "gap" is only a meaningful claim between areas that are related enough for "there's a gap between these two" to make sense.
On top of the heuristic score, a small GraphSAGE link-prediction GNN trains on the citation graph and estimates each gap's link probability. It's deliberately honest: it returns a score only if it clears a held-out AUC gate and the graph has enough edges to learn from — otherwise it reports "not available for this corpus" rather than a fabricated number (an arXiv-only corpus, which carries no citation data, always gets this). The two are shown separately and never blended.
Every one of these numbers is explained inline in the app (an "info" tooltip on each tile, a "How this is calculated" expander under each gap), so a non-specialist can see exactly what drove a score.
The default landing view. Pick a pre-saved topic from the dropdown and its full results — ranked gaps, citation graph, growth charts — load instantly with no live compute. These fixtures are genuine, unmodified output from real completed pipeline runs, so what a visitor sees is exactly what the live pipeline produces.
The core output: research gaps ranked by gap_score, each card naming the two clusters
it bridges. The list is filterable — click a cluster in the citation graph and the gap
list narrows to gaps involving that cluster.
Each gap card shows the composite score as an animated meter, then breaks it into its contributing signals as a grid of stat tiles (each with its own tooltip and progress bar). The GNN impact score sits in its own clearly-labelled "experimental" block. A collapsible "Raw signals" section exposes the exact numbers and each cluster's representative papers (linked out to source). A Get advice button calls Gemini for a short, evidence-grounded suggestion, and the whole gap can be exported as a Markdown brief.
For the top-ranked gaps, Aporia asks Gemini for a concrete research question, a testable hypothesis, and a suggested method — rendered as a distinct "Suggested research direction" block, visually separate from the on-demand advice button. Without a Gemini key, this degrades to a documented "not available" state rather than erroring.
An interactive, force-directed citation graph (Sigma.js + graphology). Nodes are papers, colored by cluster and sized by citation in-degree, so influential papers read as bigger. A legend maps each color to its cluster label. Click any node to filter the gap list to that cluster. An arXiv-only corpus (no citation data) is flagged with a caution banner, since its graph-derived signals aren't meaningful.
Per-cluster publication counts over time (Recharts), for the largest clusters, so you can
see which sub-areas are emerging, stable, or declining. Cluster trend classification
(emerging / stable / declining) is computed for every cluster and surfaced on its
card.
Submit a topic and watch the job move through its stages — fetching → embedding → clustering → graph → scoring → gnn → done — polled live, then rendered into the same
results views as showcase. (Disabled on the public deployment; run locally.)
A global semantic search box: type a query and get the most cosine-similar papers across every topic ever cached — a different axis from any single run's results.
Pick two already-run topics and Aporia compares their clusters by embedding similarity only, using a shared joint-mean frame so the numbers reflect real overlap (not the near-constant ~0.9 that raw SPECTER2 cosine produces). Deliberately no cross-topic citation score — citations are in-set per topic, so that would be degenerate.
A two-column view: a list of every past completed run on the left, its full results on the right. Re-open any run into the shared graph / growth / gap views without re-running anything.
A centered scrollytelling narrative with a sticky pipeline stepper that lights up each of the eight stages as you scroll, plus a worked example thread. The in-app version of this README's pipeline explanation.
A header System dialog shows cache stats (corpus cache size, embedding count),
completed-job count, and completed-topic list, and can clear the paper cache. A
light/dark theme toggle (persisted in localStorage) switches between the dark "Ink
Observatory" and light "Parchment Journal" themes.
Fetched corpora are cached permanently in a local SQLite store
(data/global/corpus_cache.db), so re-running the same topic is served straight from
disk with no Semantic Scholar call — this is what avoids HTTP 429 rate-limiting on
repeat runs. The cache never expires on its own.
- Force a fresh fetch: send
{"refresh": true}toPOST /api/query, or pass--refreshtoscripts/01_fetch_papers.py. - Clear it: the System dialog's "Clear paper cache", or
POST /api/system/cache/clear.
Embeddings are cached separately and per-paper (LanceDB, data/global/embeddings.lance/),
keyed by (paper_id, model_version) — so a paper that appears in two topics is only ever
encoded once, and a fully-cached re-run never even loads the SPECTER2 model.
The backend serves a REST API under http://localhost:8000/api. Interactive docs
(Swagger UI) are at http://localhost:8000/docs while the server runs.
| Method & path | Purpose |
|---|---|
POST /api/query |
Submit a live pipeline job → { job_id }. Body: { topic, limit, refresh? }. |
GET /api/status/{job_id} |
Poll job status (pending…done/failed). |
GET /api/results/{job_id} |
Full results once done (202 while running). |
GET /api/jobs |
Paginated list of past jobs (limit, offset). |
GET /api/showcase |
List pre-saved showcase fixtures (instant, no DB). |
GET /api/showcase/{slug} |
One fixture's full results. |
GET /api/search?q=&limit= |
Cosine search across every cached paper. |
GET /api/compare?topic_a=&topic_b= |
Cross-topic cluster overlap. |
GET /api/gaps/{job_id}/{gap_id}/advice |
Gemini-backed suggestion for one gap. |
GET /api/system/stats |
Cache / job / embedding stats. |
POST /api/system/cache/clear |
Clear the corpus cache (never jobs or embeddings). |
GET /api/health |
Health check. |
Validation: topic must be non-empty and ≤200 chars; limit must be in [50, 800].
Violations return 422 with a typed ErrorResponse. Only one pipeline job runs at a
time — a second concurrent POST /api/query gets 429.
| Variable | Where | Purpose |
|---|---|---|
SEMANTIC_SCHOLAR_API_KEY |
backend | Optional. Raises Semantic Scholar's rate limit for ingestion. |
GEMINI_API_KEY |
backend | Optional. Enables gap advice, gap hypotheses, topic normalization, and human-readable cluster labels. Each degrades to a documented fallback if unset. Numbered keys (GEMINI_API_KEY_1..N) enable round-robin rotation across quotas. |
ENABLE_LIVE_QUERY |
backend | Optional, defaults on. Set false to disable POST /api/query (the public-deploy showcase-only gate). |
ALLOWED_ORIGINS |
backend | Comma-separated CORS allowlist. Falls back to localhost dev origins if unset — must be set to the live frontend URL in production. |
DATABASE_URL |
backend | Optional. Overrides the default local SQLite job store. |
VITE_API_BASE_URL |
frontend (build-time) | Backend origin for a deployed frontend build, e.g. https://aporia-backend.onrender.com. Leave unset for local dev/docker (uses the relative /api proxy). |
See .env.example for the full annotated list.
Pure signal-math unit tests (no network / torch / spaCy) live in tests/ and run fast:
pip install pytest
pytest- Backend → Render, as a Docker service (
docker/backend.Dockerfile, configured byrender.yaml), health-checked at/api/health.render.yamlsetsENABLE_LIVE_QUERY=falsefor the showcase-only public default. - Frontend → Vercel, native Vite build (no Docker), Root Directory
frontend/, withVITE_API_BASE_URLset to the live Render origin at build time.
Because the public deployment is showcase-only, it needs no GPU, no live-pipeline RAM,
and no paid tier — the pre-saved data/showcase/*.json fixtures are committed to the repo
and served straight off disk.
data/showcase/*.json are genuine, unmodified result_json blobs from real completed
runs. After a change to the scoring/GNN math, regenerate them so the public demo reflects
current behavior: run a live job locally for each showcase topic and save its result
(GET /api/results/{job_id}) to data/showcase/{slug}.json.
Pipeline: SPECTER2 (via transformers + adapters) · UMAP · HDBSCAN · NetworkX ·
PyTorch + PyTorch Geometric (GraphSAGE) · spaCy · LanceDB · Google Gemini
Backend: Python · FastAPI · Uvicorn · SQLAlchemy · SQLite · httpx
Frontend: React 19 · TypeScript · Vite · Sigma.js + graphology · Recharts · Radix UI · Tailwind · Motion · Cormorant Garamond + Inter