Search Engine

A full-stack search engine project built with a C++ ranking/query core, Python indexing pipeline + backend API, and a TypeScript frontend.

🚀 Overview

This project implements a custom search pipeline:

• Preprocess and clean a large article dataset
• Build lexicon, forward index, inverted index, and barrels
• Serve search queries through a backend API
• Display results in a web frontend
• Use a C++ query processor for efficient ranking

🧱 Tech Stack

• C++ (core query processing/ranking)
• Python (data pipeline + backend service)
• TypeScript + Vite (frontend)

Language composition (GitHub):

• C++: 96.3%
• Python: 2.5%
• TypeScript: 0.9%
• C: 0.2%
• JavaScript: 0.1%

📁 Project Structure

• src/ → indexing scripts + C++ query components
  • lexicon.py
  • forward_index.py
  • inverted_index.py
  • barrel.py
  • clean_data.py
  • ProcessQuery.cpp
• backend/ → backend API (app.py) and integration logic
• Frontend/ → frontend application (Vite + TypeScript)

📦 Dataset Requirement

Add the dataset file:

• data.csv (Kaggle, ~190k+ articles)

Place it in the expected data directory used by your pipeline (e.g., engine_data/).

⚙️ Setup & Run

1) Build indexing data

From the src/ directory, run scripts in this order:

1. lexicon.py
2. forward_index.py
3. inverted_index.py
4. barrel.py
5. clean_data.py

Run order is important because each step depends on outputs from previous steps.

2) Run backend

From the backend/ directory:

python app.py

3) Run frontend

From the Frontend/ directory:

npm install
npm run dev

✅ Notes

• Keep file/folder names exactly as expected by scripts.
• Ensure Python dependencies and Node.js are installed.
• If query processing is configured to use a compiled C++ binary, make sure ProcessQuery.exe exists or compile from ProcessQuery.cpp for your platform.

📌 Future Improvements

• Add one-command setup script for full pipeline bootstrap
• Add Docker support for frontend/backend
• Add CI checks and unit tests
• Add deployment instructions