Backend Crawler

Overview

FlexiCrawl is a simple site and file crawler, storing discovered pages and relationships in PostgreSQL. It exposes an API that can be consumed by the front-end. It's operation supports pause/resume of processing info. This is work in progress.

The system is split into:

• Crawler engine
• Database layer
• API layer
• State management
• Data query layer

---

Support This Project

If you find this extension useful, you can support development via PayPal: Thank you for helping keep this project maintained and improving! written by: Mark Samios @enigma1

--

Installation

mkdir <FlexiCrawl> folder
cd <FlexiCrawl> folder
git clone https://github.com/enigma1/FlexiCrawl.git .
npm i
cp .env.template .env

Install the dependencies: python -m pip install -r requirements.txt

To start the application: uvicorn main:app --reload

Architecture

API Layer

The API layer exposes routes for:

• Starting a crawl
• Pausing and resuming a crawl
• Restarting a crawl
• Removing/resetting tables
• Fetching crawler data

The API layer is responsible for:

• Validating incoming requests
• Resolving the target domain
• Selecting the correct database table
• Converting API request models into database query inputs
• Converting database results into API response models

---

Crawler Lifecycle

Start

A crawl starts through the crawler start route.

The start process:

1. Receives a domain
2. Creates the required database tables
3. Inserts the root URL
4. Starts the crawler loop
5. Updates crawler state

The crawler runs independently from the HTTP request so API routes remain responsive.

---

Pause

The pause operation:

1. Updates crawler state to request a pause
2. The crawler loop checks the state
3. The current work completes safely
4. The crawler signals that it is idle
5. The API returns once the crawler is paused

---

Resume

Resume starts a new crawler execution after a pause.

The previous crawler task is not reused. A completed paused task has finished its execution, so a new task/thread is created for the next crawl cycle.

---

Restart

Restart performs:

1. Pause the current crawler if running
2. Wait until the crawler becomes idle
3. Reset database tables
4. Restore the root URL
5. Reset crawler counters
6. Start a new crawler execution

---

Database Layer

The database layer uses PostgreSQL.

Responsibilities:

• Creating crawler tables
• Removing/resetting schemas
• Inserting discovered URLs
• Storing page data
• Storing page relationships
• Executing generic table data queries

---

Data Fetching

The data fetching layer provides generic table querying.

Supported features:

• Dynamic table selection
• Pagination
• Multiple column sorting
• Column metadata
• Row retrieval

The query layer receives:

{
    "table": "...",
    "paging": {
        "offset": 0,
        "limit": 50
    },
    "sort_by": [
        {
            "column": "title",
            "direction": "ASC"
        }
    ]
}

🧾 License

GNU General Public License (GPL) v3