Books Web Scraper

A Scrapy-based ETL project that extracts book data from books.toscrape.com and loads it into MongoDB with duplicate prevention.

Project Overview

This project follows an ETL flow:

1. Extract: scrape title, price, URL, and image URL.
2. Transform: normalize URLs and prepare item payloads.
3. Load: store in MongoDB through a Scrapy pipeline.

The spider handles pagination automatically and the pipeline avoids duplicate inserts using a SHA256 hash of each book URL.

Features

1. Scrapes all pages from books.toscrape.com.
2. Extracts title, price, product URL, and product image URL.
3. Supports pagination with next-page traversal.
4. Stores data in MongoDB.
5. Drops duplicate books using deterministic URL hashing.
6. Uses environment variables for MongoDB configuration.

Project Structure

Web-Scraper/
|-- .env
|-- README.md
|-- info.txt
|-- steps.txt
|-- task.txt
|-- data_extracted.json
`-- books/
   |-- scrapy.cfg
   `-- books/
      |-- __init__.py
      |-- items.py
      |-- middlewares.py
      |-- pipelines.py
      |-- settings.py
      `-- spiders/
        |-- __init__.py
        `-- book.py

Tech Stack

1. Python 3
2. Scrapy
3. MongoDB
4. PyMongo
5. python-dotenv

Setup

1. Create and activate virtual environment

python -m venv venv
venv\Scripts\activate

2. Install dependencies

pip install scrapy pymongo python-dotenv

3. Create .env at project root

Place .env in the root folder (same level as README.md), not inside the books folder.

Example:

MONGO_URI="mongodb+srv://username:password@cluster.mongodb.net/?appName=MyApp"
MONGO_DATABASE="books_db"

Why .env path is written this way

In settings.py, .env is loaded with a file-based path:

BASE_DIR = Path(__file__).resolve().parents[2]
load_dotenv(dotenv_path=BASE_DIR / ".env")

This avoids current-working-directory bugs. The settings file is inside books/books/settings.py, so parents[2] resolves to the Web-Scraper root where .env exists.

Configuration

Mongo settings are read in books/books/settings.py:

MONGO_URI = os.getenv("MONGO_URI")
MONGO_DATABASE = os.getenv("MONGO_DATABASE")

If either value is missing, startup fails fast with a clear ValueError.

Pipeline registration:

ITEM_PIPELINES = {
  "books.pipelines.MongoPipeline": 300,
}

Run the spider

From the books directory:

cd books
scrapy crawl book

Output shape

Sample stored document:

{
  "_id": "sha256_hash_of_url",
  "title": "Book Title",
  "price": "PS51.77",
  "url": "https://books.toscrape.com/catalogue/book_1/index.html",
  "img": "https://books.toscrape.com/media/cache/...jpg"
}

Scrapy Components

items.py

Defines item fields used by the spider and pipeline.

spiders/book.py

1. Starts from books.toscrape.com.
2. Extracts item data using CSS selectors.
3. Uses response.urljoin() for absolute URLs.
4. Follows next-page links.

pipelines.py

MongoPipeline lifecycle methods:

1. from_crawler(): reads settings into pipeline.
2. open_spider(): creates MongoDB client and db handle.
3. process_item(): computes hash id, checks duplicates, inserts item.
4. close_spider(): closes MongoDB connection.

CSS selector notes used in project

1. article.product_pod: product card container.
2. h3 > a::attr(title): book title.
3. h3 > a::attr(href): relative product URL.
4. .price_color::text: price text.
5. .image_container > a > img::attr(src): image path.
6. li.next > a::attr(href): pagination link.

Helpful Scrapy response properties

1. response.url
2. response.status
3. response.headers
4. response.body
5. response.text

Setup Tips

1. Keep .env in the project root, alongside README.md.
2. Match the variable names in .env with the names read in settings.py.
3. Use file-based path resolution for configuration files so the project works from any terminal location.
4. Run the spider from the Scrapy project folder with the virtual environment activated.

Learning Workflow Used

1. Create virtual environment.
2. Activate environment.
3. Inspect target in Scrapy shell.
4. Define item model.
5. Generate spider.
6. Implement extraction and pagination.
7. Add Mongo pipeline.
8. Add duplicate prevention.
9. Run and verify data.

Author

Megh Patel 2026