channelscrape

A small framework for pulling product data from e-commerce channels. You write one adapter per site; you get typed results, a polite HTTP client, and an availability monitor for free. Adding a channel is one file.

pip install -e .
channelscrape channels
channelscrape search books-sandbox "light" --limit 3

It ships with a working adapter for books.toscrape.com, a site built for scraping practice, so you can run it end to end in ten seconds without agreeing to anyone's terms. Real marketplaces are yours to add, using the template.

What it does

• One adapter per channel. Implement search() and product(), return Product objects. The base class handles fetching, retries, and rate limits.
• Polite by default. The client reads robots.txt and honours it, waits between requests, retries with backoff, and identifies itself honestly. You can plug in your own proxy through config.
• Typed results. Product, Offer, Availability dataclasses, so a price or a buy-box flag means the same thing on every channel.
• Availability monitor. Give it a list of products and it reports price and stock changes since the last run.

Add a channel

Copy channelscrape/adapters/marketplace_template.py, set a channel slug, implement the two parse methods, and decorate the class with @register. Keep parsing in static methods (html in, list out) so you can unit-test it against saved HTML with no network, the way tests/test_books_adapter.py does.

from channelscrape import get_adapter

books = get_adapter("books-sandbox")
for p in books.search("light"):
    print(p.title, p.best_price())

Monitor prices and stock

Make a file with one channel<space>url per line, then:

channelscrape monitor watch.txt --state state.json

First run records a baseline. Later runs print what changed.

How it works

The PoliteClient in base.py caches one robots.txt parser per host and refuses a fetch the site disallows. It keeps a minimum delay between requests and retries transient HTTP errors with a widening pause. Adapters never talk to the network directly; they call self.client.get(url) and parse the HTML. That split is why the parsing is easy to test and the fetching stays considerate.

Please scrape responsibly

• Read a site's Terms of Service and robots.txt before you point an adapter at it. Some sites do not allow scraping. Do not disable the robots check to get around a site that has said no.
• Prefer an official API when one exists.
• Rate-limit, identify yourself, and only collect data you have a lawful reason to use. Do not use this to defeat bot protection or to gather personal data.

FAQ

Why a framework instead of a finished scraper for site X?

Sites change, and each has its own terms. A clean adapter interface plus a polite client is the part worth sharing. The per-site parsing is a small file you own.

Does it run JavaScript-heavy pages?

Not out of the box. PoliteClient fetches HTML. For a site that renders on the client, write an adapter that calls that site's JSON endpoints, or swap in a browser-backed fetch in your adapter.

How do I add proxies or change the delay?

Pass a ClientConfig to PoliteClient(config=...): set min_delay, proxy, user_agent, or respect_robots there.

License

MIT. See LICENSE.