ScrapeGuard: Python Scraper Health Monitor for Blocks, CAPTCHAs, and Selector Drift

Keywords: scraper health monitoring, web scraper monitoring python, crawler canary, scrape success rate, 403 block detection, captcha spike alert, selector drift, scrapy monitoring, playwright scraper health, sqlite scraper metrics, domain health score, web scraping alerts

Record each scrape attempt. ScrapeGuard turns those events into a per-domain score (0-100) and alerts when success drops, blocks or CAPTCHAs rise, selectors start failing, or latency crosses 8 seconds. In-process, zero dependencies, memory or a local SQLite file.

What is ScrapeGuard?

ScrapeGuard is a small Python library you call from an existing scraper. It does not fetch URLs. After each request you emit a ScrapeEvent. The monitor stores the event and later answers "is shop.example healthy right now?"

Piece	Role
ScrapeEvent	One attempt: domain, success, latency, status code, error_type, selector_ok
ScrapeGuard	Stores events in memory, or memory plus SQLite if you pass db_path
DomainHealth	Windowed counts, rates, a 0-100 health_score, and alert codes
CLI demo	python -m scrapeguard prints JSON for 20 synthetic shop.example events

Think "canary for crawlers," not another crawler.

Direct answer

How do I know a site started blocking my scraper? Call record() with error_type="block" (for example on HTTP 403). health(domain) returns block_rate and adds improved_blocks when more than 10% of events in the window are blocks. The default window is 24 hours.

What is selector drift, and can this library detect it? Selector drift is when page HTML changes and your CSS or XPath stops matching. Set selector_ok=False on those attempts. ScrapeGuard flags selector_drift when more than 15% of events in the window fail selectors. It does not inspect the DOM itself.

Is ScrapeGuard a web crawler or a Scrapy extension? No. It never opens sockets. You keep Scrapy, Playwright, httpx, or a raw requests loop. ScrapeGuard only records outcomes and scores domains. There is no plugin system and no dashboard.

Why ScrapeGuard?

	Homegrown log grep	Prometheus + Grafana	ScrapeGuard
Extra process	none	exporters, TSDB, Grafana	none
Scraper-specific alerts	you write them	you write recording rules	built in: blocks, CAPTCHA, selector, latency
Storage	files	time-series database	memory, or one SQLite file
Domain health score	DIY	DIY	0-100 formula in health()
Multi-host fleet	depends	designed for it	one process (or one shared db file)
Thresholds	yours	yours	hardcoded in health()

If you already run Prometheus, keep it. ScrapeGuard is for a single scraper or a small set of domains when you want a health score without standing up metrics infra. Alert cutoffs are not configurable except by editing health(). Connections are not locked for multi-writer access. There is no HTTP UI.

How it works

1. record(event) appends to an in-memory list. If db_path was set at init, it also INSERTs into a SQLite events table and commits.
2. health(domain, window) keeps events for that domain with ts >= now - window (default 24 hours, timestamps stored as UTC).
3. It computes success rate, average latency, block rate (error_type == "block"), captcha rate (error_type == "captcha"), and selector fail rate (selector_ok is false).
4. Score:

score  = 100 * success_rate
score -= 40 * block_rate
score -= 30 * captcha_rate
score -= 20 * selector_fail_rate
score -= 10  if avg_latency_ms > 5000
clamp to 0..100, round to 1 decimal

5. Alert codes, a domain can have several at once:

Code	When
no_data	zero events in the window
low_success_rate	success rate below 0.8
improved_blocks	block rate above 0.1
captcha_spike	captcha rate above 0.05
selector_drift	selector fail rate above 0.15
slow_responses	average latency above 8000 ms

error_type values timeout, selector, and other are stored but do not have their own rate columns. Failed attempts still lower success_rate. Selector failures are counted from selector_ok, not from error_type. The latency penalty on the score (5000 ms) and the slow_responses alert (8000 ms) use different cutoffs.

summary() returns health() for every domain that has at least one stored event.

Install

Python 3.11 or newer. No third-party runtime dependencies.

git clone https://github.com/pandeyvishwas51-oss/scrapeguard.git
cd scrapeguard
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest tests/ -v

Quick start

from scrapeguard import ScrapeGuard, ScrapeEvent

guard = ScrapeGuard()  # or ScrapeGuard(db_path="./scrapeguard.db")

guard.record(
    ScrapeEvent(
        domain="shop.example",
        success=False,
        latency_ms=900,
        status_code=403,
        error_type="block",
    )
)

h = guard.health("shop.example")
print(h.n, h.success_rate, h.health_score, h.alerts)
# 1 0.0 0.0 ['low_success_rate', 'improved_blocks']

Persist across process restarts with SQLite. The same file can be reopened later:

guard = ScrapeGuard(db_path="./scrapeguard.db")
print(guard.summary())  # {domain: DomainHealth, ...}

CLI demo

The module is a demo, not a production CLI. It records 20 synthetic events (every 5th request is a 403 block, every 7th fails the selector) and prints JSON:

python -m scrapeguard

Example shape:

{
  "domain": "shop.example",
  "n": 20,
  "success_rate": 0.8,
  "avg_latency_ms": 295.0,
  "block_rate": 0.2,
  "captcha_rate": 0.0,
  "selector_fail_rate": 0.15,
  "health_score": 69.0,
  "alerts": ["improved_blocks"]
}

Drop-in from a scraper

from scrapeguard import ScrapeEvent

def after_request(guard, domain, response, elapsed_ms, selector_matched):
    error = None
    if response.status_code == 403:
        error = "block"
    elif "captcha" in response.text.lower():
        error = "captcha"
    guard.record(
        ScrapeEvent(
            domain=domain,
            success=response.ok and selector_matched,
            latency_ms=elapsed_ms,
            status_code=response.status_code,
            error_type=error,
            selector_ok=selector_matched,
        )
    )

Wire that callback from Scrapy middleware, a Playwright page.goto wrapper, or a plain requests loop. ScrapeGuard never calls the network.

FAQ

What Python versions are supported?

Python 3.11+. Runtime dependencies: none. Tests use pytest via pip install -e ".[dev]".

Does this rotate proxies or solve CAPTCHAs?

No. It only records that a block or CAPTCHA happened so you can alert and stop digging through logs.

Can I change the 24-hour window or the alert cutoffs?

The window is an argument: health(domain, window=timedelta(hours=6)). Alert cutoffs and the score weights are hardcoded in ScrapeGuard.health. Edit that method (or subclass) if you need different numbers.

Is the SQLite store safe for several processes writing at once?

The code uses a single sqlite3 connection with a commit after each record(). There is no extra locking or WAL setup. Treat one writer as the supported case. Memory mode is one process only.

Why is my health score 0 with no_data?

No events for that domain fall inside the window. Either nothing was recorded, the domain string does not match, or the events are older than 24 hours.

How do I run tests?

pip install -e ".[dev]"
pytest tests/ -v

License

MIT.