Not another stealth browser.
SentinelScrape is the control plane that continuously discovers which browser fingerprints still work against Cloudflare, DataDome, Akamai, PerimeterX & friends — then ships proven fixes to your scrapers via REST + WebSocket.
# 60-second demo (no Docker, no API keys)
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python scripts/demo_ab_test.pyIf this saves you a week of reverse-engineering — ⭐ star the repo so others find it too.
| What people use today | What breaks |
|---|---|
| Camoufox, undetected-chromedriver, nodriver | Static patches — vendor rotates detection → you wait for a community PR |
| cloudscraper, curl-impersonate | Great for one vector (TLS/CF); no continuous learning across sites |
| Manual JA3/JA4 + canvas research | Works once; doesn't scale to 500 domains that all change |
Anti-bot vendors ship new signals weekly. Hardcoded stealth lags by design.
SentinelScrape treats bot detection as an adversarial ML loop:
probe fingerprints → learn what blocks you → detect rule changes
↑ │
└──── scrapers get new profile ← A/B winner ←──┘
Think of it as Datadog for bot fingerprints.
You already have scrapers (Playwright, Puppeteer, Scrapy + browsers).
This codebase answers three questions automatically:
- What fingerprint works on
shop.exampleright now? - What changed when success rate collapsed overnight?
- Which fix is proven (A/B) before we push it to production crawlers?
from src.scraper_client import SentinelClient
client = SentinelClient("http://localhost:8000")
fp = await client.get_fingerprint("shop.example")
# Use the recommended browser fingerprint
opts = client.apply_profile_to_playwright_context_options(
fp["recommended_profile"]
)
# context = await browser.new_context(**opts)When a patch is A/B-validated, scrapers can also get it live:
WS /ws/v1/patches → { "domain": "shop.example", "patch": "canvas_use_real", ... }
| You are… | You get… |
|---|---|
| Data / scraping platform | Fleet success rate as an SLO; faster recovery when CF rotates |
| Price intelligence / SERP / travel / e-com monitor | One fingerprint service for all crawlers |
| Security / anti-bot researcher | Feature-importance reports: which signal is burning us? |
| Infra team for browser fleets | Control plane + K8s + Docker, not scripts in Slack |
- One-off scrape of an unprotected site → overkill
- You need CAPTCHA solving → out of scope
- You need guaranteed access to every site → impossible; this raises success rate and shortens MTTR
| Feature | What it does | Why people star it |
|---|---|---|
| Fingerprint API | POST /api/v1/fingerprint |
Scrapers stop hardcoding UA/canvas hacks |
| Probe fleet | Playwright workers + Redis job queue | Continuous measurement, not gut feel |
| Detection classifier | Random forest on probe outcomes | Names the vector (WebGL, canvas, TLS…) |
| Drift detection | Feature-importance shift alerts | Knows when the site changed rules |
| A/B patch engine | Control vs candidate profiles | Only deploys fixes that beat production |
| WebSocket push | Live patch fan-out | Crawlers update without redeploy |
| Scraper SDK | SentinelClient |
Drop-in for existing Python crawlers |
| Docker + K8s | Full stack manifests | Production-shaped from day one |
┌─────────────┐ REST / WS ┌──────────────────────────┐
│ Scrapers │ ◄────────────────► │ Sentinel Controller │
│ (your app) │ fingerprint API │ FastAPI · Scheduler · ML │
└─────────────┘ └────────────┬─────────────┘
│
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
PostgreSQL Redis Probe nodes
results · patches jobs · cache Playwright
alerts · history pub/sub fingerprint matrix
Closed loop
- Probe — systematic fingerprint matrix (TLS, browser, canvas, WebGL, behavior…)
- Classify — which features correlate with blocks?
- Drift — did the site’s detection logic change?
- Patch — generate candidate mitigations from a library
- A/B test — control (prod) vs treatments; Wilson CI + success threshold
- Deploy —
active_patches+ WebSocket → scrapers
Deep dive: ARCHITECTURE.md · use cases: docs/USE_CASES.md
git clone https://github.com/pandeyvishwas51-oss/sentinelscrape.git
cd sentinelscrape
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python scripts/demo_ab_test.pyYou’ll see: probe → classify → drift → generate patches → A/B winner → deploy.
cp .env.example .env
docker compose up --build
# API: http://localhost:8000/docs
curl -X POST http://localhost:8000/api/v1/admin/targets \
-H "Content-Type: application/json" \
-d '{"domain":"httpbin.org","url":"https://httpbin.org/get","active":true}'pip install -e ".[dev]"
docker compose up -d postgres redis
python -m src.sentinel_controller.main # terminal 1
python -m src.probe_node.main # terminal 2Full guide: docs/HOW_TO_USE.md
| Step | What you do | Why |
|---|---|---|
| 1 | Install Python 3.11+ and clone the repo | Runtime |
| 2 | pip install -e ".[dev]" |
Dependencies |
| 3 | Run python scripts/demo_ab_test.py |
Understand the loop offline |
| 4 | Start Postgres + Redis (docker compose up -d postgres redis) |
Storage + job queue |
| 5 | Start controller + probe node | Control plane + workers |
| 6 | Register your target domains via admin API | What to monitor |
| 7 | In your scraper, call SentinelClient.get_fingerprint(domain) |
Use the winning profile |
| 8 | (Optional) Subscribe to WS /ws/v1/patches |
Live updates when A/B deploys |
Minimum to try the idea: steps 1–3 only (no Docker).
Minimum for production-shaped use: steps 1–7.
When detection drifts, candidates are not pushed blindly:
CONTROL (current prod fingerprint) 0% success ○
T1 canvas_use_real 100% success ★ WINNER
T2 canvas_spoofed_intel 100% success
T3 canvas_noise_injected 0% success ✗ rejected
Rules: beat control and clear PATCH_SUCCESS_THRESHOLD (default 90%).
Bad patches lose the A/B — they never ship.
pytest tests/test_ab_tester.py -v
python scripts/demo_ab_test.py| Method | Path | Purpose |
|---|---|---|
GET |
/health |
Liveness |
GET |
/ready |
DB + Redis readiness |
POST |
/api/v1/fingerprint |
Best fingerprint for a domain |
POST |
/api/v1/fingerprint/batch |
Batch lookup |
GET |
/api/v1/detections/{domain} |
Classifier / top vectors |
GET |
/api/v1/patches/{domain} |
Active validated patch |
POST |
/api/v1/admin/targets |
Register domain to probe |
WS |
/ws/v1/patches |
Live patch stream |
Interactive docs: http://localhost:8000/docs · full spec: docs/API.md
sentinelscrape/
├── src/sentinel_controller/ # FastAPI control plane + ML + A/B
├── src/probe_node/ # Playwright probe workers
├── src/scraper_client/ # SDK for scrapers
├── src/shared/ # FingerprintProfile, ProbeResult
├── scripts/demo_ab_test.py # Star-worthy demo
├── migrations/ # Postgres schema
├── k8s/ # Kubernetes manifests
└── tests/ # Classifier, A/B, API, probes
| Camoufox / uc / nodriver | SentinelScrape | |
|---|---|---|
| Role | Stealth browser implementation | Fingerprint control plane |
| Updates | Manual / community patches | Continuous probe + A/B |
| Multi-domain | Same binary everywhere | Per-domain recommended profile |
| Change detection | You notice scrapers dying | Drift alerts + named vectors |
| Deploy to fleet | Rebuild images | REST + WebSocket push |
Use both: run Camoufox/Playwright as the engine; let SentinelScrape decide which config wins today.
anti-bot · bot detection · browser fingerprinting · cloudflare bypass research ·
playwright stealth · datadome · akamai bot manager · JA3 · JA4 · TLS fingerprint ·
canvas fingerprint · webgl fingerprint · web scraping · scrapy · puppeteer ·
undetected-chromedriver · camoufox · adversarial ML · A/B testing fingerprints
Research / defensive measurement tool for teams that operate legitimate crawlers
(price monitoring, SEO rank tracking, compliance, security research).
Always respect site ToS and robots policy.
pytest tests/ -v --ignore=tests/integration # unit + A/B + ML
python scripts/verify.py # live stack checklist
make test- curl-impersonate / uTLS probe workers (true JA4 matrix)
- Grafana dashboard for per-domain success SLO
- Multi-controller Redis lock for HA scheduler
- Public fingerprint provider plugins
- Helm chart
See CONTRIBUTING.md.
If SentinelScrape is useful:
- Star the repo (helps others find adaptive fingerprinting tools)
- Watch releases
- Open an issue with your anti-bot vendor + failure mode
- PR a patch library entry for a vector you reverse-engineered
MIT — use it, fork it, ship it in your stack.
Stop waiting for stealth libraries to catch up.
Probe → Learn → A/B → Deploy.
⭐ Star SentinelScrape