Add screenshot capturing support

[ska-ibees]

Summary

Adds opt-in screenshot capturing support through:

• adv.crawl_screenshots()
• advertools screenshots CLI command
• optional dependency extra: advertools[screenshots]

Screenshots are captured with scrapy-playwright, saved as image files, and paired with JSON Lines metadata.

Highlights

• Supports PNG/JPEG screenshots
• Full-page or viewport-only capture
• Mobile viewport/context options
• Playwright actions before capture
• Timeout support
• Structured metadata for success/error diagnostics
• CLI support for inline JSON and @file.json
• Tests and docs included

Verification

python -m ruff check advertools/screenshot_spider.py tests/test_screenshot_spider.py advertools/code_recipes/screenshot_strategies.py

[eliasdabbas]

Thanks a lot @ska-ibees
Appreciate the contribution and sorry for the delay.

It's great to be able to make screenshots with Playwright.

I've looked into this again, and there might be a simpler solution. This can be achieved with a simple Scrapy middleware. Here's the full example:

screenshot_middleware.py

import hashlib
from pathlib import Path
from urllib.parse import urlsplit

from scrapy_playwright.page import PageMethod


class ScreenshotMiddleware:
    """Route every page through Playwright and save a full-page screenshot."""

    def __init__(self, settings):
        self.dir = Path(settings.get("SCREENSHOT_DIR", "screenshots"))
        self.dir.mkdir(parents=True, exist_ok=True)

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def process_request(self, request, spider):
        # robots.txt is fetched with a plain request; don't render it.
        if request.url.endswith("/robots.txt"):
            return None
        slug = urlsplit(request.url).path.strip("/").replace("/", "_") or "index"
        digest = hashlib.md5(request.url.encode()).hexdigest()[:8]
        path = self.dir / f"{slug}-{digest}.png"

        request.meta["playwright"] = True
        request.meta.setdefault("playwright_page_methods", []).append(
            PageMethod("screenshot", path=str(path), full_page=True)
        )
        return None

Right next to this file, create this one:

run_screenshot_crawl.py

import os

# Make screenshot_middleware.py importable inside the `scrapy runspider`
# subprocess that adv.crawl spawns.
os.environ["PYTHONPATH"] = os.getcwd() + os.pathsep + os.environ.get("PYTHONPATH", "")

import advertools as adv

adv.crawl(
    [
        "https://example.com",
        "https://adver.tools/seo-crawler/",
        "https://adver.tools/",
        "https://adver.tools/xml-sitemaps/",
    ],
    "screenshot_crawl.jl",
    custom_settings={
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOADER_MIDDLEWARES": {"screenshot_middleware.ScreenshotMiddleware": 543},
        "SCREENSHOT_DIR": "screenshots",
        "CLOSESPIDER_PAGECOUNT": 1,  # demo cap; remove for a full crawl
    },
)

Run:

python3 run_screenshot_crawl.py

This untangles the code from advertools and doesn't add any changes to the current code.

This is a minimal solution that "just works", and can definitely be refined by incorporating many other options like file naming, whether you want the full page or just above the fold, etc.

I suggest we run this like this, make sure it covers all the available playwright options, create several tutorials. And once we are sure the workflow is right, we can see how to incorporate into advertools.

What do you think?