Async TLS stealth, on-disk response cache, and heal report (1.5.7)

[vedaant00]

No description provided.

[Copilot]

🟡 Changes recommended

The new disk-cache implementation can raise at runtime with stealth responses (and doesn’t expand ~ cache paths), which can break successful scrapes when cache_dir is enabled.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR releases PyScrappy 1.5.7 with new observability and caching features, and extends TLS-fingerprint “stealth” impersonation to the async HTTP stack.

Changes:

• Add AdaptiveStore.heal_report() to summarize selector drift from the heal audit log.
• Add an optional persistent on-disk HTTP response cache via ScraperConfig(cache_dir=...), shared across sync/async clients.
• Add async stealth support (build_async_stealth_client / AsyncStealthClient) so ScraperConfig(impersonate=...) works with AsyncHttpClient / scrape_async, plus docs/tests and version bumps.

File summaries

File	Description
tests/test_generic/test_adaptive.py	Adds tests for AdaptiveStore.heal_report() aggregation and empty behavior.
tests/test_core/test_stealth.py	Updates stealth tests for new async stealth builder and adds async adapter coverage.
tests/test_core/test_http.py	Adds regression tests for disk cache persistence across “restart” and default-off behavior.
src/pyscrappy/generic/adaptive_store.py	Implements heal_report() summary over the heal audit log.
src/pyscrappy/core/http.py	Implements _DiskCache + shared registry and integrates disk caching into sync client.
src/pyscrappy/core/config.py	Adds cache_dir config and updates impersonate docs to include async support.
src/pyscrappy/core/async_http.py	Enables async impersonate via async stealth client and integrates disk caching.
src/pyscrappy/core/_stealth.py	Adds AsyncStealthClient and build_async_stealth_client().
src/pyscrappy/init.py	Bumps library version to 1.5.7.
server.json	Bumps server/package version references to 1.5.7.
README.md	Documents heal_report(), async impersonation, and persistent disk cache usage/semantics.
pyproject.toml	Bumps project version to 1.5.7.
CHANGELOG.md	Adds 1.5.7 entries for async stealth, disk cache, and heal_report().

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

There are correctness issues in the new disk-cache rehydration/serialization and an async-stealth HTTP error method mismatch that should be addressed before release.

Review details

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/pyscrappy/generic/adaptive_store.py:175

• heal_report() computes last_confidence from the last non-None confidence in the list, which can diverge from last_healed when the most recent heal entry has confidence=None (the report would show an older confidence but a newer timestamp). Consider taking last_confidence directly from the most recent heal entry and keeping min/avg as the aggregate over non-None values.

        for (identifier, ns), heals in grouped.items():
            confidences = [h["confidence"] for h in heals if h.get("confidence") is not None]
            report.append(
                {
                    "identifier": identifier,
                    "namespace": ns,
                    "heals": len(heals),
                    "last_confidence": confidences[-1] if confidences else None,
                    "min_confidence": min(confidences) if confidences else None,
                    "avg_confidence": (
                        round(sum(confidences) / len(confidences), 2) if confidences else None
                    ),
                    "last_healed": heals[-1].get("timestamp"),
                }

src/pyscrappy/core/http.py:168

• _DiskCache.get() can raise (and break the "best-effort" guarantee) when cached data has a missing/invalid url because it unconditionally builds httpx.Request("GET", data.get("url", "")). Also, serializing resp.text and re-encoding as UTF-8 can corrupt non-UTF8/binary bodies and can mismatch Content-Type charset headers on rehydrate. Storing raw bytes losslessly (e.g., via a reversible encoding) and guarding response reconstruction keeps disk-cache reads safe and faithful.

        return httpx.Response(
            status_code=data["status"],
            headers=data.get("headers", {}),
            content=data["body"].encode("utf-8"),
            request=httpx.Request("GET", data.get("url", "")),

tests/test_core/test_http.py:570

• test_disk_cache_off_by_default currently asserts that tmp_path / "httpcache" doesn't exist, but the code under test never uses tmp_path (no cache_dir is set), so this assertion will always pass even if disk caching were accidentally enabled elsewhere. This test should instead assert that the disk-cache path factory is not invoked (or that the disk-cache registry remains empty) when cache_dir is None.

    def test_disk_cache_off_by_default(self, tmp_path):
        # No cache_dir => nothing written to disk (in-memory only).
        cfg = ScraperConfig(rate_limit=0, cache_ttl=60)
        client, _ = self._disk_mock_client(cfg)
        client.get("https://example.com")

src/pyscrappy/core/_stealth.py:200

• Async stealth now supports post_json() via AsyncStealthClient.post(), but the response wrapper’s raise_for_status() currently hardcodes a GET request when building httpx.HTTPStatusError. That means HTTP errors from async POSTs will report exc.request.method == "GET", which can confuse callers/logging and may break code that keys off the method. Consider letting _StealthResponse know the request method (and URL) so it can construct an accurate httpx.Request for errors on both sync and async paths.

    async def _request(self, method: str, url: str, **kwargs: Any) -> _StealthResponse:
        if "follow_redirects" in kwargs:
            kwargs["allow_redirects"] = kwargs.pop("follow_redirects")
        try:
            raw = await self._session.request(method, url, **kwargs)
        except self._cffi_errors as exc:  # network/transport failure
            request = httpx.Request(method, url)
            raise httpx.RequestError(str(exc), request=request) from exc
        return _StealthResponse(raw)

    async def get(self, url: str, **kwargs: Any) -> _StealthResponse:
        return await self._request("GET", url, **kwargs)

    async def post(self, url: str, **kwargs: Any) -> _StealthResponse:
        return await self._request("POST", url, **kwargs)

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The disk cache currently fails to persist correct URLs for stealth responses (leading to cached responses with a dummy request URL), and the async client now violates its own type contracts after introducing the async stealth adapter.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

src/pyscrappy/core/http.py:166

• _DiskCache.put() tries to persist the response URL for stealth responses, but the fallback getattr(resp, "url", "") will be empty for _StealthResponse (it only stores _raw.url). That causes cached stealth responses to reload with a dummy request URL ("http://cached"), which can break callers that inspect response.request.url / response.url.

            # The URL may come from an httpx.Response (resp.request.url) or from
            # the stealth adapter's _StealthResponse (raw .url, no .request). Read
            # it defensively so caching never raises on a stealth response.
            request = getattr(resp, "request", None)
            url = str(request.url) if request is not None else str(getattr(resp, "url", ""))

src/pyscrappy/core/async_http.py:49

• AsyncHttpClient can now hold an AsyncStealthClient instance when config.impersonate is set, but _client is still annotated as httpx.AsyncClient | None. This is now an incorrect type hint and will mislead type checkers and IDEs.

This issue also appears on line 194 of the same file.

    def __init__(self, config: ScraperConfig | None = None) -> None:
        self.config = config or ScraperConfig()
        self._client: httpx.AsyncClient | None = None
        self._last_request_time: dict[str, float] = {}

src/pyscrappy/core/async_http.py:198

• AsyncHttpClient._build_client() is annotated to return httpx.AsyncClient, but it can return the stealth adapter (AsyncStealthClient) when config.impersonate is set. Adjusting the return type avoids a now-false contract for callers and static analysis.

    def _build_client(self) -> httpx.AsyncClient:
        proxy = self.config.pick_proxy(exclude=self._current_proxy)
        self._current_proxy = proxy
        # TLS impersonation: swap httpx for a curl_cffi AsyncSession that mimics a
        # real browser's fingerprint. It presents the same async surface

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The new disk-cache header serialization should preserve/round-trip multi-value headers (e.g., repeated Set-Cookie) to avoid semantic changes on disk cache hits.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/pyscrappy/core/http.py:159

• _DiskCache.get() currently passes the JSON "headers" value straight into httpx.Response. If headers are stored as a list of pairs (to preserve duplicates), they should be normalized back to a list of tuples before constructing the Response to avoid type/compat issues.

This issue also appears on line 177 of the same file.

            content = base64.b64decode(data["body"])
            return httpx.Response(
                status_code=data["status"],
                headers=data.get("headers", {}),
                content=content,

src/pyscrappy/core/http.py:177

• Serializing headers with dict(resp.headers) will drop duplicate header fields (e.g., multiple Set-Cookie), so a cached response may not preserve the original header semantics. Prefer storing multi-value headers when available (httpx.Headers.multi_items()) and fall back only when not supported.

                "headers": dict(resp.headers),

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Lite