MDE (Mid Devon) scraper failing — CERTIFICATE_VERIFY_FAILED and intermittent 503

[symroe]

Error (2026-06-11)

exceptions.ConnectionError: wreq::Error { kind: Connect, source: CERTIFICATE_VERIFY_FAILED }

On Lambda the scraper completes but finds 0 councillors ("Not many councillors found (0)"), suggesting the Lambda environment has a different CA bundle that trusts the cert, but gets an Incapsula JS challenge instead of valid XML.

Investigation

Two layered issues found:

1. Certificate not trusted by wreq's BoringSSL

https://democracy.middevon.gov.uk has a TLS certificate that wreq's embedded BoringSSL CA bundle does not trust. Locally, the scraper fails immediately with CERTIFICATE_VERIFY_FAILED.

Fix: Add verify_requests = False to the Scraper class.

2. Server intermittently returning 503

After applying verify_requests = False, the scraper returns:

exceptions.StatusError: wreq::Error { kind: Status(503, None) }

Direct curl requests also time out (exit code 28). The 503 may be transient — the dashboard shows the scraper ran (returning 0 councillors) rather than failing with a network error, suggesting it was reachable during the Lambda run.

3. Possible Incapsula WAF challenge

The dashboard error "Not many councillors found (0)" in Lambda (where the cert is trusted) suggests the server may be returning an Incapsula JS challenge instead of councillor XML, resulting in 0 records parsed. If so, http_lib = "playwright" would be needed in addition to verify_requests = False.

Fix when server is back up

Once the server stops returning 503, apply and verify:

class Scraper(ModGovCouncillorScraper):
    verify_requests = False

If that still yields 0 councillors (Incapsula challenge), add:

class Scraper(ModGovCouncillorScraper):
    verify_requests = False
    http_lib = "playwright"