WYC (Wychavon) scraper failing — ModGov server returning 503 and http_lib needs update

[symroe]

Error (2026-06-07)

Connection reset during SSL handshake
Duration: 0.3s

Investigation

Two issues found:

1. Server returning 503

https://mgov.wychavon.gov.uk/modern.gov/mgWebService.asmx/GetCouncillorsByWard currently returns HTTP 503 Service Unavailable:

HTTP/2 503
content-length: 239
content-type: text/plain

The base domain (mgov.wychavon.gov.uk) also returns 503. This is the primary cause of failure.

2. http_lib = "requests" causing TLS fingerprint rejection

The scraper still uses http_lib = "requests" (left over from before the URL was fixed in PR #313). When tested with the default wreq client (removing http_lib = "requests"), the endpoint returns CERTIFICATE_VERIFY_FAILED — wreq's BoringSSL bundle does not trust the Wychavon cert.

The combination of these means:

• With requests: "Connection reset during SSL handshake" (requests' TLS fingerprint is rejected by server/WAF)
• With wreq: CERTIFICATE_VERIFY_FAILED (cert not trusted by BoringSSL)
• With verify_requests = False (wreq, no cert check): 503 (server down)

What needs to happen

1. Short term: Check whether mgov.wychavon.gov.uk is temporarily down or has moved.
2. Code fix needed: Once the server is back, replace http_lib = "requests" with verify_requests = False (using wreq's Firefox TLS fingerprint instead of Python requests, which bypasses the TLS fingerprint rejection). The requests library is a last-resort option per our conventions and is causing the connection reset.

[symroe]

Update 2026-06-10

Still failing on the 2026-06-08 scheduled run. Error changed from 503 to connection reset:

('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Duration: 0.3s

Local reproduction confirms: mgov.wychavon.gov.uk/modern.gov/mgWebService.asmx/GetCouncillorsByWard returns HTTP 503 with both requests (locally) and wreq + verify_requests=False (tested). The connection reset in the dashboard (vs. a 503 response locally) is consistent with Lambda's IP range getting the TLS fingerprint actively rejected before a response is sent, while other IPs receive the 503 body.

The code fix (replacing http_lib = "requests" with verify_requests = False) is still blocked by the server-side 503. Once the server returns, that one-line change should be applied.

---

Generated by Claude Code

[symroe]

2026-06-12 check: https://mgov.wychavon.gov.uk/modern.gov/mgWebService.asmx/GetCouncillorsByWard still returns HTTP 503. Still appearing in the failing feed. No code fix possible until the server is back up.

---

Generated by Claude Code

[symroe]

2026-06-13 check: https://mgov.wychavon.gov.uk/modern.gov/mgWebService.asmx/GetCouncillorsByWard still returning HTTP 503. Still appearing in the failing feed. The http_lib = "requests" code fix (→ verify_requests = False) remains blocked by the server-side 503.

---

Generated by Claude Code

[symroe]

Still failing as of 2026-06-14. Error: Connection reset by peer during SSL handshake negotiation (0.33s). Server remains unreachable.

---

Generated by Claude Code

[symroe]

2026-06-17 check: Still failing — https://mgov.wychavon.gov.uk/modern.gov/mgWebService.asmx/GetCouncillorsByWard returns HTTP 503 (upstream connect error or disconnect/reset before headers. retried and the latest reset reason: TLS_error). Server remains unreachable.

---

Generated by Claude Code

[symroe]

2026-06-18 update

Still failing. Error today: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) from the requests library — same TLS handshake rejection as before. Server (mgov.wychavon.gov.uk) is still returning this error.

---

Generated by Claude Code

[symroe]

2026-06-19 update: Still failing — mgov.wychavon.gov.uk is returning HTTP 503. The fix (replace http_lib = "requests" with verify_requests = False) documented in this issue remains the right approach. Branch fix/WYC-wychavon exists but has no changes yet. Cannot open PR until the server recovers and the fix can be verified.

---

Generated by Claude Code