AMB (Amber Valley) scraper failing — intermittent HTTP 500 from custom JSON API

[symroe]

Error (2026-06-11)

exceptions.StatusError: wreq::Error { kind: Status(500, None), uri: https://info.ambervalley.gov.uk/WebServices/AVBCFeeds/DemocracyJSON.asmx/GetAllCouncillors }
Duration: 15.7s

Investigation

The scraper calls https://info.ambervalley.gov.uk/WebServices/AVBCFeeds/DemocracyJSON.asmx/GetAllCouncillors — a custom ASMX/JSON web service (not ModGov).

On 2026-06-13 (today), the endpoint is responding correctly, returning a JSON array of 50 councillors with full contact details, ward assignments, and photo stream URLs. The service description page (/DemocracyJSON.asmx) is also accessible and lists all 71 methods. This strongly suggests the 2026-06-11 HTTP 500 was a transient server-side error.

Fix patterns ruled out

1. HTTPS migration — already HTTPS
2. verify_requests = False — not a cert issue; endpoint responds correctly today
3. URL changes — same URL is working today; no change needed
4. Selector/CSS changes — not applicable (JSON API)

Scraper code

The scraper code is correct and requires no changes. It calls GetAllCouncillors for the list and GetCouncillor?councillorRef={memberRef} for detail pages, which matches the live API.

What needs to happen

Monitor whether the 500 recurs. If it appears consistently in the failing feed over several consecutive days, investigate whether the server has a recurring maintenance window or whether the API contract has changed (e.g. method renamed or authentication added).