Release 1.5.6: robots.txt fail-closed, offset pagination, save() mkdir

[vedaant00]

Summary

Release 1.5.6. Bundles the robots.txt server-error fix (#152) with two
contributor fixes already merged to main (#151, #150), and bumps the version
across all sites.

Changes

Fixed

• robots.txt server errors now fail closed (robots.txt: server errors (5xx) are treated as allow-all instead of disallow-all #152). A 5xx (or a connection
  error / timeout) while fetching robots.txt was parsed as an empty allow-all
  body and cached for the session, so a site whose robots.txt was temporarily
  500ing got crawled as if unrestricted. The fetch now uses get_raw so the real
  status code is visible, and handling is split by class per RFC 9309 / Googlebot:
  2xx uses the rules, 4xx is allow-all (both cacheable), and 5xx / network
  errors fail closed with a disallow-all parser that is not cached, so the
  next request re-fetches. Sync and async share one decision helper.
• Offset-based pagination advances by the page size, not by 1 (Offset-based pagination (offset=/start=) advances by 1 instead of the page size #151). For
  offset=/start= URLs, find_next_page_url infers the step from the gaps
  between the page links (e.g. 0/20/40/60 → +20) instead of +1, which
  silently under-collected on offset-paginated sites. page/p URLs still
  advance by 1.
• ScrapeResult.save() creates parent directories (save() fails when the target directory does not exist #150). Saving to a path
  whose directory does not exist (e.g. out/nested/data.json) now creates the
  tree instead of raising FileNotFoundError.

Chore

• Bump to 1.5.6 across pyproject.toml, __init__.py, and server.json; update
  CHANGELOG.

Testing

• Full suite: 523 passed, 19 deselected.
• ruff check and ruff format --check: clean.
• The robots robots.txt: server errors (5xx) are treated as allow-all instead of disallow-all #152 regression tests (5xx and network-error) fail on the pre-fix
  code and pass after, confirming the fix is exercised.

Closes #152.