feat: add SerpBase (Google) search engine via REST API - #158
Open
gefsikatsinelou wants to merge 2 commits into
Open
feat: add SerpBase (Google) search engine via REST API#158gefsikatsinelou wants to merge 2 commits into
gefsikatsinelou wants to merge 2 commits into
Conversation
Adds a new built-in scraper that returns Google organic search results as clean JSON from the SerpBase Search API (POST /google/search, X-API-Key header), with no browser, CAPTCHA handling, or selector maintenance. - Mirror the IMDBScraper API-key pattern: SERPBASE_API_KEY env var or api_key=..., graceful degradation with a helpful error when unset - Optional language/country params map to Google's hl/gl; results are truncated to max_results client-side - Business error envelope (HTTP 200 + status!=0) surfaces as a ScrapeError, never as a silent empty result - Registered in scrapers/__init__.py and package __init__.py (import, registration tuple, __all__) so MCP/CLI expose it automatically - 6 unit tests (parse, missing key, error envelope, truncation, request shape, async path); full suite: 490 passed, 9 skipped Closes nothing; standalone feature addition.
gefsikatsinelou
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new built-in, API-backed Google web search scraper (GoogleSearchScraper) that retrieves organic results as structured JSON via SerpBase, and wires it into the package’s built-in scraper exports/registry with accompanying documentation and unit tests.
Changes:
- Introduces
GoogleSearchScraperwith sync + native async implementations usingHttpClient.post_json/AsyncHttpClient.post_json. - Registers/exports the new scraper via
pyscrappy.scrapersand top-levelpyscrappyso it appears inlist_scrapers()and related entry points. - Adds unit tests and updates README + changelog to reflect the new built-in scraper.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/pyscrappy/scrapers/google_search.py |
New SerpBase-backed Google search scraper (sync/async) and result mapping. |
tests/test_scrapers/test_google_search.py |
Unit tests for parsing, error handling, request shape, truncation, and async path. |
src/pyscrappy/scrapers/__init__.py |
Exposes GoogleSearchScraper via scraper package imports and __all__. |
src/pyscrappy/__init__.py |
Registers and exports GoogleSearchScraper at the top-level package. |
README.md |
Updates built-in scraper count and mentions GoogleSearchScraper in the list. |
CHANGELOG.md |
Adds an Unreleased entry documenting GoogleSearchScraper. |
Suppressed comments (1)
src/pyscrappy/scrapers/google_search.py:72
- The
max_resultsdocstring implies the SerpBase API response is truncated, but truncation is performed client-side after parsing. Update the wording to match the implementation (and the PR description).
max_results: Maximum number of organic results to return (the API
response is truncated to this many records).
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+47
to
+52
| def test_missing_key_returns_helpful_error(self): | ||
| s = GoogleSearchScraper() | ||
| r = s.scrape(query="anything") | ||
| assert r.data == [] | ||
| assert "SERPBASE_API_KEY" in r.errors[0].message | ||
| assert "serpbase.dev" in r.errors[0].message |
|
|
||
| - **`GenericScraper`** — scrape any URL with auto-extraction (text, links, images, tables, metadata) | ||
| - **Data / research** — **`WikipediaScraper`**, **`StockScraper`** (Yahoo Finance), **`NewsScraper`** (RSS/Atom), **`GitHubScraper`**, **`HackerNewsScraper`**, plus weather, crypto, currency, dictionary, image, LinkedIn-jobs, and book search | ||
| - **Data / research** — **`WikipediaScraper`**, **`StockScraper`** (Yahoo Finance), **`NewsScraper`** (RSS/Atom), **`GitHubScraper`**, **`HackerNewsScraper`**, **`GoogleSearchScraper`** (Google organic results as JSON via the SerpBase API), plus weather, crypto, currency, dictionary, image, LinkedIn-jobs, and book search |
Comment on lines
+11
to
+13
| pass ``api_key`` to the constructor. New accounts start with 100 free searches | ||
| (no credit card required); afterwards it is pay-as-you-go ($0.30 per 1k | ||
| queries). |
| detail = payload.get("error") or payload.get("message") or "request failed" | ||
| return ScrapeResult( | ||
| data=[], | ||
| metadata=ScrapeMetadata(scraper=self.name), |
- test: clear SERPBASE_API_KEY via monkeypatch for deterministic missing-key test - docs: mention GoogleSearchScraper alongside IMDBScraper in README API-key note - docs: link to serpbase.dev instead of hardcoding trial limits/pricing - fix: include source_urls in error-path metadata (consistent with success path) - style: ruff format compliance
Author
|
Addressed the review comments in dc0a105:
Full local run: 493 passed, 6 skipped; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new built-in scraper to PyScrappy:
GoogleSearchScraperreturns Google organic search results as clean JSON via the SerpBase Search API — no browser, no CAPTCHA handling, no selector maintenance.Background
PyScrappy's 25 scrapers cover Wikipedia, news, GitHub, image search, and more, but there was no Google web search scraper. That's not an oversight: Google's search pages are effectively unreachable for scripted clients (consent interstitial, CAPTCHA walls, frequently changing markup — the same class of problem the IMDB scraper already solved by switching from blocked HTML to a JSON API). The IMDB scraper sets the precedent in this codebase: blocked page → structured API with an env-var key.
SerpBase (https://serpbase.dev) is a Google Search Results API that returns organic results (title, link, snippet, position) as JSON from a single POST. It follows the exact pattern the IMDB scraper uses, right down to the key-missing behavior.
Changes
src/pyscrappy/scrapers/google_search.py—GoogleSearchScraper(name = "google_search"):https://api.serpbase.dev/google/searchwith a JSON body (q, optionalhl/gl) and the key in theX-API-Keyheaderorganic[]array to{title, link, snippet, position}records in Google's ranking ordermax_results(default 10) client-sidescrape) and native async (scrape_async) paths, mirroring the IMDBScraper structuresrc/pyscrappy/scrapers/__init__.py— import +__all__entry (alphabetical)src/pyscrappy/__init__.py— import, built-in registration tuple, and__all__entry, so the MCP server, CLI, andlist_scrapers()expose it automatically (no core or MCP changes needed)tests/test_scrapers/test_google_search.py— 6 unit tests (see Testing)README.md(scraper count 24 → 25 + listing) andCHANGELOG.md(Unreleased entry)Design decisions
SERPBASE_API_KEYenv var /api_key=OMDB_API_KEYpattern inIMDBScraperX-API-KeyheaderScrapeResultwith a helpfulScrapeError(not an exception, not a silent empty list)IMDBScraperdegrades; no behavior change for users without a keyHTTP 200+status != 0) surfaced asScrapeErrorHttpClient.post_json/AsyncHttpClient.post_json(httpx is already a core dependency)max_resultslanguage/country→hl/glTesting
post_json, no network): parse, missing key, error-envelope, max_results truncation, request shape/headers, async pathruff checkclean on all changed/new files;compileallclean;git diff --checkcleanNo API key is ever committed; tests mock upstream HTTP per the contributing guide.