Background
The current keyword-search implementation (Scraper.search, added in 268d527) drives the browser SERP at /explore/search/keyword/?q=<keyword> and collects XHR-intercepted posts while scrolling (browser_session.py:751, worker.py:203). Two known limitations:
- Results aren't reliably chronological. As noted in the docstrings (
scraper.py:118-124, browser_session.py:759-762), the SERP doesn't return results in time order, so there's currently no date cutoff for search — unlike user_timeline, which filters by start_date/end_date via post_date_filterer (worker.py:192-198).
- We only have one entry point (the SERP), so we can't bound a query to a time window the way we can for a user timeline.
Goals
1. Investigate alternate search methods
Explore endpoints/approaches beyond the /explore/search/keyword/ SERP that might give better coverage and/or chronological ordering, e.g.:
- The web/topsearch and
fbsearch-style endpoints (/web/search/topsearch/, GraphQL search queries) and what result types they return (posts vs. users vs. hashtags vs. places).
- Hashtag and location feeds as a proxy for keyword search, which tend to be more chronological.
- Whether any search surface exposes pagination cursors we can walk, rather than relying on infinite scroll + XHR interception.
- Trade-offs: result completeness, ordering guarantees, rate-limit/blocking behavior, and how each fits the existing account-rotation / session-reuse machinery.
2. Date-range filtering via API query params
Determine whether any search surface accepts start_date/end_date (or since/until/timestamp) query parameters so we can bound results server-side, and wire that through:
- Add
start_date/end_date to Scraper.search (scraper.py:110) mirroring user_timeline (scraper.py:57).
- Thread the dates into the
Query and apply server-side filtering where supported; fall back to client-side post_date_filterer (worker.py:196) where it isn't.
Acceptance criteria
- A short writeup of candidate search methods with the trade-offs above.
- A determination of whether date-range query params are viable on any surface, with evidence.
- If viable:
Scraper.search accepts start_date/end_date and respects them.
Background
The current keyword-search implementation (
Scraper.search, added in268d527) drives the browser SERP at/explore/search/keyword/?q=<keyword>and collects XHR-intercepted posts while scrolling (browser_session.py:751,worker.py:203). Two known limitations:scraper.py:118-124,browser_session.py:759-762), the SERP doesn't return results in time order, so there's currently no date cutoff for search — unlikeuser_timeline, which filters bystart_date/end_dateviapost_date_filterer(worker.py:192-198).Goals
1. Investigate alternate search methods
Explore endpoints/approaches beyond the
/explore/search/keyword/SERP that might give better coverage and/or chronological ordering, e.g.:fbsearch-style endpoints (/web/search/topsearch/, GraphQL search queries) and what result types they return (posts vs. users vs. hashtags vs. places).2. Date-range filtering via API query params
Determine whether any search surface accepts
start_date/end_date(orsince/until/timestamp) query parameters so we can bound results server-side, and wire that through:start_date/end_datetoScraper.search(scraper.py:110) mirroringuser_timeline(scraper.py:57).Queryand apply server-side filtering where supported; fall back to client-sidepost_date_filterer(worker.py:196) where it isn't.Acceptance criteria
Scraper.searchacceptsstart_date/end_dateand respects them.