Skip to content

Investigate alternate Instagram search methods + date-range (start/end) filtering via API query params #4

Description

@bendavidsteel

Background

The current keyword-search implementation (Scraper.search, added in 268d527) drives the browser SERP at /explore/search/keyword/?q=<keyword> and collects XHR-intercepted posts while scrolling (browser_session.py:751, worker.py:203). Two known limitations:

  1. Results aren't reliably chronological. As noted in the docstrings (scraper.py:118-124, browser_session.py:759-762), the SERP doesn't return results in time order, so there's currently no date cutoff for search — unlike user_timeline, which filters by start_date/end_date via post_date_filterer (worker.py:192-198).
  2. We only have one entry point (the SERP), so we can't bound a query to a time window the way we can for a user timeline.

Goals

1. Investigate alternate search methods

Explore endpoints/approaches beyond the /explore/search/keyword/ SERP that might give better coverage and/or chronological ordering, e.g.:

  • The web/topsearch and fbsearch-style endpoints (/web/search/topsearch/, GraphQL search queries) and what result types they return (posts vs. users vs. hashtags vs. places).
  • Hashtag and location feeds as a proxy for keyword search, which tend to be more chronological.
  • Whether any search surface exposes pagination cursors we can walk, rather than relying on infinite scroll + XHR interception.
  • Trade-offs: result completeness, ordering guarantees, rate-limit/blocking behavior, and how each fits the existing account-rotation / session-reuse machinery.

2. Date-range filtering via API query params

Determine whether any search surface accepts start_date/end_date (or since/until/timestamp) query parameters so we can bound results server-side, and wire that through:

  • Add start_date/end_date to Scraper.search (scraper.py:110) mirroring user_timeline (scraper.py:57).
  • Thread the dates into the Query and apply server-side filtering where supported; fall back to client-side post_date_filterer (worker.py:196) where it isn't.

Acceptance criteria

  • A short writeup of candidate search methods with the trade-offs above.
  • A determination of whether date-range query params are viable on any surface, with evidence.
  • If viable: Scraper.search accepts start_date/end_date and respects them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions