Indeed Job Scraper (private Apify Actor)

A custom Apify Actor that scrapes Indeed job listings via Apify's residential proxy group and headless Chromium. Built to run on the Apify Creator plan, which blocks running other developers' public Actors but allows you to publish and run your own.

Why custom: the popular misceres/indeed-scraper is a "public Actor" and those are blocked on the Creator plan ($1/mo base). Building our own gives us full control, no per-run rental fees, and a reusable component for every future job-search client.

What it does

Input:

• position — job title query (e.g. "Program Director")
• location — geographic filter (e.g. "United States", "Atlanta, GA")
• country — controls which Indeed domain we hit (US, UK, CA, etc.)
• maxItems — stop after N jobs are written
• freshnessDays — only jobs posted within N days
• fetchDetails — open each job's detail page for the full description

Output (dataset items):

{
  "positionName": "Senior Program Director",
  "company":      "United Way of Atlanta",
  "location":     "Atlanta, GA (Remote)",
  "salary":       "$120,000 - $140,000 a year",
  "postedAt":     "3 days ago",
  "url":          "https://www.indeed.com/viewjob?jk=abc123",
  "description":  "Full posting text...",
  "scrapedAt":    "2026-06-09T18:00:00+00:00"
}

The schema matches the field mapping in the langgraph-jobsearch-agent profile.yaml so the downstream scoring node can ingest items unchanged.

Stack

• Python 3.11
• Apify SDK 2.x
• Crawlee Python with PlaywrightCrawler
• Headless Chromium (preinstalled in apify/actor-python-playwright:3.11)
• selectolax for fast HTML parsing
• Apify residential proxy group

Cost shape

• ~3 to 5 seconds of headless Chromium per page render
• 1 search page = ~15 jobs; with detail fetching, 1 search page + 15 detail pages = ~50 seconds total
• For maxItems=50: roughly 0.05 to 0.10 compute units, residential proxy bandwidth ~3 to 5 MB
• Apify cost: ~$0.01 to $0.03 per run

Develop locally

cd apify-indeed-scraper
apify login           # one-time
apify run             # runs against your local Apify storage

Deploy

apify push            # builds + uploads + tags 'latest'

After deploy, the Actor is available at <your-apify-username>/indeed-scraper and can be called from the langgraph-jobsearch-agent profile.yaml:

target_companies:
  apify:
    - actor_id: "<your-apify-username>/indeed-scraper"
      label: "indeed-program-director"
      input:
        position: "Program Director"
        location: "United States"
        country: "US"
        maxItems: 50
        freshnessDays: 14
        fetchDetails: true
      mapping:
        title: "positionName"
        company: "company"
        location: "location"
        url: "url"
        description: "description"

Known caveats

• Indeed periodically rotates CSS class names. The parser uses multiple fallback selectors per field; if all of them break, update src/parser.py _*_SELECTORS tuples.
• Cloudflare occasionally fires a JS challenge even on residential IPs. The handler raises and Crawlee retries with a different proxy; default 3 retries per request usually clears it.
• Indeed enforces fromage <= 30. We cap input accordingly.