Aqar Scraper

A compact, bilingual scraper for Saudi Arabia's largest property portal (Aqar.fm). It builds Arabic-slug URLs, fetches through a two-tier cascade (ScraperAPI primary, Firecrawl fallback), parses either raw HTML or markdown with interchangeable regex parsers, and writes clean, deduplicated sale listings to CSV — price, area, district, room counts, photos.

---

Why it's interesting

Scraping a right-to-left, Arabic-language portal breaks most of the assumptions a Western scraper is built on: the URLs are Arabic, the prices are written in Arabic-Indic numerals, and the same portal serves different markup depending on how you reach it. This scraper handles all three explicitly rather than hoping a generic extractor copes.

• Arabic all the way down. City and property-type names are kept in Arabic and percent-encoded into Aqar.fm's URL scheme; prices and areas written in Arabic-Indic (٠-٩) and Eastern-Arabic/Persian (۰-۹) digits are normalized to ASCII before any numeric parsing.
• One pipeline, two source formats. The primary fetch (ScraperAPI, JS-rendered) returns raw HTML; the fallback (Firecrawl CLI) returns markdown. A single dispatch decides which regex parser to run, and both emit the same 15-field record — the rest of the pipeline never knows or cares which path was taken.
• Free-first fetch cascade. ScraperAPI is tried first and only accepted if the response is HTTP 200 and actually contains listing markup (the Arabic word للبيع, "for sale"); only then does it fall back to the paid Firecrawl CLI. No key configured? It skips straight to the fallback.
• Honest pagination. Listing IDs are tracked in a set for cross-page dedup, and the crawl stops the moment a page yields zero new IDs (or fewer than a small threshold) — no fixed page count, no re-scraping the same cards.

How it works, precisely

No hand-waving — here is exactly what each stage does (scraper.py):

Concern	Mechanism
Arabic-slug URLs	quote(f"{type}-للبيع", safe="") percent-encodes the entire Arabic slug (e.g. شقق-للبيع → %D8%B4%D9%82%D9%82-...); page N is appended as /{N}
Digit normalization	a str.maketrans table maps Arabic-Indic ٠-٩ and Persian/Eastern ۰-۹ onto ASCII 0-9, applied in _to_float before re.sub strips non-numerics
Fetch cascade	scraperapi_fetch (with render=true) is accepted only on HTTP 200 and presence of للبيع; otherwise firecrawl_scrape returns markdown from a temp file
Format-agnostic parsing	dispatch on content.lstrip().startswith("<") → parse_listings_html (HTML) vs parse_listings (markdown); both yield the identical 15-column schema
Rental exclusion	any card containing للإيجار ("for rent"), /سنوي (yearly) or /شهري (monthly) is dropped, keeping the output to genuine sale listings
Dedup + stop heuristic	listing IDs are extracted as -(\d{5,}) from the URL and held in a set; the loop breaks when a page adds 0 new — or fewer than STOP_THRESHOLD (3) — IDs
Rate limiting	time.sleep(RATE_LIMIT_SECONDS) (1.5s) between page fetches

The CSV is written with a UTF-8 BOM (utf-8-sig) so the Arabic columns open correctly in Excel.

Architecture

  Arabic city + type  (e.g. الرياض / شقق)
          │
          │  build_url:  quote(…, safe="")  → percent-encoded Aqar.fm URL
          ▼
   ┌──────────────────────── fetch_page(url) ────────────────────────┐
   │                                                                 │
   │   ScraperAPI (render=true)                                      │
   │        │                                                        │
   │        ├─ HTTP 200 + contains "للبيع" ?  ──yes──►  raw HTML ────┼──┐
   │        │                                                        │  │
   │        └─ no / empty ─►  Firecrawl CLI  ──►  markdown ──────────┼──┤
   │                                                                 │  │
   └─────────────────────────────────────────────────────────────────┘ │
                                                                        │
                        content.lstrip().startswith("<") ?              │
                          ┌───────────────┴───────────────┐            ◄─┘
                          ▼                               ▼
                 parse_listings_html            parse_listings
                     (HTML regex)                (markdown regex)
                          └───────────────┬───────────────┘
                                          ▼
              normalize digits · filter rentals · dedup by listing_id
                                          ▼
                       output/*.csv   (utf-8-sig, 15 columns)

Tech stack

Python 3 · requests · standard-library re / csv / argparse · ScraperAPI (HTTP API) · Firecrawl CLI (fallback)

Running it locally

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Optional: a ScraperAPI key enables the primary (fast) fetch path.
# Without it, the scraper falls back to the Firecrawl CLI.
cp .env.example .env        # then edit, or just export the var:
export SCRAPERAPI_KEY=your_key_here

# Preview the URLs that would be crawled (no network calls):
python3 scraper.py --dry-run

# Scrape apartments + villas in Riyadh, 10 pages per combo:
python3 scraper.py --city الرياض --types شقق فلل --max-pages 10 --output output/riyadh.csv

CLI options:

Flag	Meaning	Default
--city	one or more cities, in Arabic (الرياض جدة الدمام مكة المدينة أبها)	الرياض (Riyadh)
--types	property types, in Arabic (شقق فلل أراضي عمائر دور بيت)	شقق فلل أراضي
--max-pages	max pages per city/type combination	50
--output	output CSV path	output/aqar_listings.csv
--api-key	ScraperAPI key (overrides SCRAPERAPI_KEY)	—
--dry-run	print URLs only, no scraping	off

Data & privacy

This repository contains code only. It ships no scraped listings, no CSV output, and no API keys. The SCRAPERAPI_KEY credential is read exclusively from an environment variable (or the --api-key flag) — there are no secrets in the source. Everything the scraper collects comes from Aqar.fm's public, publicly-listed for-sale ads; run it against your own key and it reconstructs the dataset locally into output/.

License

MIT — see LICENSE.