compete - Competitive Intelligence Platform

Watch every move your competitors make. compete runs scheduled agents over a configurable list of competitors (websites, blogs, pricing pages, news, job postings), extracts structured signals with an LLM, warehouses everything in DuckDB, detects meaningful changes, and serves a polished, responsive dashboard with auto-generated weekly reports.

➜ Live demo  ·  Docs  ·  Write-up

The public demo is fully server-less: every API response is baked to static JSON and served from Vercel at $0 (see ADR 0004). A weekly GitHub Action refreshes the data. Config-driven collection (static/dynamic/RSS/jobs), snapshot-diff change detection, provider-agnostic LLM extraction, a dbt warehouse (staging → marts + tests), a typed FastAPI, a light/dark Next.js dashboard, and weekly markdown→PDF reports - all on free/local tooling.

Screenshots

Overview (dark)	Competitor detail (dark)

Changes (light)	Mobile (dark)

More in docs/screenshots/ (reports, settings, light/dark).

Tech stack

Layer	Tooling
Collection	Python, httpx + selectolax, trafilatura, Playwright, feedparser
LLM extraction	provider-agnostic (Gemini Flash default · Groq · Ollama · mock), instructor structured outputs
Storage	DuckDB (file-based warehouse) + Parquet raw layer
Transform	dbt-core + dbt-duckdb (staging → marts, tests, docs)
API	FastAPI (typed, OpenAPI)
Frontend	Next.js + TypeScript + Tailwind + shadcn/ui + Tremor

Optimized for free / open-source tooling throughout.

Quickstart

Requires uv (handles Python 3.12 automatically).

# 1. Install dependencies into a local venv (all extras for the full demo)
uv sync --extra dev --extra dbt --extra api --extra report

# 2. Configure environment
cp .env.example .env            # Windows: copy .env.example .env

# 3. Initialize the warehouse + mirror competitor config
uv run compete init-db
uv run compete sync-competitors

# 4. Collect (fetch tracked URLs → store snapshots), then inspect
uv run compete collect
uv run compete status

# 5. Run the test suite
uv run pytest

No LLM key is needed for Phases 0-1. Extraction (Phase 2) uses Gemini Flash by default; set GOOGLE_API_KEY in .env, or switch COMPETE_LLM_PROVIDER to groq / ollama / mock.

One-command demo (no keys)

Seed curated, realistic data (5 competitors, signals across all types, pricing history, hiring, weekly reports), build the marts, and start both the API and the dashboard - no API key or live scraping required:

uv run python scripts/demo.py        # or: make demo

Then open http://localhost:3000 (dashboard) and http://127.0.0.1:8000/docs (API). The script runs npm install for the web app on first use.

Prefer the pieces separately:

uv run python scripts/seed_demo.py --build   # seed + build marts
uv run compete-api                            # API on :8000
cd web && npm install && npm run dev          # dashboard on :3000

Key endpoints: GET /stats/overview, GET /signals, GET /changes, GET /competitors, GET /competitors/{id}/pricing-history|hiring|cadence, GET /reports, GET /reports/{id}/pdf, POST /pipeline/run.

Run the real pipeline

uv run compete run-all                 # sync → collect → extract → dbt → report
uv run compete run-all -p mock -n 5    # offline (mock LLM), capped for a quick demo

Set GOOGLE_API_KEY (Gemini, default) or switch COMPETE_LLM_PROVIDER to groq / ollama / mock. No key is needed for collection or the mock path.

Add a competitor

Edit config/competitors.yaml - no code changes required:

competitors:
  - id: example
    name: Example Inc
    domain: example.com
    industry: SaaS
    tier: 1
    tracked_urls:
      - url: https://example.com/blog
        source_type: static      # static | dynamic | rss | jobs
        signal_hint: blog_post

Then uv run compete sync-competitors && uv run compete collect -c example.

Repository layout

pipeline/    collect · extract (llm + embeddings) · detect · report · storage · transform · cli
warehouse/   dbt project (staging → marts, tests, docs)
api/         FastAPI app
web/         Next.js dashboard
config/      competitors.yaml (config-driven)
scripts/     demo · seed_demo · screenshots · inspect_marts
docs/        ARCHITECTURE · DATA_MODEL · AGENTS · RUNBOOK · DEPLOYMENT · adr/

Scraping ethics

Public pages only; honors robots.txt (including Crawl-delay) by default; polite per-host throttling; honest User-Agent; never scrapes behind logins.

Free-tier cost

Component	Service	Cost
Pipeline schedule	GitHub Actions cron	Free
LLM extraction	Gemini Flash / Groq free tier · Ollama/mock local	$0
Embeddings	local hashing (default) or MiniLM	$0
Warehouse	DuckDB + Parquet (file)	$0
API host	none — responses baked to static JSON (or Oracle/Fly.io for live mode)	$0
Dashboard	Vercel Hobby (static)	$0

$0 with default/local providers. The public demo needs no backend at all - see DEPLOYMENT and ADR 0004.

Documentation

• ARCHITECTURE - system diagram, data flow, component responsibilities
• DATA_MODEL - every table, lineage, dbt docs
• AGENTS - extraction schema, prompt, retry loop, providers, cost controls
• RUNBOOK - add a competitor, backfill, fix a scraper, rate limits
• DEPLOYMENT - GitHub Actions, Vercel, API host, env/secrets
• ADRs - DuckDB vs Postgres · snapshot-diffing · deterministic pipeline · static-snapshot deploy
• CONTRIBUTING - setup and quality gates

License

Released under the MIT License.