alpha-xtract

X bookmarks → either summarized into markdown digests, or run through an LLM-driven investment research pipeline.

What it does. Pull the bookmarks you've curated on X (formerly Twitter) and feed them through one of two workflows:

• Summarize — turn bookmarks into a markdown digest (combined or per-post), with an optional lightweight topic filter and a JSON state tracker so each run only covers what's new.
• Markets — classify each post (company / industry / macro / event / ignore), generate a markdown research note for each one worth writing up, and maintain a searchable reports/index.md that groups every note by category + ticker / theme.

---

Core features

Unified (powers both modes)

• Bookmark polling — full-library or per-folder, via OAuth 2.0 PKCE (bookmarks are private and need user-context auth).
• Thread / quote / parent expansion — parent, quoted, and reply-chain context is stitched together before the LLM sees the post.
• Pluggable LLM provider — Anthropic, OpenAI, and Gemini APIs supported, plus the local Claude Code CLI (subscription-billed, no API key). A single llm.provider / llm.model pair drives every call.
• SQLite by default, JSON optional — same interface, swap via config.
• Cron-friendly — one shell script per workflow, runs idempotently.

Summarizer (xresearch summarize / mode: summarize)

How bookmarked posts become a digest. For the full pipeline with every stage, decision branch, and output shape, see the detailed summarize diagram.

• Bookmark digests — turn bookmarks into a markdown digest, combined into one file or split per-post.
• Lightweight topic filter — an optional yes/no relevance pass so a digest only covers a theme you name.
• Customizable filter + prompt — pin a standing topic and tailor the relevance filter and digest prompts to it (e.g. always filter for AI-infrastructure posts, then summarize them with your own prompt). See docs/summarize.md.
• Incremental state tracker — a JSON tracker so each run only covers what's new. See docs/summarize.md.

Markets pipeline (xresearch run / mode: markets)

How a bookmarked post becomes a research report. For the full pipeline with every stage, decision branch, and output folder layout, see the detailed markets diagram.

• LLM classification — JSON-only output: company_specific, industry_focused, global_macro, event_driven, or ignore.
• Optional image analysis — when enabled, a bookmarked post's photos are fetched in-memory and sent to the multimodal LLM at classify time (off by default; nothing is stored).
• Category-specific report prompts — different templates for company, industry, macro, and event notes.
• Markdown reports on disk — per-topic subfolders with room for xlsx models, notebooks, and side research.
• Optional Excel models — for posts flagged needs_model, an LLM authors a Jupyter notebook that runs in a sandbox (bwrap on Linux, or the lighter macOS sandbox-exec) to produce model.xlsx, then a reviewer LLM grades it (off by default).
• Deterministic reports/index.md — rebuilt from stored metadata on every run.

⚠️ Security — read before enabling Excel models. The notebook that builds each model.xlsx is LLM-authored from your bookmark content, which you do not control — treat it as untrusted, prompt-injectable code. The sandbox is the only thing stopping a malicious notebook from reading .env / data/x_tokens.json and exfiltrating them. So, if you turn modeling on:

• Keep require_sandbox: true on any host with secrets. require_sandbox: false runs the notebook with full host access — only acceptable on a throwaway machine.
• bwrap (Linux) is the recommended production backend. sandbox-exec (macOS) is a lighter, install-free option for local Mac dev: it denies your secret stores ($HOME, .env, tokens, ~/.ssh, ~/.aws) and defaults network off, but grants broader macOS system reads than bwrap — so keep secrets out of /usr/local and /opt/homebrew, or use bwrap. See docs/sandboxing.md.
• Leave network_access at the backend default unless a model truly needs live data. Enabling egress lets a prompt-injected notebook phone home with anything it can reach.

Details: docs/sandboxing.md · SETUP.md.

---

Quickstart

First-time user? Follow SETUP.md for the full walkthrough — creating your X developer account, configuring the OAuth 2.0 app, and picking an LLM provider. The commands below are the short version for users who already have credentials in hand.

# 1. clone + env
git clone <this-repo> alpha-xtract && cd alpha-xtract
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# 2. credentials
cp .env.example .env                       # then edit with your X + LLM keys
cp config/run_config.example.yaml config/run_config.yaml
cp config/pipeline_config.example.yaml config/pipeline_config.yaml

# 3. validate + one-time OAuth handshake (opens a browser)
xresearch validate-config --config config/run_config.yaml
xresearch auth --config config/run_config.yaml

# 4. first run
xresearch run --config config/run_config.yaml

Reports land under reports/<category>/<topic>/<date>__<slug>__<post_id>/report.md, and reports/index.md is rebuilt automatically.

---

Credentials

Both are covered step-by-step in SETUP.md:

• X API — bookmarks are private, so they need OAuth 2.0 user-context tokens (the app-only Bearer Token / OAuth 1.0a key won't work). xresearch auth does a one-time browser handshake and persists a refresh-able token pair to data/x_tokens.json (chmod 600).
• LLM provider — the default config uses Anthropic (claude-opus-4-8). OpenAI and Gemini also work; set llm.provider / llm.model / llm.api_key_env in config/pipeline_config.yaml and the matching key in .env. The claude_code provider instead shells out to the locally-authenticated Claude Code CLI — usage bills to your Claude subscription, no API key needed. OpenAI's Codex CLI is not set up as an equivalent provider; OpenAI is only available via the metered API.

One model for every call: a single llm.provider / llm.model pair drives the entire pipeline — classification, report writing, Excel-model building, and model review all use the same model. If you want smaller/cheaper models for some stages (e.g. a light model for classification, a heavyweight one for reports), that requires altering both the config schema and the get_llm wiring, not just the config files.

---

Running it

• Commands, run modes (--dry-run / --skip-fetch), the human-in-the-loop manual command, and author/date filters — see docs/usage.md.
• Summarizing bookmarks into a digest (xresearch summarize / mode: summarize) — see docs/summarize.md.
• Weekly scheduling (cron) — see SETUP.md Part 3.
• Every config knob — see docs/config.md.

Guided skills (Claude Code)

Two interactive skills under .claude/skills/ wrap the runs in a short Q&A so you don't have to remember the flags. Each does a pre-flight peek at your bookmarks first, then asks a few questions, runs the right command, and shows the results back in the chat.

Skill	Wraps	Asks about
markets-run	xresearch run / backfill (mode=markets)	scope (incremental / date / authors / backfill), execution mode (full / dry-run / skip-fetch), Excel models, image analysis
bookmark-summary	xresearch summarize	how far back, topic filter, combined-vs-per-post detail, tracked vs one-off

Invoke them from Claude Code by name (e.g. "run the markets pipeline" or "summarize my bookmarks").

---

Documentation

Doc	Read it when
SETUP.md	First-time setup: X account, OAuth, LLM provider, cron.
docs/usage.md	Day-to-day CLI: commands, run modes, filtering.
docs/summarize.md	The bookmark-digest workflow: summarize command / mode: summarize, topic filtering, state tracker.
docs/config.md	What each config key does + the file layout.
docs/architecture.md	5-minute tour of how a bookmark becomes a report, plus the on-disk layout.
docs/storage.md	DB schemas / JSON layout / writing a new backend.
docs/prompts.md	Customizing what the LLM sees and how reports are structured.
docs/sandboxing.md	Optional Excel modeling setup: the bwrap (Linux) and sandbox-exec (macOS) backends, the network toggle, .venv-modeling, and the sandbox security model.

---

Testing

pytest -q

Unit tests cover config parsing, slug / filename generation, markdown writing, both storage backends, dedupe rules, classifier coercion, the indexer, media fetch, and the sandboxed model builder / reviewer. tests/integration/test_runner.py drives run_once end-to-end with a mocked X client and a deterministic stub LLM.

---

License

MIT — see LICENSE.