🧵 Threads-Research-Scraper-Skill

A powerful, agent-ready AI Skill and Playwright scraper for researching and extracting live posts, user engagement, and child comment/reply threads on Meta Threads (threads.net).

---

🤖 Using as an AI Skill (Recommended)

This repository is designed as a native AI Skill for agentic coding assistants (such as Google Antigravity, Gemini CLI, Claude, Cursor, and custom LLM agent tools).

When installed as a skill, AI agents automatically discover and execute this skill via SKILL.md whenever a user asks to research or scrape Threads topics, hashtags, user posts, or comment sections.

---

📥 How to Install This Skill for Your AI Assistant

Method 1: Install via NPX (Fastest)

Run the following command in your terminal to automatically install the skill into your AI environment:

npx skills add KarlYu130/Threads-Research-Scraper-Skill

Method 2: Git Clone to Workspace (Project Skill)

Clone this repository directly into your active workspace or project skills folder:

git clone https://github.com/KarlYu130/Threads-Research-Scraper-Skill.git

Method 3: Global AI Assistant Skill Directory

To make this skill available across all agent sessions globally:

# For Antigravity / Gemini CLI:
git clone https://github.com/KarlYu130/Threads-Research-Scraper-Skill.git ~/.gemini/skills/Threads-Research-Scraper-Skill

⚡ Prerequisites & Dependencies

Ensure Python and Playwright browser dependencies are installed in your shell environment:

pip install playwright
playwright install chromium

---

💡 How AI Agents Trigger This Skill

Once installed, simply talk to your AI agent using natural language:

• 💬 "Can you research recent posts and comments about HKUST on Threads?"
• 💬 "Scrape the latest posts for hashtag #AI and summarize user sentiment."
• 💬 "Extract posts and reply threads chronologically for topic 'Hong Kong tech'."

The AI agent will read SKILL.md, execute scripts/threads_live_scraper.py, parse the generated JSON datasets, and present markdown reports with direct data file links!

---

🌟 Key Features

• 💬 Deep Post & Comment Scraping: Navigates directly into post URLs (threads.net/@user/post/...) to extract the main post and all child reply threads.
• ⚡ Top & Chronological Sorting: Supports sorting search results by Top/Relevance (--sort top) or Latest/Chronological (--sort latest, newest first).
• 🍪 Automatic Cookie Handling: Dismisses cookie consent popups for seamless headless execution.
• 📊 Structured JSON Export: Saves extracted raw posts and comments into clean JSON output files.

---

🐍 Standalone Python / CLI Usage

If you prefer to run the scraper directly from your command line without an AI agent:

1. Basic Post & Comment Scraping

python scripts/threads_live_scraper.py --query "HKUST"

2. Multi-Topic & Latest Chronological Sorting

python scripts/threads_live_scraper.py --query "HKUST" "香港科技大學" --sort latest --output latest_hkust.json

---

📁 Repository Structure

Threads-Research-Scraper-Skill/
├── SKILL.md                          # Main AI Skill instruction specification
├── README.md                         # Installation & usage documentation
├── .gitignore                        # Git ignore rules for scraped datasets
└── scripts/
    └── threads_live_scraper.py      # Playwright Chromium scraper engine

---

📜 License

Distributed under the MIT License. See LICENSE for details.