Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 22 additions & 83 deletions external/basic/firecrawl-search/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,16 @@
---
name: firecrawl-search
description: |
Web search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find recent news, discover sources, or says "search for", "find me", "look up", "what are people saying about", or "find articles about". Returns real search results with optional full-page markdown — not just snippets. Provides capabilities beyond Claude's built-in WebSearch.
Web search with full page content. Use when no URL is known: finding sources, articles, or news. For papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl search

Web search with optional content scraping. Returns search results as JSON, optionally with full page content.

## When to use

- You don't have a specific URL yet
- You need to find pages, answer questions, or discover sources
- First step in the [workflow escalation pattern](firecrawl-cli): search → scrape → map → crawl → interact

## Quick start

```bash
Expand All @@ -28,53 +22,18 @@ firecrawl search "your query" --scrape -o .firecrawl/scraped.json --json

# News from the past day
firecrawl search "your query" --sources news --tbs qdr:d -o .firecrawl/news.json --json

# Programming question: search GitHub issues, merged PRs, READMEs, and docs
firecrawl search "your query" --categories developer -o .firecrawl/developer.json --json
```

## Developer search

`--categories developer` adds an index built for coding agents. It covers GitHub
issues, merged pull requests, repository READMEs, and curated documentation
sites. Use it for a programming question: an error message, an API contract, a
library behaviour, or a known bug.

The hits arrive in their own `data.developer` group beside `data.web`. Each hit
holds `url`, `title`, and `description`, where `description` is the matched
passage. Read the passages with
`jq -r '.data.developer[] | .url, .description' .firecrawl/developer.json`.

The dedicated `firecrawl developer` command searches only that index and keeps
the full matched passages:
Run `firecrawl search --help` for the full option list.

```bash
# Developer search only, with full passages
firecrawl developer "your query" --limit 10 -o .firecrawl/developer.json --json
```
`--categories developer` weighs the developer index beside ordinary web results in this same call (no passage control, no index filters). `--categories research` is a website filter, not the paper index. Dedicated skills: [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md) and [firecrawl-research-index](../firecrawl-research-index/SKILL.md).

Each result holds `id`, `type` (`issue`, `pull_request`, `readme`, `doc`),
`url`, `title`, and `passages`. Read them with
`jq -r '.results[] | .url, .passages[].text' .firecrawl/developer.json`.

## Options

| Option | Description |
| ---------------------------------------------- | --------------------------------------------- |
| `--limit <n>` | Max number of results |
| `--sources <web,images,news>` | Source types to search |
| `--categories <github,research,pdf,developer>` | Filter by category |
| `--tbs <qdr:h\|d\|w\|m\|y>` | Time-based search filter |
| `--location` | Location for search results |
| `--country <code>` | Country code for search |
| `--scrape` | Also scrape full page content for each result |
| `--scrape-formats` | Formats when scraping (default: markdown) |
| `-o, --output <path>` | Output file path |
| `--json` | Output as JSON |
**Done when:** results are saved under `.firecrawl/`, verified non-empty, processed for the request, and one feedback event is sent within the time window (unless opted out).

## Tips

- **`--scrape` fetches full content** — don't re-scrape URLs from search results. This saves credits and avoids redundant fetches.
- **`--highlights` on by default:** results are query-relevant excerpts, not full-page snippets. Use `--no-highlights` for the original snippets.
- **`--scrape` fetches full content** — reuse that content instead of re-scraping result URLs. This saves credits and avoids redundant fetches.
- Always write results to `.firecrawl/` with `-o` to avoid context window bloat.
- Use `jq` to extract URLs or titles: `jq -r '.data.web[].url' .firecrawl/search.json`
- Naming convention: `.firecrawl/search-{query}.json` or `.firecrawl/search-{query}-scraped.json`
Expand All @@ -88,7 +47,7 @@ Search costs 2 credits. After you've actually used the results (or decided they
**Rules to know before you call this:**

- **Time window:** must be sent within ~2 minutes of the search. Late feedback is rejected.
- **`--missing-content` is the most important field.** It's a list of _specific pieces_ of content you expected but did not find. One topic per entry — do not pack multiple topics into one string. These aggregate across teams and tell us what to index next.
- **`--missing-content` is the most important field.** It's a list of _specific pieces_ of content you expected but did not find. One topic per entry, each in its own string. These aggregate across teams and tell us what to index next.
- **Substantive content required** (zero-effort feedback is rejected with HTTP 400):
- `good` → must include at least one `--valuable-sources` entry.
- `partial` → must include `--valuable-sources` or `--missing-content`.
Expand All @@ -97,42 +56,19 @@ Search costs 2 credits. After you've actually used the results (or decided they
- **Idempotent:** re-submitting for the same search id returns success but no extra refund.
- **`--silent &`** is the right pattern — exit code 0 even on failure, so a rejected/expired call never crashes your pipeline.

Read the search response's `id`:

```bash
SEARCH_ID=$(jq -r '.id' .firecrawl/search-react-hooks.json)
```

Then send feedback. Pick the rating that matches what actually happened:
Verify the search returned results before reading its `id`. Zero-result searches write no output file, so the file may be missing — or left over from an earlier search. The guard below skips feedback when the file is missing or has zero results; call `search-feedback` only inside it:

```bash
# Results were useful, with notes on what was still missing
firecrawl search-feedback "$SEARCH_ID" \
--rating good \
--valuable-sources '[{"url":"https://react.dev/reference/react/hooks","reason":"Most authoritative"}]' \
--missing-content '[
{"topic":"useDeferredValue","description":"No example of useDeferredValue with Suspense"},
{"topic":"useTransition","description":"No coverage of useTransition for routing"}
]' \
--query-suggestions "Boost react.dev for queries about react hooks" \
--silent &

# Results were partially useful — multiple missing topics, one entry per topic
firecrawl search-feedback "$SEARCH_ID" \
--rating partial \
--missing-content '[
{"topic":"useDeferredValue"},
{"topic":"useTransition","description":"Need React 18+ examples"},
{"topic":"Server Components hooks"}
]' \
--silent &

# Quick form — repeat --missing-content or use comma-separated topics
firecrawl search-feedback "$SEARCH_ID" \
--rating bad \
--missing-content "official api reference: missing v2 endpoints" \
--missing-content "code examples in python" \
--silent &
# Send once per search. Rate honestly and replace the placeholder with the
# rating that matches what actually happened. The two fields shown
# satisfy the substantive-content rule for every rating.
if SEARCH_ID=$(jq -er 'select(any(.data[]; length > 0)) | .id' .firecrawl/search-react-hooks.json); then
firecrawl search-feedback "$SEARCH_ID" \
--rating "<good|partial|bad>" \
--valuable-sources '[{"url":"https://react.dev/reference/react/hooks","reason":"Most authoritative"}]' \
--missing-content '[{"topic":"useDeferredValue","description":"No example of useDeferredValue with Suspense"}]' \
--silent &
fi
```

**`--missing-content` accepts:**
Expand All @@ -149,3 +85,6 @@ firecrawl search-feedback "$SEARCH_ID" \
- [firecrawl-scrape](../firecrawl-scrape/SKILL.md) — scrape a specific URL
- [firecrawl-map](../firecrawl-map/SKILL.md) — discover URLs within a site
- [firecrawl-crawl](../firecrawl-crawl/SKILL.md) — bulk extract from a site
- [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md) — issues, merged PRs, READMEs, and docs
- [firecrawl-research-index](../firecrawl-research-index/SKILL.md) — published papers, not `search --categories research`
- [firecrawl-build-search](https://github.com/firecrawl/skills/tree/main/skills/build/firecrawl-build-search) — building search into an app instead of running it here
2 changes: 1 addition & 1 deletion skills-lock.json
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@
"source": "firecrawl/cli",
"sourceType": "github",
"skillPath": "skills/firecrawl-search/SKILL.md",
"computedHash": "4c512eb1a73efbf1f961929ee1da490c9d424c7ee1a1dfea8c195a3b860717c0"
"computedHash": "cd13571285ba1f2f1a08f0d3786dac4430f50ace76985817b7c7ddbf276ebaa9"
},
"firstsun-harness": {
"source": "firstsun-dev/skills",
Expand Down