feat: search.open page reading via Jina Reader + top-10 search default - #112
feat: search.open page reading via Jina Reader + top-10 search default#112faresobeid wants to merge 2 commits into
Conversation
…with Jina Reader - search: accept a single query or a list batched into one Serper call; send num explicitly; default 10 results per query (was 5) - new open_webpage built-in skill: Jina Reader API when JINA_API_KEY is set (markdown, JS-rendered pages), direct fetch with local HTML/PDF parsing (bs4/pdfminer) otherwise - prompt hints for batched search and open_webpage; README + env table Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| headers={"Authorization": f"Bearer {api_key}", "X-Timeout": str(int(timeout))}, | ||
| ) | ||
| response.raise_for_status() | ||
| return response.text.strip() |
There was a problem hiding this comment.
Jina request drops URL query
Medium Severity
When JINA_API_KEY is set, _fetch_jina builds the Reader request as https://r.jina.ai/ plus the raw target URL. If the target includes a ? query string, HTTP parsing treats everything after the first ? as query parameters on r.jina.ai, not as part of the embedded page URL, so Jina may fetch the wrong resource or apply unintended Reader options while still returning 200.
Reviewed by Cursor Bugbot for commit bbb518e. Configure here.
- search.run is single-query again (top-10 default kept, num sent explicitly) - page reading is search.open on the same skill instead of a separate open_webpage built-in: Jina Reader when JINA_API_KEY is set, direct fetch (bs4 HTML / pdfminer PDF) otherwise - built-in stubs re-export the skill module's __all__ so extra public functions ride along; kernel logs every public async function call - prompt hint restores the asyncio.gather fan-out and teaches search.open Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
faresobeid
changed the title
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2a25b0d. Configure here.
| def _fetch_jina(url: str, api_key: str, timeout: float) -> str: | ||
| response = httpx.get( | ||
| f"{JINA_READER_URL}/{url}", | ||
| timeout=timeout, |
There was a problem hiding this comment.
Jina URL query mangling
Medium Severity
_fetch_jina builds the Reader GET URL by interpolating the target URL after https://r.jina.ai/. Characters such as ? and & in typical result links are treated as the Reader request’s own query string, so the page Jina fetches may omit query parameters. With JINA_API_KEY set, agents can get wrong or incomplete text without falling back to direct fetch.
Reviewed by Cursor Bugbot for commit 2a25b0d. Configure here.
2 participants
Summary
Upgrades the built-in
searchskill for deep-research runs:search: top-10 defaultnumis now sent explicitly in the Serper payload and the default is 10 results (was 5, sliced client-side from Serper's default page).search.open: page reading on the same skillsearchskill now also exposesawait search.open(url="..."), which fetches a URL and returns its text. WhenJINA_API_KEYis set, pages go through the Jina Reader API (markdown output, handles JS-rendered pages), falling back to a direct fetch on failure. Without a key it's a plain direct fetch: HTML parsed with BeautifulSoup/lxml (already in the base toolkit), PDFs withpdfminer.six(added to dependencies).RLM_SKILLS=searchenables both, so existing verifiers v1 configs need no changes — just exportJINA_API_KEY(optional) alongsideSERPER_API_KEY.Mechanism
__all__(from rlm.skills.search import *) instead of justrun, so extra public functions ride along on the wrapped callable module (search.run,search.open—await search(...)shorthand unchanged).run) toprogrammatic_tool_calls.jsonl, sosearch.opencalls count in tool-call metrics under the skill's name.asyncio.gather(search(...), search(...)), read a promising result in full withsearch.open.Validation
uv run pytest tests/— 88 passed (includes the real-kernel skill tests exercising the wrapper change); ruff check + format clean.import *→_CallableModule→ logged functions):await search(...),await search.run(...), andawait search.open(...)all work and all log under tool namesearch.search.open: HTML page, a real arXiv PDF (39.7k chars extracted), and the Jina→direct fallback with an invalid key.num=10— worth one real rollout before relying on them at scale.🤖 Generated with Claude Code