Skip to content

feat(research): deep_research tool β€” autonomous multi-step web research (#293)#294

Merged
mattmezza merged 4 commits into
mainfrom
feat/deep-research
Jul 10, 2026
Merged

feat(research): deep_research tool β€” autonomous multi-step web research (#293)#294
mattmezza merged 4 commits into
mainfrom
feat/deep-research

Conversation

@mattmezza

Copy link
Copy Markdown
Owner

Closes #293.

What

A new deep_research native tool: plan β†’ search β†’ read β†’ iterate β†’ synthesize, as a deterministic pipeline (core/deep_research.py) rather than a free-form agent loop.

  1. Plan β€” the question is decomposed into up to 5 focused search queries (fast model)
  2. Search β€” each query runs through the existing web_search tool (SearXNG or Tavily), URLs deduped across cycles
  3. Read — plain HTTP fetch (httpx, already a dep) + stdlib HTML→text; no Playwright spin-up. A failed fetch falls back to the search snippet
  4. Extract β€” per-source notes on the fast model, concurrently; irrelevant sources dropped
  5. Iterate β€” up to depth cycles: a gap-analysis call produces follow-up queries, or stops early
  6. Synthesize β€” a structured Markdown report with inline [n] citations

Decisions (per the issue discussion)

  • Models β€” sub-calls (plan/extract/gap) run on a configurable fast/cheap model (default deepseek / deepseek-v4-flash); synthesis defaults to the main agent model (honouring per-agent LLM overrides via _agent_llm), with an optional dedicated synthesis provider/model. All in the Tools tab.
  • Source reading β€” lightweight HTTP fetch, not the browser tool.
  • Result format β€” the report is published as a themed web artifact (artifacts/research-<slug>-<hex>/index.html, one shared look and feel across runs) and the tool instructs the model to reply briefly with takeaways + link. Without a servable workspace, the full Markdown report rides back in the tool result for the model to relay.
  • Cancellation β€” the pipeline runs inside the turn's tool call and polls the turn's abort Event between phases, so /stop (and the Stop button) cancels it. Inside a background subagent, task.cancel() propagates through the awaits as usual.
  • Cost guardrails β€” hard per-run token_budget (default 300k, counted from real usage), plus depth and max_sources ceilings the model may dial down but never exceed (resolve_cap, same semantics as subagents).
  • Search backend β€” provider-agnostic: goes through _tool_web_search, so SearXNG and Tavily both work.
  • Streaming progress β€” phase updates (πŸ” Researching…, πŸ“– Reading N sources…, πŸ•³οΈ Following up on gaps…, ✍️ Synthesizing…) are sent to the originating chat.
  • State persistence β€” skipped: the run lives inside one tool call; a restart kills it like any in-flight turn. Making it resumable would need a persistent job store β€” out of scope, per the issue answers.
  • Background runs β€” not a separate mechanism: the tool is offered inside subagent loops too, so spawn_subagent(background=true) with a research task gives a non-blocking run for free.

Wiring

  • DeepResearchToolConfig under tools.deep_research (hot-applied via the existing config PATCH path)
  • Feature-gated like every native tool: hidden unless enabled and web search is ready; per-agent allowlisting via GATEABLE_TOOLS; default permission ALWAYS (read-only; budgets cap the cost)
  • Tools tab card: enable, sub-call provider/model, synthesis provider/model (blank = main agent), depth, max sources, token budget
  • Docs: README feature block + structure tree, admin-ui.mdx, configuration.mdx, permissions.mdx

Tests

12 new tests in tests/test_deep_research.py (pipeline happy path, gap iteration + URL dedup, irrelevant-source drop, token-budget early stop, cancellation, max-sources cap, snippet fallback, HTML/text/slug/report-rendering helpers, feature gating). Full suite: 1079 passed.

…#293)

Plan β†’ search β†’ read (plain HTTP fetch) β†’ iterate on gaps β†’ synthesize a
structured, cited report. Sub-calls run on a configurable fast model;
synthesis defaults to the main agent model. Hard token budget, depth and
max-sources ceilings configurable in the Tools tab; /stop cancels between
phases; progress updates stream to the originating chat; report is
published as a themed web artifact when the workspace is enabled.
- expanduser on the artifact write path so a '~/…' workspace serves what
  was written (matches artifacts.serving_base)
- only URLs actually read are marked seen β€” candidates dropped by the
  max_sources cap stay eligible for later cycles
- per-source fetch+extract run as one independent chain (no cross-source
  barrier stalling extraction on the slowest fetch)
- extraction failures are logged instead of silently dropped
- '##' report sections render as the styled h2 (was h3, inverting the
  artifact's visual hierarchy)
- deep_research honours the per-agent web_search scope (hidden from the
  offer and refused in the handler)
- clear upfront error when the configured sub-call/synthesis provider has
  no API key (instead of an opaque auth failure mid-run)
- Tools tab: one get_many read + defaults sourced from the pydantic model;
  provider dropdowns reuse the shared LLM_PROVIDERS catalog
- dead in_list bookkeeping removed from the report renderer; docstring/UI
  note that background subagent runs cancel from Jobs, not /stop
- dedupe set stores the normalized (stripped) URL, matching the check
- provider-key guard skips the synthesis provider when it isn't actually
  used (blank synthesis model = main agent LLM) and accepts base_url-only
  keyless sidecar setups
@mattmezza

Copy link
Copy Markdown
Owner Author

Self-review done (two passes). Fixed: artifact path expanduser, seen-URL starvation across cycles, per-agent web_search scope bypass, missing-provider-key upfront error (base_url-only sidecars still allowed), silent extraction failures, report heading hierarchy, fetch/extract pipelining, Tools-tab single get_many + defaults sourced from the pydantic model, shared provider dropdown catalog. 1080 tests green, ruff clean.

@mattmezza
mattmezza merged commit 00fd57d into main Jul 10, 2026
3 checks passed
@mattmezza
mattmezza deleted the feat/deep-research branch July 10, 2026 20:55
@mattmezza

Copy link
Copy Markdown
Owner Author

Follow-up from a live run: the model's reply omitted the artifact links. The handler now posts the report link straight to the chat (same path as the progress updates) instead of relying on the model to echo it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Deep Research tool β€” autonomous multi-step web research and report synthesis

1 participant