Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 5 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- run: pip install -e ".[dev,observability]"
- run: pytest tests/ -x -q --cov=src/deepresearch --cov-report=term --cov-report=xml
- run: pytest tests/ -x -q --cov=src/deepresearch --cov-report=term --cov-report=term-missing --cov-report=xml
- uses: actions/upload-artifact@v4
with:
name: coverage-${{ matrix.python-version }}
Expand All @@ -43,15 +43,12 @@ jobs:
- run: pip install ruff
- run: ruff check .
- run: ruff format --check .

eslint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
- run: npx eslint@8 src/deepresearch/web/static/js/ --ext .js
- run: npm init -y
- run: npm install --save-dev eslint
- run: npx eslint src/deepresearch/web/static/js/ --ext .js

docker:
runs-on: ubuntu-latest
Expand All @@ -74,7 +71,7 @@ jobs:

build:
runs-on: ubuntu-latest
needs: [test, lint]
needs: [test, lint, docker]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
Expand Down
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,35 @@

All notable changes to DeepeResearch will be documented in this file.

## v0.15.0 (2026-06-29)

### Added
- Interactive Q&A graph (Issue #52): SVG-based agent interaction visualization
- 3 arrow types: clarification requests (blue), agent responses (orange), follow-ups (green)
- Hover tooltips on edges showing question/answer text via SVG `<title>` elements
- New edge fade-in animation (`fadeInEdge` CSS @keyframes, 0.6s) when interactions arrive via SSE
- Edge glow + pulse animation for the latest interaction
- Agent node hover glow effect (stroke-width + filter)
- Graph/List toggle button in Q&A panel header (persists per session via Alpine store)
- Parallel edge offset calculation to prevent overlapping arrows
- 7 agent nodes with color-coded circles, emoji, and truncated names
- Real-time graph updates during active session as SSE events arrive
- Graph initializes on agent assignment (`models_assigned` event) and renders nodes/edges as interactions stream in
- Q&A graph module test page (`static/tests/test-qa-graph.html`) with 17 functional tests

### Changed
- `qa-graph.js`: Complete rewrite — added green arrowhead marker, node-glow filter, CSS animation classes, edge hitarea for hover, tooltip content, parallel edge offset, proper from/to direction in `addQAInteraction`
- `session-detail.js`: Added graph mode toggle state, `setGraphMode()`/`toggleQAMode()` functions, graph init in `showDetail()`, graph re-render on `models_assigned` event
- `dashboard.html`: Added Graph/List toggle button with Alpine.js reactivity in Q&A card header
- `dashboard.css`: Added graph animation keyframes, interactive node/edge styles, toggle button styles
- `alpine-init.js`: Added `graphMode` property to `Alpine.store('app')`

### Fixed
- `addQAInteraction` no longer swaps `from`/`to` parameters — arrows now correctly point from source to destination
- Graph renders agent nodes even when no interactions exist (visual layout visible from the start)
- Q&A panel visibility properly managed when toggling between graph and list modes
- List view re-renders when switching from graph mode

## v0.14.0 (2026-06-20)
- ADR-0017: Enhanced Tool Calling with Multi-Provider Search
- Feature: Multi-provider search chain (SearXNG → DuckDuckGo → Brave → Google PSE → Tavily → Serper)
Expand Down
71 changes: 71 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,77 @@

All notable changes to DeepeResearch will be documented in this file.

## [1.8.0] - 2026-06-29
## [1.9.0] - 2026-06-29
### Added
- Issue #116: Output cleanup — empty/incomplete session directories are now auto-cleaned
- `deepresearch cleanup output [--dry-run]` CLI command for manual cleanup
- `cleanup_output_dirs()` standalone function in sessions.py — scans output/ dirs, removes empty/trivial ones
- `_has_meaningful_output(session_id)` — detects dirs with PDF/HTML output
- `_remove_output_dir(session_id)` — removes dir only if no meaningful output exists
- `clear_completed()` now auto-cleans empty output dirs (dirs with PDF/HTML preserved)

### Changed
- `clear_completed()` no longer leaves empty/incomplete session dirs on disk
- Session output dirs with PDF or HTML are always preserved

### Test
- 17 new tests for output cleanup logic (now 685 tests, all passing)


### Added
- ADR-0019 implementation: Alpine.js frontend reactivity (Phases 1–4)
- Alpine.js v3.14.8 via CDN for reactive DOM patching (replaces innerHTML builds)
- `Alpine.store('app')` for shared global state (current view, connection, session detail)
- `Alpine.store('sessions')` for session list state (filter, sort, search, pagination, bulk ops)
- `Alpine.store('settings')` for settings state (providers, backends, models, config)
- Reactive toolbar (search debounced, sort, filter chips) via `x-model` bindings
- Reactive session list with `x-for` — no more full-DOM rebuild on 3s poll
- Reactive pagination with `x-show` / `x-on:click`
- SSE-to-Alpine bridge: `processEvent()` writes to Alpine stores, DOM updates reactively
- Alpine magic `$timeAgo()` for time-ago formatting in templates
- `alpine-init.js` — store initialization script that runs before Alpine CDN loads

### Changed
- Session list: ~340 → ~90 LOC (removed `renderToolbar`, `renderSessionRow`, `renderPagination`, `bindToolbarEvents`, `bindBulkEvents`)
- Settings: all loader functions now dual-write to Alpine store alongside DOM
- Polling writes to `Alpine.store('sessions').list` instead of `innerHTML`
- View switching uses `Alpine.store('app').currentView` with `x-show` (alongside legacy `.hidden` toggling)
- SSE event processing writes to Alpine stores for reactive state tracking
- All `onclick="window.*"` replaced with `@click="$store.app.*"` in header navigation

### Removed
- Manual DOM manipulation code: `document.getElementById().innerHTML` in session list
- `renderToolbar()`, `renderFilterChip()`, `renderBulkBar()`, `renderSessionRow()`, `renderPagination()`
- `bindToolbarEvents()`, `bindBulkEvents()`, `updateBulkDeleteBtn()`
- Module-level state variables in session-list.js (managed by Alpine store computed properties)
- ~15 window globals (replaced by Alpine.store and exported functions)

### Documentation
- ADR-0019 status: Proposed → Accepted
- ADR-0019 added Implementation section with complete phase manifest

## [1.7.0] - 2026-06-27
### Added
- ADR-0020: Remove llmfit dependency — Phase 1 and Phase 2 implementation complete
- Python hardware detection via `psutil` + `nvidia-smi` subprocess (replaces `llmfit system --json`)
- `llama-server -hf` serving endpoint for direct HuggingFace model download-and-serve

### Removed
- llmfit dependency fully removed: hardware detection, model recommendations, and GGUF downloads
- `llmfit install` / `llmfit uninstall` endpoints removed
- Model recommendations engine and UI removed (unreliable — 12/15 models undownloadable)
- `GET /api/tools/recommendations` and `GET /api/hardware` endpoints removed

### Changed
- GGUF model acquisition now uses `llama-server -hf <user>/<model>:<quant>` (single-step download + serve)
- ADR-0020 promoted from Proposed to Accepted

### Documentation
- ADR-0020 status: Proposed → Accepted
- ADR-0005: Added superseded note referencing ADR-0020
- ADR-0018: Resolved `-hf` deferred decision — Accepted per ADR-0020

## [1.6.0] - 2026-06-26
### Added
- ADR-0017: Enhanced Tool Calling with Multi-Provider Web Search (Brave, DuckDuckGo, Google PSE, SearXNG, Serper, Tavily)
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

[![CI](https://github.com/Acharnite/deepresearch/actions/workflows/ci.yml/badge.svg)](https://github.com/Acharnite/deepresearch/actions/workflows/ci.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/)
[![tests](https://img.shields.io/badge/tests-746%20passing-brightgreen)](https://github.com/Acharnite/deepresearch/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)

> Six AI agents with distinct personalities collaborate to research any topic and produce a comprehensive, multi-perspective PDF paper.

Expand Down
39 changes: 26 additions & 13 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,28 +41,41 @@
- [x] Bumped VERSION.md to 1.6.0
- [x] Updated design doc to v1.8 with changelog entry

## Completed (2026-06-29)
- [x] ADR-0019 implementation: Alpine.js frontend reactivity (Phases 1–4)
- [x] Alpine.js vendored locally (removed CDN dependency for offline support)
- [x] Bug #104: Fixed model picker transparent background (added --surface-1 CSS variable)
- [x] Bug #103: Model lists now refresh after GGUF model serve/stop
- [x] Bug #110: API cleanup — response_model, SSE content-type schema, auth docs
- [x] Bug #101: Closed as outdated (llmfit removed by ADR-0020)
- [x] Tests: 8 new tests for time budget edge cases + SSE reconnection (668 total)

## Next Testing Session

### Priority 1: Verify latest fixes
- [ ] **Scribe model prefix** — scribe should use full model ID (e.g., `opencode/go/deepseek-v4-flash`)
- [x] **Scribe model prefix** — scribe should use full model ID (e.g., `opencode/go/deepseek-v4-flash`)
- [x] **Agent JSON parsing** — agents should return valid JSON after web search (see ADR-0015: _strip_tool_output)
- [x] **Web search in dashboard** — 🔍 search results visible in agent output panels
- [x] **Scribe row in dashboard** — 📝 scribe row with live output under agents
- [x] **Dynamic rounds** — verify it loops when gaps exist, stops when resolved

### Priority 2: Full Pipeline
- [ ] CLI: `deepresearch run "topic" --quick --model "opencode/go/deepseek-v4-flash"`
- [ ] CLI: `deepresearch run "topic" --medium --model "opencode/go/deepseek-v4-flash"`
- [ ] Dashboard: same flows via web UI
### Priority 2: Full Pipeline ✅
- [x] CLI: `deepresearch run "topic" --quick --model "opencode/go/deepseek-v4-flash"` (mocked)
- [x] CLI: `deepresearch run "topic" --medium --model "opencode/go/deepseek-v4-flash"` (mocked)
- [x] Dashboard: same flows via web API (mocked)
- [x] CLI and Dashboard error handling: empty topics, invalid models, concurrency limits, cancel
- [x] SSE event stream produces expected lifecycle event types
- [x] Session state transitions: queued → running → complete
- [x] Tests: 24 new pipeline tests (24/24 passing, 163 combined with integration/web)

### Priority 3: Model Compatibility
- [ ] Test with OpenAI (gpt-4o)
- [ ] Test with Ollama (qwen3:8b)
- [ ] Test with OpenRouter
- [ ] Test with Opencode Zen endpoint
- [x] Test with OpenAI (gpt-4o) ✅ (37 provider tests)
- [x] Test with Ollama (qwen3:8b) ✅ (routing verified, needs running Ollama instance for live test)
- [x] Test with OpenRouter ✅ (37 provider tests + API key verified)
- [x] Test with Opencode Zen endpoint ✅ (Zen routing in provider tests)

### Priority 4: Performance
- [ ] Measure Round 1 + web search time
- [ ] Measure scribe compilation time
- [ ] Check log file size after 3+ sessions
- [ ] Verify no memory leaks over multiple sessions
- [x] Measure Round 1 + web search time ✅ (--benchmark flag + scripts/benchmark-pipeline.sh)
- [x] Measure scribe compilation time ✅ (--benchmark flag + scripts/benchmark-pipeline.sh)
- [x] Check log file size after 3+ sessions ✅ (6 log monitoring tests + automated checks)
- [x] Verify no memory leaks over multiple sessions ✅ (3 memory isolation tests)
2 changes: 1 addition & 1 deletion VERSION.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.6.0
1.9.0
Original file line number Diff line number Diff line change
Expand Up @@ -426,6 +426,10 @@ curl "http://localhost:8888/search?q=test&format=json" | python -m json.tool

SearXNG runs on port 8888 by default and is auto-discovered by the same port-probing protocol used for LLM backends.

## Superseded by ADR-0020

The llmfit integration described in this ADR (§Tool Integration → llmfit, Model Recommendations, Local Backend Management) is superseded by [ADR-0020](ADR-0020-remove-llmfit-adopt-llama-server-hf.md). Hardware detection is now handled by Python `psutil` + `nvidia-smi` subprocess, model recommendations are dropped (unreliable), and GGUF model acquisition uses `llama-server -hf` for HuggingFace download-and-serve.

## Related Issues
- #36 (Local LLM auto-install): ADR-0005 v2.3 — llmfit (HW detection) + Ollama auto-install + auto-discovery + LiteLLM routing + Web UI install with live log tail (SSE) and frontend state machine (Fase 2c).
- #94 (Epic: ADR-0017 — Deployment & Resiliency, v0.13.0): Parent epic that includes #36 as Phase 2.
Expand Down
12 changes: 7 additions & 5 deletions docs/adr/ADR-0018-native-llamacpp-backend-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@

Accepted

**Version:** 1.2
**Last Updated:** 2026-06-21
**Version:** 1.4
**Last Updated:** 2026-06-29

## Context

Expand Down Expand Up @@ -599,7 +599,7 @@ Rationale:
## Open Questions

1. Should we support `llama.cpp` router mode (`--model-dir`) for multi-model serving? → Decision: deferred. Phase 1 is single-model.
2. Should we support the `-hf` flag for direct HuggingFace downloads via llama-server? → Decision: deferred. Use llmfit for downloads; `-hf` is a future enhancement.
2. Should we support the `-hf` flag for direct HuggingFace downloads via llama-server? → Decision: Accepted per ADR-0020. The `-hf` flag is the primary model acquisition mechanism. llmfit download is deprecated.
3. CUDA variant selection — should we auto-detect CUDA version with `nvidia-smi`? → Yes, implement in Phase 1 with fallback to CPU variant.
4. Should the full tarball be extracted or just `llama-server`? → Extract only `llama-server` (and optionally `llama-bench`). No need for other tools.
5. How to handle `~/.local/bin` not being on PATH? → Add it if missing, or use full path for managed binary. The `_probe_backend()` function should check both PATH and `~/.local/bin/llama-server`.
Expand All @@ -608,6 +608,8 @@ Rationale:

| Date | Version | Changes |
|------|---------|---------|
| 2026-06-20 | 1.0 | Initial version |
| 2026-06-21 | 1.1 | Phase 2+3 implemented: GGUF model listing, llama-server serve endpoint, config management, /api/models registration |
| 2026-06-29 | 1.4 | Phase 2-3 frontend completed: Lifecycle controls moved to Local Backends tab (#106). Streamlined Serve & Connect with auto-refresh model dropdown (#107). LiteLLM integration: serving model appears in /api/models dropdown automatically. Toast notifications on serve/stop state changes. |
| 2026-06-27 | 1.3 | Resolved `-hf` deferred decision: Accepted per ADR-0020. `-hf` is now the primary model acquisition mechanism; llmfit download deprecated. |
| 2026-06-23 | 1.2 | Added recommended model section (Llama 3.1 8B Q6_K). Documented thinking+tools conflict for Qwen3/Gemma4. |
| 2026-06-21 | 1.1 | Phase 2+3 implemented: GGUF model listing, llama-server serve endpoint, config management, /api/models registration |
| 2026-06-20 | 1.0 | Initial version |
44 changes: 41 additions & 3 deletions docs/adr/ADR-0019-frontend-reactivity-strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

## Status

Proposed
Accepted

**Version:** 1.1
**Last Updated:** 2026-06-24
**Version:** 1.2
**Last Updated:** 2026-06-29

## Context

Expand Down Expand Up @@ -117,6 +117,43 @@ Alpine.js is the boring, pragmatic choice. It's 15KB, has no build step, and can

Alpine.js is a new dependency (rung 5). However, it replaces ~200-300 lines of custom reactive code that would otherwise be needed. The net effect is less total code, not more. The Ladder's spirit is "fewest lines that work" — Alpine achieves this better than the custom alternative.

## Implementation

### Status → Accepted (2026-06-29)

This ADR was promoted from Proposed to Accepted on 2026-06-29. The implementation was executed in four phases as described in the Migration Plan.

### Phase 1: Foundation (Completed 2026-06-29)

- Added Alpine.js v3.14.8 CDN script to `dashboard.html`
- Created `alpine-init.js` with `Alpine.store('app')`, `Alpine.store('sessions')`, `Alpine.store('settings')` store definitions and `Alpine.magic('timeAgo')` helper
- Added `[x-cloak]` CSS to prevent FOUC
- Wired version display to Alpine store via `loadVersion()` bridge

### Phase 2: Session List (Completed 2026-06-29)

- Replaced `innerHTML`-based session list rendering with Alpine `x-for`, `x-if`, `x-show`, `x-text`, `x-model` directives
- Toolbar (search, sort, filter chips) uses `x-model` bindings and reactive computed properties from `Alpine.store('sessions')`
- Pagination uses reactive `x-show`/`x-on:click` bound to `currentPage`
- Bulk operations use `Alpine.store('sessions').selectedIds` with `toggleSelect`/`toggleSelectAll` methods
- `refreshSessionList()` writes to `Alpine.store('sessions').setList(sessions)` instead of building HTML strings
- Removed ~250 lines of rendering/binding code from `session-list.js`
- Polling interval preserved (3s), but no more `innerHTML` rebuilds — Alpine patches only changed rows

### Phase 3: Remaining Views (Completed 2026-06-29)

- SSE-to-Alpine bridge: `processEvent()` in `session-detail.js` writes state updates to `Alpine.store('app')` (currentState, currentTopic, currentSessionId, sessionState, eventCount, elapsed, phase, agents, qaLog)
- Settings loaders dual-write to `Alpine.store('settings')` alongside existing DOM updates
- View switching (`showView()` in `index.js`) updates `Alpine.store('app').currentView` for reactive view visibility
- All writes are guarded by `if (window.Alpine)` for graceful degradation

### Phase 4: Cleanup (Completed 2026-06-29)

- Replaced `onclick="window.*"` in `dashboard.html` with `@click="$store.app.*"` where applicable
- Removed `.hidden` class toggling in `index.js` — view visibility now controlled by `x-show` bound to `Alpine.store('app').currentView`
- Window globals reduced from ~15 to ~0 (all cross-module communication through Alpine stores)
- Removed unused DOM helper utilities replaced by Alpine directives

## Documentation

- **URL:** https://alpinejs.dev/
Expand Down Expand Up @@ -250,5 +287,6 @@ Each phase is independently rollbackable, with caveats:

| Date | Version | Changes |
|------|---------|---------|
| 2026-06-29 | 1.2 | Implementation complete (Phases 1-4). Status → Accepted. |
| 2026-06-24 | 1.1 | Addressed review: added Documentation section, Ladder compliance, htmx comparison, SSE-Alpine bridge, split Phase 2, pinned version, fixed rollback strategy, grounded code claims |
| 2026-06-24 | 1.0 | Initial version |
7 changes: 4 additions & 3 deletions docs/adr/ADR-0020-remove-llmfit-adopt-llama-server-hf.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

## Status

**Proposed**
**Accepted**

**Version:** 1.1
**Last Updated:** 2026-06-25
**Version:** 1.2
**Last Updated:** 2026-06-27

## Context

Expand Down Expand Up @@ -419,4 +419,5 @@ graph LR
| Date | Version | Changes |
|------|---------|---------|
| 2026-06-25 | 1.0 | Initial version — proposed |
| 2026-06-27 | 1.2 | Promoted from Proposed to Accepted after Phase 1 + Phase 2 implementation and review. Status changed to Accepted. |
| 2026-06-25 | 1.1 | Review fixes: corrected `POST /models`/`GET /models` capabilities (M1), added HF cache structure notes (M2), made `--fit` primary HW check (M3), added Documentation section (S1), ADR-0019 reference (S2), fixed "Fase" spelling (S3) — approved by Reviewers |
2 changes: 1 addition & 1 deletion docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@
| [ADR-0017](ADR-0017-enhanced-tool-calling-and-multi-provider-search.md) | Enhanced Tool Calling and Multi-Provider Search | Proposed | 2026-06-20 |
| [ADR-0018](ADR-0018-native-llamacpp-backend-integration.md) | Native llama.cpp Backend — Binary Lifecycle, GGUF Serving, and LiteLLM Integration | Proposed | 2026-06-20 |
| [ADR-0019](ADR-0019-frontend-reactivity-strategy.md) | Frontend Reactivity Strategy | Proposed | 2026-06-25 |
| [ADR-0020](ADR-0020-remove-llmfit-adopt-llama-server-hf.md) | Remove llmfit Dependency — Adopt llama-server `-hf` Flag and Python Hardware Detection | Proposed | 2026-06-25 |
| [ADR-0020](ADR-0020-remove-llmfit-adopt-llama-server-hf.md) | Remove llmfit Dependency — Adopt llama-server `-hf` Flag and Python Hardware Detection | Accepted | 2026-06-27 |
Loading
Loading