Integrated Browser Surface: native in-app browser with annotation/capture modes and agent handoff

[Daniel-tabz]

Summary

Add a first-class, native-feeling browser surface inside Helmor's workspace pane — a peer to the Monaco editor. Users can open multiple web tabs (localhost or any URL), navigate smoothly, and use a set of annotation & capture modes that turn anything on a page into rich, structured context. That context can be handed off to the active agent, a new agent, or a terminal agent (Claude Code / Codex), and shows up as an attachment in the chatbox. It must feel incredible to use and never degrade the performance of the rest of the app.

This is a large, ambitious feature ("kitchen sink" v1). It is organized into phases below, but all listed capabilities are in scope.

---

Goals

• A browser surface that expands to fill the workspace pane, just like opening a file in the editor does today.
• Smooth, native navigation: URL bar, back/forward/reload, history, multiple tabs, keyboard nav. Localhost or any external URL.
• Annotation/capture modes to comment on, draw on, inspect, and screenshot live pages.
• Seamless agent handoff: send a capture to the active agent, a new agent, or a terminal agent. The result is attached in the chatbox with a preview, and the whole flow feels effortless.
• Zero perf regression for chat/editor — the user keeps total control of the browser.

---

Architecture

Rendering — embedded native child webview (not an iframe)

• The browser chrome (tab strip, URL bar, mode toolbar) is React in the main webview. The page content renders in a Tauri child webview positioned over the pane's content region.
• Rationale: iframes can't load most real sites (X-Frame-Options / CSP) and give weak control. A native webview navigates anything and feels native.
• Annotation lives inside the content webview. Because a native webview paints above the DOM, we can't reliably overlay React chrome on top of page content. Hover-highlight, comment pins, and the drawing canvas are injected into the page via Tauri initialization/eval scripts — a "Helmor inspector bridge." The bridge ↔ React chrome communicate over a typed event channel.
• CSP fallback: sites that block injected scripts → freeze the page to a screenshot and annotate the image in a React canvas overlay instead. Same UX, graceful degradation. The mode toolbar auto-detects and switches.

Backend

• New src-tauri/src/browser/ module manages content-webview lifecycle (create / position / resize / navigate / destroy), bridge-script injection, and screenshot capture.
• New Tauri commands in src-tauri/src/commands/browser.rs, wired through src/lib/api.ts.
• All backend → frontend notifications go through a new UiMutationEvent variant (src-tauri/src/ui_sync/events.rs), mirrored in src/lib/api.ts, handled in src/shell/hooks/use-ui-sync-bridge.ts. No ad-hoc app.emit channels.

Surface & state integration (follows the existing editor pattern)

• Add "browser" to ShellViewMode (src/shell/controllers/use-selection-controller.ts).
• New render branch in src/shell/components/workspace-pane-surface.tsx → <WorkspaceBrowserSurface>.
• New BrowserSessionController mirroring EditorSessionController: enterBrowserMode() / exitBrowserMode(), owns the web-tab list with its own tab strip (separate from editor file-tabs).
• Per-workspace persistence: open tabs, last URL, cookies/localStorage survive reloads and app restarts (so localhost logins stick).
• Entry points: a Browser button in the workspace toolbar + an "open in browser" affordance on detected dev-server URLs (read from helmor.json run scripts).

---

Interaction modes (v1)

A single mode toolbar in the browser chrome; one active mode at a time. Esc returns to Navigate. Bridge scripts are passive until a mode is activated (zero overhead in Navigate).

• Navigate (default) — URL bar, back/forward/reload, multi-tab, history, keyboard nav.
• Hover-to-comment — hover highlights the element (outline + dimensions tag); click pins a numbered comment anchored by CSS selector; captures outerHTML, computed styles, bounding box, and a cropped screenshot. Comments re-anchor across reloads when possible.
• Element picker / inspect — like hover-comment but no note: grab selector + computed styles + HTML as structured context (devtools-lite).
• Draw + screenshot — annotation toolbar: freehand, arrow, box, text label, and blur/redact (mask secrets before sending). Drawing renders on an injected canvas; capture bakes the drawing into the page screenshot.
• Region & full-page capture — drag-select a region or stitch the full scrollable page into one image.
• Console / network capture — collect console errors + failed/slow network requests via the bridge to hand the agent runtime context.

---

Coding-specific extras (v1)

These are what make Helmor's browser more than "embedded Chrome":

• Element → source jump — click an element, jump to its source file/line in Monaco (best-effort via source maps / framework dev data-attrs like data-source / React fiber debug source). Degrades to "selector only" when unavailable.
• Inline pick + prompt — select an element, type a prompt inline ("make this button blue"), send straight to the agent with the element context attached.
• Before/after visual diff — snapshot a baseline, let the agent edit, re-snapshot, attach a side-by-side + pixel-diff image. Answers "did the agent actually fix it?"
• Live reload awareness (localhost) — detect dev-server reload (or HMR signal) and auto-refresh so the agent's edits appear live.
• Responsive / device preview — mobile / tablet / desktop / custom viewport toggles; capture at any size.
• Flow recording — record clicks / inputs / navigation as numbered, reproducible repro-steps to hand the agent.

---

Capture payload & agent handoff

Payload = screenshot + structured context. One capture produces a BrowserCapture:

type BrowserCapture = {
  url: string;
  title: string;
  viewport: { w: number; h: number; devicePreset?: string };
  images: string[];            // absolute file paths (annotated PNGs) → terminal-agent safe
  comments: Array<{ id: string; text: string; selector: string; outerHTML: string; computedStyles?: Record<string, string>; rectCropPath: string }>;
  picks:    Array<{ selector: string; outerHTML: string; computedStyles: Record<string, string>; sourceRef?: string }>;
  drawings: Array<{ kind: string; screenshotPath: string }>;
  console?: Array<unknown>;
  network?: Array<unknown>;
  flow?:    Array<unknown>;     // recorded steps
  diff?:    { beforePath: string; afterPath: string; diffPath: string };
};

• Attached in the chatbox as a new browser-capture badge — a Lexical decorator node mirroring ImageBadgeNode (src/features/composer/editor/image-badge-node.tsx) — with a hover preview that expands to show comments/picks.
• Images flow through the existing AgentSendRequest.images (absolute paths); structured text is serialized into the prompt context. This guarantees terminal agents (Claude Code / Codex) work too — file paths + text, never UI-only data.
• Handoff targets:
  1. Attach to active agent — drops into the current session's composer, ready to send with a prompt.
  2. Spawn new agent — creates a fresh session pre-loaded with the capture attached.
  3. Terminal-agent-safe — same format works for CLI agents.

---

Performance (explicit acceptance criteria)

The user must keep total control and the rest of the app must stay perfectly smooth.

• Browser surface is lazy-mounted — content webview created only on first use.
• Background / inactive web tabs are suspended or throttled; only the visible tab is live.
• Screenshot / diff / full-page-stitch work runs off the React main thread (backend or worker); no jank in chat.
• Bridge scripts are passive until a mode is activated (zero overhead in Navigate).
• Tearing down the surface fully releases the webview + memory.
• No dropped frames in chat/editor when opening/closing the browser or switching tabs — validated with the perf HUD (VITE_HELMOR_PERF_HUD=1).

---

Backend / data changes

• src-tauri/src/browser/ module + src-tauri/src/commands/browser.rs (create / navigate / capture / inject / destroy).
• Bridge script bundle (TS → injected) for inspect / draw / console.
• Per-workspace browser state (tabs / URLs / cookies) persisted in SQLite via src-tauri/src/models/; schema.rs migration.
• BrowserCapture attachment type added to the composer-insert union (src/lib/composer-insert.ts) + a new badge node.
• 🚨 If any capture data touches the session_messages storage shape, add snapshot test coverage in src-tauri/tests/ (per repo rules for pipeline/ / persistence / schema.rs changes).
• New UiMutationEvent variant for browser state sync.

---

Risks / open questions

• Native webview overlay/positioning fidelity across macOS window resize, overlay traffic-light title bar, and fullscreen.
• CSP-blocked sites → coverage of the screenshot-fallback path.
• Element → source jump reliability is framework-dependent; ship as best-effort with graceful fallback.
• Full-page stitching for lazy-loaded / virtualized pages.
• Cross-origin limits on console / network capture.

---

Suggested build order (phased — all in v1 scope)

1. Surface + chrome — native webview navigation, tabs, per-workspace persistence.
2. Capture pipeline — screenshot + structured context, attachment badge, handoff (active / new / terminal).
3. Inspector bridge — hover-comment, element picker, console / network.
4. Drawing / annotation — toolbar + redact + region / full-page capture.
5. Coding extras — element→source, inline pick+prompt, before/after diff, live reload, responsive, flow recording.
6. Perf hardening — perf-HUD validation pass against all acceptance criteria.

---

Acceptance criteria (high level)

• Browser opens as a full workspace surface with its own multi-tab strip; navigates localhost and external URLs smoothly.
• All six interaction modes + all six coding extras function as described (with documented graceful fallbacks).
• A capture produces a BrowserCapture, renders as a chatbox badge with preview, and can be sent to active / new / terminal agents.
• All performance acceptance criteria pass under the perf HUD.
• Backend changes covered by snapshot tests where they touch persistence / pipeline.

---

Filed from a planning session in Helmor. Architecture grounded in the existing editor-surface, composer-attachment, and UiMutationEvent patterns.