diff --git a/docs/architecture.md b/docs/architecture.md index 81f7e201..5840bd8a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -92,6 +92,53 @@ provider conditionals. A new provider difference is a new or extended `CuaRuntimeSpec`/`CuaPayloadContext` field plus provider code in `@onkernel/cua-ai` — never a branch in `@onkernel/cua-agent`. +## Action planes and modes + +The canonical action vocabulary is split into two planes, delineated in code +under `packages/ai/src/actions/`: + +- **Computer plane** (`actions/computer.ts`) — real OS-level input against the browser + VM: mouse, keyboard, display capture, executed through Kernel's + `browsers.computer` REST API. Coordinates are pixels in the OS screenshot + frame. +- **Browser plane** (`actions/browser.ts`, ids prefixed `browser_`) — CDP-driven page + tools: accessibility snapshots with element refs, element-targeted + interaction, navigation, tabs, viewport screenshots. Executed by + `packages/agent/src/translator/browser.ts` (`BrowserExecutor`) over a raw CDP websocket + (`translator/cdp.ts`) to the browser's `cdp_ws_url` — no Playwright. + Coordinates, where used, are viewport pixels. + +A `CuaMode` selects which plane(s) the model sees: + +| mode | tools | coordinate frame | +| --- | --- | --- | +| `computer` (default) | computer actions under their canonical ids (`click`, `screenshot`, …) | OS screenshot pixels | +| `browser` | browser actions with the `browser_` prefix stripped (`snapshot`, `click`, …) plus `wait` | none for refs; viewport pixels where coordinates are allowed | +| `hybrid` | both planes, one tool per capability: computer actions as `computer_*`, browser reads/element-writes as `browser_*` (ref-only) | OS screenshot pixels — the single live frame | + +Hybrid deduplicates capabilities: navigation and tabs live on the browser plane, +pointer/keyboard input and the (only) screenshot live on the computer plane, and +hybrid browser tools take element refs only so exactly one coordinate frame exists. +Element refs are snapshot-scoped (`e12`); a stale ref resolves to an error +string that tells the model to re-snapshot. + +The mode is set at construction (`mode` on `CuaAgent`/`CuaAgentHarness`, +`--mode` in the CLI) and can be switched at runtime with `setMode()` (the +TUI's `/mode` command), which refreshes CUA-owned tools and the default +system prompt. + +**Native tools.** `resolveCuaRuntimeSpec(model, { nativeTool })` drives an +Anthropic model through its provider-defined tool schema instead of the +canonical function tools: `computer_20260701` pairs with `computer` mode and +`browser_20260701` with `browser` mode (mismatches throw, mirroring the API's +own rejection of mixed frames). The spec routes the model to a CUA-owned api +id; the registered `anthropic` provider dispatches it to pi's builtin +`anthropic-messages` transport with the tool's `anthropic-beta` header, an +`onPayload` hook swaps the placeholder tool for the native declaration, and +`providers/anthropic/native.ts` maps incoming `tool_use` inputs onto the same +canonical actions the mode uses — so canonical vs native is purely a wire +format difference over one execution path. + ## Layers `cua` is a thin TypeScript monorepo on top of the diff --git a/package-lock.json b/package-lock.json index 75af476a..ef407244 100644 --- a/package-lock.json +++ b/package-lock.json @@ -6111,12 +6111,12 @@ }, "packages/agent": { "name": "@onkernel/cua-agent", - "version": "0.4.0", + "version": "0.5.0", "license": "MIT", "dependencies": { "@earendil-works/pi-agent-core": "0.80.3", "@earendil-works/pi-ai": "0.80.3", - "@onkernel/cua-ai": "0.4.0", + "@onkernel/cua-ai": "0.5.0", "@onkernel/sdk": "0.49.0", "sharp": "^0.34.5" }, @@ -6127,7 +6127,7 @@ }, "packages/ai": { "name": "@onkernel/cua-ai", - "version": "0.4.0", + "version": "0.5.0", "license": "MIT", "dependencies": { "@earendil-works/pi-ai": "0.80.3", @@ -6141,13 +6141,13 @@ }, "packages/cli": { "name": "@onkernel/cua-cli", - "version": "0.2.0", + "version": "0.3.0", "license": "MIT", "dependencies": { "@earendil-works/pi-coding-agent": "0.80.3", "@earendil-works/pi-tui": "0.80.3", - "@onkernel/cua-agent": "0.4.0", - "@onkernel/cua-ai": "0.4.0", + "@onkernel/cua-agent": "0.5.0", + "@onkernel/cua-ai": "0.5.0", "@onkernel/sdk": "0.49.0" }, "bin": { diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index e8c88f6a..e094246d 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,35 @@ # Changelog +## 0.5.0 - 2026-07-09 + +Adds the browser action plane and runtime mode switching. Breaking: the +`computerUseExtra` option is removed — the `computer_use_extra` navigation +helper is always registered. + +- New `BrowserExecutor`: drives the browser plane over CDP. Accessibility + snapshots with element refs (`[e12]`), node states + (checked/expanded/disabled/value/…), and cursor:pointer clickable hints + for elements with no interactive ARIA role; iframe and OOPIF stitching + with per-frame session-aware refs; StaticText dedupe and wrapper + collapsing; an unchanged-snapshot short-circuit; lexical `find`, `fill`, + CDP navigation and tab management; and a JavaScript dialog guard. Refs invalidate on real + navigations (`Page.frameNavigated`), self-heal via (role, name, nth) when + the page changes but the element is still unambiguous, and the ref table is + bounded (per-target cap, generation sweeps). `exportRefState()` / + `importRefState()` persist refs across processes against the same browser. +- `CuaAgent` and `CuaAgentHarness` accept `mode` (`"computer"` | `"browser"` + | `"hybrid"`) and `nativeTool`, and support runtime plane switching via + `setMode()` / `getMode()`. Mode switches preserve the requested activation + state of surviving tools and keep the translator — CDP connection, tabs, + and element refs — alive; the translator is only rebuilt when a model + switch changes the provider's coordinate system or screenshot transform. + Both switches roll back cleanly on failure. +- Post-action grounding captures and the navigation helper are mode-aware: + browser mode grounds on the viewport and routes navigation through CDP + (browser and hybrid modes both route `computer_use_extra` navigation over + the browser plane so refs invalidate correctly). +- Updated `@onkernel/cua-ai` to 0.5.0. + ## 0.4.0 - 2026-07-07 Breaking: follows pi-agent-core 0.80's `Models`-based harness. diff --git a/packages/agent/README.md b/packages/agent/README.md index cb60d4f3..5371d013 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -97,7 +97,6 @@ Both classes mirror pi constructor shapes and behavior, with minimal additions: - `client` (Kernel SDK client) - CUA model refs (`"provider:model"`) accepted where pi expects a concrete model - `extraTools` to add your own pi tools alongside the built-in browser tools -- `computerUseExtra: true` to let the model use a small navigation helper - `playwright: true` to let the model run Playwright/TypeScript against the live browser session @@ -122,7 +121,7 @@ or handing off to another service while it also controls the browser. Not every provider's native computer-use vocabulary includes browser navigation — some models can click and type but have no direct way to open a -URL or go back. `computerUseExtra: true` adds `computer_use_extra`, a +URL or go back. The classes therefore always add `computer_use_extra`, a provider-neutral escape hatch exposing `goto`, `back`, `forward`, and `url` so navigation works uniformly regardless of which model is driving. diff --git a/packages/agent/examples/anthropic-native-smoke.ts b/packages/agent/examples/anthropic-native-smoke.ts new file mode 100644 index 00000000..3061d665 --- /dev/null +++ b/packages/agent/examples/anthropic-native-smoke.ts @@ -0,0 +1,61 @@ +// Smoke-test the mode/native-tool matrix against a live Kernel browser: +// +// MODEL_REF=anthropic:claude-opus-4-8 CONFIG=native-browser tsx examples/anthropic-native-smoke.ts +// +// CONFIG selects the runtime shape: +// computer (default) canonical computer-plane (OS input) tools +// browser canonical browser-plane (CDP page) tools +// hybrid both planes, deduplicated +// native-computer Anthropic computer_20260701 (requires the computer-use beta) +// native-browser Anthropic browser_20260701 (requires the browser-use beta) +import Kernel from "@onkernel/sdk"; +import { requireCuaEnvApiKeyForModel, type CuaModelRef, type CuaMode, type CuaNativeToolSpec } from "@onkernel/cua-ai"; +import { CuaAgent } from "../src/index"; +import { logAgentEvent, logAssistant } from "./shared/logging"; + +const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "anthropic:claude-opus-4-8"; +const config = process.env.CONFIG ?? "computer"; + +const CONFIGS: Record = { + computer: { mode: "computer" }, + browser: { mode: "browser" }, + hybrid: { mode: "hybrid" }, + "native-computer": { nativeTool: { type: "computer_20260701", enable_zoom: true } }, + "native-browser": { nativeTool: { type: "browser_20260701" } }, +}; + +const PROMPT = [ + "Navigate to https://example.com, read the page, and tell me:", + "1) the main heading text", + "2) the text of the link on the page", + "Then follow that link and tell me the title of the page you land on.", +].join("\n"); + +async function main(): Promise { + const runtime = CONFIGS[config]; + if (!runtime) throw new Error(`unknown CONFIG "${config}" (expected: ${Object.keys(CONFIGS).join(" | ")})`); + const kernelApiKey = process.env.KERNEL_API_KEY; + if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); + requireCuaEnvApiKeyForModel(modelRef); + const client = new Kernel({ apiKey: kernelApiKey }); + const browser = await client.browsers.create({ stealth: true }); + + try { + const agent = new CuaAgent({ + browser, + client, + ...runtime, + initialState: { model: modelRef }, + }); + agent.subscribe(logAgentEvent); + + console.log(`running config=${config} model=${modelRef} live_view=${browser.browser_live_view_url}`); + await agent.prompt(PROMPT); + const assistant = [...agent.state.messages].reverse().find((message) => message.role === "assistant"); + logAssistant(assistant?.role === "assistant" ? assistant : undefined); + } finally { + await client.browsers.deleteByID(browser.session_id); + } +} + +void main(); diff --git a/packages/agent/package.json b/packages/agent/package.json index 8c0b0b78..94a5b25d 100644 --- a/packages/agent/package.json +++ b/packages/agent/package.json @@ -1,6 +1,6 @@ { "name": "@onkernel/cua-agent", - "version": "0.4.0", + "version": "0.5.0", "description": "Kernel browser computer-use Agent and AgentHarness classes built on pi-agent-core", "license": "MIT", "type": "module", @@ -42,7 +42,7 @@ "dependencies": { "@earendil-works/pi-agent-core": "0.80.3", "@earendil-works/pi-ai": "0.80.3", - "@onkernel/cua-ai": "0.4.0", + "@onkernel/cua-ai": "0.5.0", "@onkernel/sdk": "0.49.0", "sharp": "^0.34.5" }, diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index 1c289c56..38ca37b9 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -14,7 +14,9 @@ import { CUA_NAVIGATION_TOOL_NAME, CUA_PLAYWRIGHT_TOOL_NAME, cuaModels, + type CuaMode, type CuaModelRef, + type CuaNativeToolSpec, type CuaRuntimeSpec, type CuaSimpleStreamOptions, getCuaEnvApiKey, @@ -70,8 +72,10 @@ export type CuaAgentOptions = Omit & { initialState: CuaAgentInitialState; /** Add your own pi tools alongside the built-in browser tools. */ extraTools?: AgentTool[]; - /** Expose a helper for browser navigation and URL reads. */ - computerUseExtra?: boolean; + /** Which canonical action plane(s) to expose: "computer" (default), "browser", or "hybrid". */ + mode?: CuaMode; + /** Drive the model through a provider-native tool declaration (validated against `mode`). */ + nativeTool?: CuaNativeToolSpec; /** Expose a tool that runs Playwright code against the browser session. */ playwright?: boolean; }; @@ -104,8 +108,10 @@ export type CuaAgentHarnessOptions< models?: Models; /** Add your own pi tools alongside the built-in browser tools. */ extraTools?: AgentTool[]; - /** Expose a helper for browser navigation and URL reads. */ - computerUseExtra?: boolean; + /** Which canonical action plane(s) to expose: "computer" (default), "browser", or "hybrid". */ + mode?: CuaMode; + /** Drive the model through a provider-native tool declaration (validated against `mode`). */ + nativeTool?: CuaNativeToolSpec; /** Expose a tool that runs Playwright code against the browser session. */ playwright?: boolean; /** Optional payload hook composed after the provider-specific CUA payload hook. */ @@ -121,6 +127,7 @@ export type CuaAgentHarnessOptions< class CuaRuntimeController { private runtimeSpec: CuaRuntimeSpec; private translator: InternalComputerTranslator; + private currentMode?: CuaMode; constructor( private readonly options: { @@ -128,34 +135,99 @@ class CuaRuntimeController { client: Kernel; model: CuaRuntimeInput; extraTools?: AgentTool[]; - computerUseExtra?: boolean; + mode?: CuaMode; + nativeTool?: CuaNativeToolSpec; playwright?: boolean; onPayload?: SimpleStreamOptions["onPayload"]; }, ) { - this.runtimeSpec = resolveCuaRuntimeSpec(options.model); + this.currentMode = options.mode; + this.runtimeSpec = this.resolveSpec(options.model); this.translator = this.createTranslator(); } + private resolveSpec(model: CuaRuntimeInput, mode: CuaMode | undefined = this.currentMode): CuaRuntimeSpec { + return resolveCuaRuntimeSpec(model, { + mode, + nativeTool: this.options.nativeTool, + }); + } + get model(): Model { return this.runtimeSpec.model; } + get mode(): CuaMode { + return this.runtimeSpec.mode; + } + + setMode(mode: CuaMode): void { + if (mode === this.runtimeSpec.mode) return; + this.beginSwitch(this.resolveSpec(this.runtimeSpec.model, mode)); + this.currentMode = mode; + } + get systemPrompt(): string { return this.runtimeSpec.defaultSystemPrompt; } setModel(model: CuaRuntimeInput): void { - this.runtimeSpec = resolveCuaRuntimeSpec(model); + this.beginSwitch(this.resolveSpec(model)); + } + + // A mode/model switch is two-phase: when the new spec needs a different + // translator configuration, the outgoing translator must stay alive until + // the new toolset is actually installed, because on failure the + // still-exposed pre-switch tools keep executing against it. + private previousRuntime?: { spec: CuaRuntimeSpec; translator?: InternalComputerTranslator; mode?: CuaMode }; + + private beginSwitch(spec: CuaRuntimeSpec): void { + // The translator only cares about the provider's coordinate system and + // screenshot transform. Keep it — and its CDP connection, tabs, and + // refs — whenever those are unchanged (always true for mode switches). + const replaceTranslator = + JSON.stringify([spec.coordinateSystem, spec.screenshot]) !== + JSON.stringify([this.runtimeSpec.coordinateSystem, this.runtimeSpec.screenshot]); + if (!this.previousRuntime) { + this.previousRuntime = { spec: this.runtimeSpec, mode: this.currentMode }; + } + this.runtimeSpec = spec; + if (!replaceTranslator) return; + if (this.previousRuntime.translator) { + // An earlier pending switch already replaced the translator; its + // replacement was never installed into the exposed tools, so + // dispose it rather than orphaning it. + this.translator.dispose(); + } else { + this.previousRuntime.translator = this.translator; + } this.translator = this.createTranslator(); } + /** Dispose the pre-switch translator (when one was replaced) once the new toolset is installed. */ + commitSwitch(): void { + this.previousRuntime?.translator?.dispose(); + this.previousRuntime = undefined; + } + + /** Restore the pre-switch runtime; the translator the exposed tools wrap stays live. */ + rollbackSwitch(): void { + if (!this.previousRuntime) return; + if (this.previousRuntime.translator) { + this.translator.dispose(); + this.translator = this.previousRuntime.translator; + } + this.runtimeSpec = this.previousRuntime.spec; + this.currentMode = this.previousRuntime.mode; + this.previousRuntime = undefined; + } + tools(): AgentTool[] { return [ ...buildCuaComputerTools( { toolExecutors: this.runtimeSpec.toolExecutors, - computerUseExtra: this.options.computerUseExtra, + mode: this.runtimeSpec.mode, playwright: this.options.playwright, }, this.translator, @@ -179,7 +251,7 @@ class CuaRuntimeController { keepToolNames(): string[] { return [ ...(this.options.extraTools ?? []).map((tool) => tool.name), - ...(this.options.computerUseExtra ? [CUA_NAVIGATION_TOOL_NAME] : []), + CUA_NAVIGATION_TOOL_NAME, ...(this.options.playwright ? [CUA_PLAYWRIGHT_TOOL_NAME] : []), ]; } @@ -221,7 +293,8 @@ export class CuaAgent extends Agent { streamFn, prepareNextTurn, extraTools, - computerUseExtra, + mode, + nativeTool, playwright, ...agentOptions } = options; @@ -230,7 +303,8 @@ export class CuaAgent extends Agent { client, model: initialState.model, extraTools, - computerUseExtra, + mode, + nativeTool, playwright, onPayload, }); @@ -313,6 +387,24 @@ export class CuaAgent extends Agent { return this.stateProxy; } + /** Switch the action plane(s) exposed to the model; takes effect next turn. */ + setMode(mode: CuaMode): void { + if (mode === this.runtime.mode) return; + this.runtime.setMode(mode); + this.runtimeDirty = true; + const state = super.state; + state.tools = this.runtime.tools(); + if (this.ownsSystemPrompt) { + state.systemPrompt = this.runtime.systemPrompt; + } + this.runtime.commitSwitch(); + } + + /** The action plane(s) currently exposed to the model. */ + getMode(): CuaMode { + return this.runtime.mode; + } + private applyRuntime(model: CuaRuntimeInput): void { this.runtime.setModel(model); this.runtimeDirty = true; @@ -322,6 +414,7 @@ export class CuaAgent extends Agent { if (this.ownsSystemPrompt) { state.systemPrompt = this.runtime.systemPrompt; } + this.runtime.commitSwitch(); } } @@ -347,7 +440,8 @@ export class CuaAgentHarness< model, models, extraTools, - computerUseExtra, + mode, + nativeTool, playwright, systemPrompt, onPayload, @@ -359,7 +453,8 @@ export class CuaAgentHarness< client, model, extraTools, - computerUseExtra, + mode, + nativeTool, playwright, onPayload, }); @@ -393,7 +488,15 @@ export class CuaAgentHarness< override async setModel(model: CuaRuntimeInput): Promise { this.runtime.setModel(model); const tools = this.runtime.tools(); - await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name)); + try { + await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name)); + } catch (err) { + // The pre-switch tools stay exposed, so restore the runtime they are + // bound to — including its still-live translator. + this.runtime.rollbackSwitch(); + throw err; + } + this.runtime.commitSwitch(); await super.setModel(this.runtime.model); } @@ -401,6 +504,41 @@ export class CuaAgentHarness< await super.setActiveTools(toolNames); this.requestedActiveToolNames = [...toolNames]; } + + /** + * Switch the action plane(s) exposed to the model and refresh CUA-owned + * tools. Throws when the harness was configured with a `nativeTool` whose + * plane conflicts with the requested mode. + */ + async setMode(mode: CuaMode): Promise { + if (mode === this.runtime.mode) return; + const previousNames = new Set(this.getTools().map((tool) => tool.name)); + this.runtime.setMode(mode); + const tools = this.runtime.tools(); + // Tools that survive the mode switch (extraTools, shared names) keep + // their requested activation state; names new in this mode activate. + const requested = this.requestedActiveToolNames; + const active = requested + ? tools.map((tool) => tool.name).filter((name) => !previousNames.has(name) || requested.includes(name)) + : tools.map((tool) => tool.name); + try { + await super.setTools(tools, active); + } catch (err) { + // The pre-switch tools stay exposed, so restore the runtime they are + // bound to — including its still-live translator. + this.runtime.rollbackSwitch(); + throw err; + } + this.runtime.commitSwitch(); + // The requested subset now reflects this mode's toolset; without this a + // later setModel would restore the pre-switch names. + if (requested) this.requestedActiveToolNames = active; + } + + /** The action plane(s) currently exposed to the model. */ + getMode(): CuaMode { + return this.runtime.mode; + } } function composeOnPayload(first: AgentOptions["onPayload"], second: AgentOptions["onPayload"]): AgentOptions["onPayload"] { diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index 75bea96f..2f79b50f 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -2,6 +2,11 @@ export * from "@earendil-works/pi-agent-core"; export { NodeExecutionEnv } from "@earendil-works/pi-agent-core/node"; export type { KernelBrowser } from "./translator/translator"; +export { InternalComputerTranslator } from "./translator/translator"; +export { CdpConnection } from "./translator/cdp"; +export { BrowserExecutor } from "./translator/browser"; +export type { BrowserFindCandidate, BrowserRefState } from "./translator/browser"; +export type { BatchExecutionResult, BatchReadResult } from "./translator/types"; export { createCuaComputerTools } from "./tools"; export type { BatchDetails, diff --git a/packages/agent/src/tools.ts b/packages/agent/src/tools.ts index a326116a..9a227434 100644 --- a/packages/agent/src/tools.ts +++ b/packages/agent/src/tools.ts @@ -7,6 +7,7 @@ import { createCuaPlaywrightToolDefinition, type ComputerToolCoordinateSystem, type CuaBatchInput, + type CuaMode, type CuaNavigationInput, type CuaPlaywrightInput, type CuaScreenshotSpec, @@ -22,7 +23,8 @@ export interface ComputerToolOptions { toolExecutors: CuaToolExecutorSpec[]; coordinateSystem?: ComputerToolCoordinateSystem; screenshot?: CuaScreenshotSpec; - computerUseExtra?: boolean; + /** Action plane(s) in play; controls whether the post-action fallback capture is the OS display or the viewport. Default "computer". */ + mode?: CuaMode; playwright?: boolean; } @@ -30,7 +32,12 @@ type ToolContent = Array; export interface BatchDetails { statusText: string; - readResults: Array<{ type: "url"; url: string } | { type: "screenshot"; bytes: number } | { type: "cursor_position"; x: number; y: number }>; + readResults: Array< + | { type: "url"; url: string } + | { type: "screenshot"; bytes: number } + | { type: "cursor_position"; x: number; y: number } + | { type: "browser_text"; label: string; bytes: number } + >; } export interface NavigationDetails { @@ -78,16 +85,16 @@ export function createCuaComputerTools(args: ComputerToolOptions): CuaExecutorTo /** Build executor tools against an existing translator (internal; not part of the package surface). */ export function buildCuaComputerTools( - args: Pick, + args: Pick, translator: InternalComputerTranslator, ): CuaExecutorTool[] { - return withExtraTools(args).map((executor) => createExecutorTool(executor, translator)); + return withExtraTools(args).map((executor) => createExecutorTool(executor, translator, args.mode ?? "computer")); } -function withExtraTools(args: Pick): ComputerExecutorSpec[] { +function withExtraTools(args: Pick): ComputerExecutorSpec[] { const executors: ComputerExecutorSpec[] = [...args.toolExecutors]; const existing = new Set(executors.map((executor) => executor.definition.name)); - if (args.computerUseExtra && !existing.has(CUA_NAVIGATION_TOOL_NAME)) { + if (!existing.has(CUA_NAVIGATION_TOOL_NAME)) { executors.push({ kind: "navigation", definition: createCuaNavigationToolDefinition() }); } if (args.playwright && !existing.has(CUA_PLAYWRIGHT_TOOL_NAME)) { @@ -96,7 +103,7 @@ function withExtraTools(args: Pick> { - return executeNavigationTool(translator, asNavigationInput(params)); + return executeNavigationTool(translator, asNavigationInput(params), mode); }, }; return tool; @@ -130,7 +137,7 @@ function createExecutorTool(executor: ComputerExecutorSpec, translator: Internal parameters: definition.parameters, executionMode: "sequential", async execute(_toolCallId: string, params: unknown): Promise> { - return executeBatchTool(translator, { actions: executor.toActions(params) }); + return executeBatchTool(translator, { actions: executor.toActions(params) }, mode); }, }; return tool; @@ -144,7 +151,11 @@ function isPlaywrightExecutor(executor: ComputerExecutorSpec): executor is Playw return "kind" in executor && executor.kind === "playwright"; } -async function executeBatchTool(translator: InternalComputerTranslator, params: CuaBatchInput): Promise> { +async function executeBatchTool( + translator: InternalComputerTranslator, + params: CuaBatchInput, + mode: CuaMode = "computer", +): Promise> { const content: ToolContent = []; const readResults: BatchDetails["readResults"] = []; try { @@ -156,13 +167,18 @@ async function executeBatchTool(translator: InternalComputerTranslator, params: } else if (read.type === "cursor_position") { readResults.push({ type: "cursor_position", x: read.x, y: read.y }); content.push({ type: "text", text: `cursor_position(): ${read.x},${read.y}` }); + } else if (read.type === "browser_text") { + readResults.push({ type: "browser_text", label: read.label, bytes: read.text.length }); + content.push({ type: "text", text: read.text }); } else { readResults.push({ type: "screenshot", bytes: read.data.length }); content.push({ type: "image", data: read.data.toString("base64"), mimeType: read.mimeType }); } } if (content.length === 0) { - const screenshot = await translator.screenshot(); + // Post-action grounding capture: the OS display in computer/hybrid mode, + // the browser viewport in browser mode (the only frame the model sees). + const screenshot = mode === "browser" ? await translator.browser().screenshot() : await translator.screenshot(); readResults.push({ type: "screenshot", bytes: screenshot.data.length }); content.push({ type: "image", data: screenshot.data.toString("base64"), mimeType: screenshot.mimeType }); } @@ -172,20 +188,31 @@ async function executeBatchTool(translator: InternalComputerTranslator, params: return { content, details: { statusText: "Actions executed successfully.", readResults } }; } -async function executeNavigationTool(translator: InternalComputerTranslator, params: CuaNavigationInput): Promise> { +async function executeNavigationTool( + translator: InternalComputerTranslator, + params: CuaNavigationInput, + mode: CuaMode, +): Promise> { const action = params.action; try { let statusText = `${action} executed successfully.`; let url: string | undefined; + // When the browser plane is exposed (browser/hybrid), navigation stays + // on it (CDP) so it invalidates element refs and reads the tab-aware URL; + // the OS keyboard-shortcut path would navigate outside that plane. if (action === "url") { - url = await translator.currentUrl(); + url = mode === "computer" ? await translator.currentUrl() : await translator.browser().currentUrl(); statusText = `Current URL: ${url}`; + } else if (mode !== "computer") { + await translator.executeBatch([{ type: "browser_navigate", url: action === "goto" ? (params.url ?? "") : action }]); } else if (action === "goto") { await translator.executeBatch([{ type: "goto", url: params.url ?? "" }]); } else { await translator.executeBatch([{ type: action }]); } - const screenshot = await translator.screenshot(); + // Same grounding frame as post-action captures: the browser viewport in + // browser mode, the OS display otherwise. + const screenshot = mode === "browser" ? await translator.browser().screenshot() : await translator.screenshot(); return { content: [ { type: "text", text: statusText }, diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts new file mode 100644 index 00000000..9cb21a14 --- /dev/null +++ b/packages/agent/src/translator/browser.ts @@ -0,0 +1,1179 @@ +import { + normalizeGotoUrl, + type CuaActionBrowserClick, + type CuaActionBrowserDrag, + type CuaActionBrowserFill, + type CuaActionBrowserFind, + type CuaActionBrowserHover, + type CuaActionBrowserKey, + type CuaActionBrowserNavigate, + type CuaActionBrowserScroll, + type CuaActionBrowserScrollTo, + type CuaActionBrowserSnapshot, + type CuaBrowserAction, +} from "@onkernel/cua-ai"; +import { CdpConnection, type CdpEventMessage } from "./cdp"; +import type { BatchReadResult } from "./types"; + +const SNAPSHOT_CHAR_LIMIT = 50_000; +const DEFAULT_SNAPSHOT_DEPTH = 15; +const FIND_MATCH_LIMIT = 20; +const REF_LIMIT_PER_TARGET = 1000; +const SCROLL_NOTCH_PX = 120; + +const STALE_REF_HINT = "Call snapshot (or find) to get fresh element references."; +const REF_PLACEHOLDER = "\u0000"; +const UNCHANGED_SNAPSHOT = "Page unchanged since the last snapshot; previous element refs are still valid."; + +interface AXNode { + nodeId: string; + ignored?: boolean; + role?: { value?: string }; + name?: { value?: string }; + value?: { value?: unknown }; + properties?: Array<{ name: string; value?: { value?: unknown } }>; + backendDOMNodeId?: number; + parentId?: string; + childIds?: string[]; +} + +interface RefEntry { + backendNodeId: number; + targetId: string; + /** Generation key: the owning page target id for main-frame refs, the frame id for iframe refs. */ + frameId: string; + /** Session to route DOM/Input calls through: the frame's own session for OOPIFs, the page session otherwise. */ + sessionId: string; + generation: number; + role: string; + name: string; + nth: number; + /** Size of the (role, name) cohort in the tree the ref was minted from. */ + cohort: number; +} + +interface NthIndex { + index: Map; + cohorts: Map; +} + +interface RenderContext { + targetId: string; + frameKey: string; + sessionId: string; + generation: number; + interactiveOnly: boolean; + nthIndex: NthIndex; + cursorIds?: ReadonlySet; +} + +interface RenderedLine { + text: string; + refNode?: AXNode; + ctx: RenderContext; +} + +interface FrameStitch { + byId: Map; + roots: string[]; + ctx: RenderContext; +} + +export interface BrowserFindCandidate { + ref: string; + role: string; + name: string; + score: number; +} + +/** + * Serializable ref state, so refs minted in one process (e.g. a `cua + * snapshot` invocation) can be resolved in a later one against the same + * browser. Session ids are process-local and deliberately not exported; + * imported refs rebind lazily. Backend node ids stay valid for the life of + * the document, and the usual generation/self-heal machinery covers pages + * that changed in between. + */ +export interface BrowserRefState { + refCounter: number; + activeTargetId?: string; + generations: Array<[string, number]>; + refs: Array<[string, Omit]>; +} + +/** + * Executes browser-plane canonical actions over CDP. + * + * Element refs are snapshot-scoped: each snapshot/find mints `e` ids + * mapped to CDP backend node ids for the target's current generation. Any + * main-frame navigation — our own navigate() or a page-initiated one seen + * via Page.frameNavigated — bumps the generation and prunes that target's + * refs, so refs from earlier generations resolve to a stale-ref error whose + * message tells the model how to recover. + * + * A ref whose backend node vanished without a navigation (DOM churn) is + * self-healed: the AX tree is re-fetched and the ref re-resolved by the + * (role, name, nth) triple recorded at mint time, but only when the fresh + * cohort has the same size as at mint time. + * + * Snapshots stitch iframe content under each iframe node: same-process + * frames via the page session's AX tree with a frameId, out-of-process + * frames via their auto-attached session. Refs record their frame and + * session so actions resolve through the right one, and each frame's refs + * are invalidated independently when that frame navigates. Re-snapshotting + * an unchanged page with the same params returns a short unchanged notice + * instead of the full tree. + * + * Native JavaScript dialogs are auto-handled so they never wedge the CDP + * session: alert and beforeunload dialogs are accepted (so navigation can + * proceed), confirm and prompt dialogs are dismissed. The dialog message + * is surfaced as an extra read result on the next executed action. + */ +export class BrowserExecutor { + private readonly refs = new Map(); + private readonly generations = new Map(); + private readonly targetsBySession = new Map(); + private readonly frameSessions = new Map(); + private readonly frameTargets = new Set(); + private readonly lastSnapshots = new Map(); + private readonly selfNavigations = new Set(); + private readonly dialogNotes: string[] = []; + private refCounter = 0; + private activeTargetId?: string; + private readonly cdp: CdpConnection; + + constructor(cdp: string | CdpConnection) { + this.cdp = typeof cdp === "string" ? new CdpConnection(cdp) : cdp; + this.cdp.onEvent((event) => this.handleCdpEvent(event)); + } + + private handleCdpEvent(event: CdpEventMessage): void { + switch (event.method) { + case "Page.frameNavigated": { + const frame = event.params.frame as { id?: string; parentId?: string } | undefined; + if (!event.sessionId || !frame) return; + const targetId = this.targetsBySession.get(event.sessionId); + if (!targetId) return; + if (this.frameTargets.has(targetId)) { + // Refs from a frame target's tree (its root and any same-process + // subframes inlined in it) are all minted against the target's key, + // so any navigation observed in its session stales them. + this.invalidateFrame(targetId); + return; + } + if (frame.parentId) { + if (frame.id) this.invalidateFrame(frame.id); + return; + } + if (!this.selfNavigations.delete(targetId)) this.invalidateRefs(targetId); + return; + } + case "Page.navigatedWithinDocument": { + // A navigate() that turns out same-document never fires frameNavigated; + // consume the pending flag here so it can't swallow the next real navigation. + if (!event.sessionId) return; + const targetId = this.targetsBySession.get(event.sessionId); + const { frameId } = event.params as { frameId?: string }; + if (targetId && frameId === targetId) this.selfNavigations.delete(targetId); + return; + } + case "Target.attachedToTarget": { + const { sessionId, targetInfo } = event.params as { sessionId?: string; targetInfo?: { targetId?: string; type?: string } }; + if (!sessionId || !targetInfo?.targetId || targetInfo.type !== "iframe") return; + this.frameSessions.set(targetInfo.targetId, sessionId); + this.frameTargets.add(targetInfo.targetId); + this.targetsBySession.set(sessionId, targetInfo.targetId); + void this.cdp.send("Page.enable", {}, sessionId).catch(() => {}); + return; + } + case "Page.javascriptDialogOpening": { + if (!event.sessionId) return; + const { type, message } = event.params as { type?: string; message?: string }; + const accept = type === "alert" || type === "beforeunload"; + void this.cdp.send("Page.handleJavaScriptDialog", { accept }, event.sessionId).catch(() => {}); + const summary = + type === "beforeunload" + ? "Accepted a beforeunload dialog so navigation could proceed" + : accept + ? "Acknowledged a JavaScript alert dialog" + : `Dismissed a JavaScript ${type ?? "dialog"} dialog (answered No/cancel)`; + this.dialogNotes.push(`${summary}: ${JSON.stringify(message ?? "")}`); + return; + } + case "Target.detachedFromTarget": { + const sessionId = event.params.sessionId; + if (typeof sessionId !== "string") return; + const targetId = this.targetsBySession.get(sessionId); + this.targetsBySession.delete(sessionId); + if (targetId) this.dropTarget(targetId); + return; + } + } + } + + /** Close the CDP connection. Safe to call when never connected. */ + close(): void { + this.cdp.close(); + } + + /** Snapshot the ref table for persistence across invocations; see {@link BrowserRefState}. */ + exportRefState(): BrowserRefState { + return { + refCounter: this.refCounter, + ...(this.activeTargetId ? { activeTargetId: this.activeTargetId } : {}), + generations: [...this.generations], + refs: [...this.refs].map(([ref, { sessionId: _sessionId, ...entry }]) => [ref, entry]), + }; + } + + /** Restore a ref table exported by a previous invocation against the same browser. */ + importRefState(state: BrowserRefState): void { + this.refCounter = Math.max(this.refCounter, state.refCounter); + this.activeTargetId = state.activeTargetId ?? this.activeTargetId; + for (const [frameId, generation] of state.generations) this.generations.set(frameId, generation); + for (const [ref, entry] of state.refs) this.refs.set(ref, { ...entry, sessionId: "" }); + } + + async execute(action: CuaBrowserAction): Promise { + const results = await this.dispatch(action); + const dialogs = this.drainDialogNotes(); + if (dialogs) results.push({ type: "browser_text", label: "dialog", text: dialogs }); + return results; + } + + private async dispatch(action: CuaBrowserAction): Promise { + switch (action.type) { + case "browser_snapshot": + return [{ type: "browser_text", label: "snapshot", text: await this.snapshot(action) }]; + case "browser_text": + return [{ type: "browser_text", label: "text", text: await this.pageText(tabOf(action)) }]; + case "browser_find": + return [{ type: "browser_text", label: "find", text: await this.find(action) }]; + case "browser_click": + await this.click(action); + return []; + case "browser_hover": + await this.hover(action); + return []; + case "browser_drag": + await this.drag(action); + return []; + case "browser_fill": + await this.fill(action); + return []; + case "browser_scroll_to": + await this.scrollTo(action); + return []; + case "browser_scroll": + await this.scroll(action); + return []; + case "browser_type": { + const session = await this.session(tabOf(action)); + await this.cdp.send("Input.insertText", { text: action.text }, session); + return []; + } + case "browser_key": + await this.key(action); + return []; + case "browser_navigate": + return [{ type: "browser_text", label: "navigate", text: await this.navigate(action) }]; + case "browser_list_tabs": + return [{ type: "browser_text", label: "tabs", text: await this.listTabs() }]; + case "browser_new_tab": + return [{ type: "browser_text", label: "new_tab", text: await this.newTab() }]; + case "browser_screenshot": + return [{ type: "screenshot", ...(await this.screenshot(action.region, action.tab_id)) }]; + case "browser_evaluate": + return [{ type: "browser_text", label: "evaluate", text: await this.evaluate(action.code, tabOf(action)) }]; + } + } + + async screenshot(region?: [number, number, number, number], tabId?: string): Promise<{ data: Buffer; mimeType: string }> { + const session = await this.session(tabId); + const clip = region + ? { + clip: { + x: Math.min(region[0], region[2]), + y: Math.min(region[1], region[3]), + width: Math.max(1, Math.abs(region[2] - region[0])), + height: Math.max(1, Math.abs(region[3] - region[1])), + scale: 1, + }, + } + : {}; + const { data } = await this.cdp.send<{ data: string }>("Page.captureScreenshot", { format: "png", ...clip }, session); + return { data: Buffer.from(data, "base64"), mimeType: "image/png" }; + } + + private async snapshot(action: CuaActionBrowserSnapshot): Promise { + const targetId = await this.resolveTarget(action.tab_id); + const pageSession = await this.attach(targetId); + const refEntry = action.ref ? this.resolveRef(action.ref, targetId) : undefined; + const frameKey = refEntry?.frameId ?? targetId; + const { nodes, sessionId } = await this.frameAxTree(frameKey, targetId, pageSession); + const byId = new Map(nodes.map((node) => [node.nodeId, node])); + let rootIds = nodes.filter((node) => !node.parentId).map((node) => node.nodeId); + if (action.ref && refEntry) { + const rootNode = + nodes.find((node) => node.backendDOMNodeId === refEntry.backendNodeId) ?? this.healEntry(action.ref, refEntry, nodes); + rootIds = [rootNode.nodeId]; + } + + const interactiveOnly = action.filter === "interactive"; + const ctx: RenderContext = { + targetId, + frameKey, + sessionId, + generation: this.generation(frameKey), + interactiveOnly, + nthIndex: buildNthIndex(nodes), + cursorIds: frameKey === targetId ? await this.cursorPointerIds(pageSession) : undefined, + }; + const stitches = frameKey === targetId ? await this.stitchFrames(nodes, targetId, pageSession, interactiveOnly) : new Map(); + const lines: RenderedLine[] = []; + const maxDepth = action.depth ?? DEFAULT_SNAPSHOT_DEPTH; + const walk = (tree: Map, treeCtx: RenderContext, nodeId: string, depth: number, parentName: string): void => { + const node = tree.get(nodeId); + if (!node) return; + let childDepth = depth; + if (!node.ignored) { + const rendered = this.renderNode(node, depth, parentName, treeCtx); + if (rendered) { + lines.push({ ...rendered, ctx: treeCtx }); + childDepth = depth + 1; + } + } + if (childDepth > maxDepth) return; + const stitch = treeCtx === ctx && node.backendDOMNodeId !== undefined ? stitches.get(node.backendDOMNodeId) : undefined; + if (stitch) { + for (const frameRootId of stitch.roots) walk(stitch.byId, stitch.ctx, frameRootId, childDepth, ""); + return; + } + const name = node.name?.value ?? ""; + const childName = name || parentName; + const childIds = node.childIds ?? []; + for (let i = 0; i < childIds.length; i += 1) { + const run = staticTextRun(tree, childIds, i); + if (run) { + const rendered = this.renderNode(run.node, childDepth, childName, treeCtx); + if (rendered) lines.push({ ...rendered, ctx: treeCtx }); + i = run.end; + continue; + } + walk(tree, treeCtx, childIds[i]!, childDepth, childName); + } + }; + for (const rootId of rootIds) walk(byId, ctx, rootId, 0, ""); + + const shape = lines.map((line) => line.text).join("\n"); + const frameGenerations = [...stitches.values()].map((stitch) => `${stitch.ctx.frameKey}:${stitch.ctx.generation}`); + const key = [action.ref ?? "", action.depth ?? "", action.filter ?? "", `${frameKey}:${ctx.generation}`, ...frameGenerations].join("|"); + const cached = this.lastSnapshots.get(targetId); + this.lastSnapshots.set(targetId, { key, shape }); + if (cached && cached.key === key && cached.shape === shape) return UNCHANGED_SNAPSHOT; + + let text = ""; + for (const line of lines) { + if (text.length > SNAPSHOT_CHAR_LIMIT) break; + const rendered = line.refNode ? line.text.replace(REF_PLACEHOLDER, this.mintRef(line.refNode, line.ctx)) : line.text; + text = text ? `${text}\n${rendered}` : rendered; + } + if (text.length > SNAPSHOT_CHAR_LIMIT) { + text = `${text.slice(0, SNAPSHOT_CHAR_LIMIT)}\n… truncated at ${SNAPSHOT_CHAR_LIMIT} characters. Re-request with a smaller depth, filter: "interactive", or a ref to narrow the subtree.`; + } + this.pruneRefs(targetId); + return text || "(empty accessibility tree)"; + } + + private renderNode(node: AXNode, depth: number, parentName: string, ctx: RenderContext): { text: string; refNode?: AXNode } | undefined { + const role = node.role?.value ?? ""; + const name = node.name?.value ?? ""; + const interactive = INTERACTIVE_ROLES.has(role); + const pointer = node.backendDOMNodeId !== undefined && (ctx.cursorIds?.has(node.backendDOMNodeId) ?? false); + if (ctx.interactiveOnly && !interactive && !pointer) return undefined; + if (role === "StaticText" && name === parentName) return undefined; + if (!ctx.interactiveOnly && !name && !interactive && !pointer && SKIPPED_ROLES.has(role)) return undefined; + let line = `${" ".repeat(Math.min(depth, 20))}${role || "node"}${name ? ` ${JSON.stringify(name)}` : ""}`; + let refNode: AXNode | undefined; + const refWorthy = interactive || pointer || FRAME_ROLES.has(role) || (name !== "" && CONTENT_ROLES.has(role)); + if (node.backendDOMNodeId !== undefined && refWorthy) { + line += ` [${REF_PLACEHOLDER}]`; + refNode = node; + } + const states = collectStates(node); + if (pointer && !interactive) states.push("cursor:pointer"); + if (states.length > 0) line += ` [${states.join(", ")}]`; + return { text: line, refNode }; + } + + /** Fetch a frame's AX tree: OOPIFs through their own session, same-process frames through the page session with a frameId. */ + private async frameAxTree(frameKey: string, targetId: string, pageSession: string): Promise<{ nodes: AXNode[]; sessionId: string }> { + const frameSession = this.frameSessions.get(frameKey); + if (frameSession) { + const { nodes } = await this.cdp.send<{ nodes: AXNode[] }>("Accessibility.getFullAXTree", {}, frameSession); + return { nodes, sessionId: frameSession }; + } + const params = frameKey === targetId ? {} : { frameId: frameKey }; + const { nodes } = await this.cdp.send<{ nodes: AXNode[] }>("Accessibility.getFullAXTree", params, pageSession); + return { nodes, sessionId: pageSession }; + } + + /** Resolve each iframe node's child frame and fetch its AX tree for stitching. One nesting level only. */ + private async stitchFrames( + nodes: AXNode[], + targetId: string, + pageSession: string, + interactiveOnly: boolean, + ): Promise> { + const stitches = new Map(); + for (const node of nodes) { + if (node.ignored || !FRAME_ROLES.has(node.role?.value ?? "") || node.backendDOMNodeId === undefined) continue; + try { + const { node: dom } = await this.cdp.send<{ node: { frameId?: string; contentDocument?: { frameId?: string } } }>( + "DOM.describeNode", + { backendNodeId: node.backendDOMNodeId, depth: 1 }, + pageSession, + ); + const frameId = dom.contentDocument?.frameId ?? dom.frameId; + if (!frameId || frameId === targetId) continue; + const { nodes: frameNodes, sessionId } = await this.frameAxTree(frameId, targetId, pageSession); + stitches.set(node.backendDOMNodeId, { + byId: new Map(frameNodes.map((frameNode) => [frameNode.nodeId, frameNode])), + roots: frameNodes.filter((frameNode) => !frameNode.parentId).map((frameNode) => frameNode.nodeId), + ctx: { + targetId, + frameKey: frameId, + sessionId, + generation: this.generation(frameId), + interactiveOnly, + nthIndex: buildNthIndex(frameNodes), + }, + }); + } catch { + // Cross-origin or already-detached frames can refuse the fetch; the iframe renders without children. + } + } + return stitches; + } + + /** Resolve backend node ids for elements whose own computed cursor is "pointer", without touching the DOM. */ + private async cursorPointerIds(session: string): Promise> { + const ids = new Set(); + const { result } = await this.cdp.send<{ result: { objectId?: string } }>( + "Runtime.evaluate", + { expression: CURSOR_POINTER_SCAN, returnByValue: false, objectGroup: CURSOR_SCAN_GROUP }, + session, + ); + if (!result.objectId) return ids; + try { + const { result: properties } = await this.cdp.send<{ result: Array<{ name: string; value?: { objectId?: string } }> }>( + "Runtime.getProperties", + { objectId: result.objectId, ownProperties: true }, + session, + ); + const objectIds = properties + .filter((property) => /^\d+$/.test(property.name) && property.value?.objectId) + .map((property) => property.value!.objectId!); + const described = await Promise.all( + objectIds.map((objectId) => this.cdp.send<{ node: { backendNodeId?: number } }>("DOM.describeNode", { objectId }, session)), + ); + for (const { node } of described) { + if (node.backendNodeId !== undefined) ids.add(node.backendNodeId); + } + } finally { + await this.cdp.send("Runtime.releaseObjectGroup", { objectGroup: CURSOR_SCAN_GROUP }, session).catch(() => {}); + } + return ids; + } + + private async find(action: CuaActionBrowserFind): Promise { + const candidates = await this.findCandidates(action.query, action.tab_id); + if (candidates.length === 0) return `No elements matched ${JSON.stringify(action.query)}. Try snapshot for the full tree.`; + return candidates + .map(({ ref, role, name }) => `${role || "node"}${name ? ` ${JSON.stringify(name)}` : ""} [${ref}]`) + .join("\n"); + } + + /** + * Score elements against a natural-language query and mint refs for the + * matches, best first. Structured counterpart of the `browser_find` action. + */ + async findCandidates(query: string, tabId?: string, roles?: ReadonlySet): Promise { + const targetId = await this.resolveTarget(tabId); + const session = await this.attach(targetId); + const { nodes } = await this.cdp.send<{ nodes: AXNode[] }>("Accessibility.getFullAXTree", {}, session); + const pools: Array<{ nodes: AXNode[]; ctx: RenderContext }> = [ + { + nodes, + ctx: { + targetId, + frameKey: targetId, + sessionId: session, + generation: this.generation(targetId), + interactiveOnly: false, + nthIndex: buildNthIndex(nodes), + }, + }, + ]; + // Search stitched frames too so find sees everything snapshot renders. + for (const stitch of (await this.stitchFrames(nodes, targetId, session, false)).values()) { + pools.push({ nodes: [...stitch.byId.values()], ctx: stitch.ctx }); + } + const queryTokens = tokenize(query); + const scored = pools + .flatMap(({ nodes: poolNodes, ctx }) => + poolNodes + .filter( + (node) => !node.ignored && node.backendDOMNodeId !== undefined && (node.name?.value || INTERACTIVE_ROLES.has(node.role?.value ?? "")), + ) + .map((node) => ({ node, ctx, score: overlapScore(queryTokens, tokenize(`${node.role?.value ?? ""} ${node.name?.value ?? ""}`)) })), + ) + .filter((entry) => entry.score > 0 && (!roles || roles.has(entry.node.role?.value ?? ""))) + .sort((a, b) => b.score - a.score) + .slice(0, FIND_MATCH_LIMIT); + const candidates = scored.map(({ node, ctx, score }) => ({ + ref: this.mintRef(node, ctx), + role: node.role?.value ?? "", + name: node.name?.value ?? "", + score, + })); + this.pruneRefs(targetId); + return candidates; + } + + private async click(action: CuaActionBrowserClick): Promise { + const targetId = await this.resolveTarget(action.tab_id); + const session = await this.attach(targetId); + const point = await this.resolvePoint(action, targetId, session); + const modifiers = modifierBits(action.modifiers); + const button = action.button ?? "left"; + const clicks = action.num_clicks ?? 1; + await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: point.x, y: point.y, modifiers }, point.session); + // Native multi-clicks are separate press/release cycles with an + // incrementing clickCount; a single pair with the final count is not how + // real input arrives and can register as one click. + for (let clickCount = 1; clickCount <= clicks; clickCount++) { + await this.cdp.send( + "Input.dispatchMouseEvent", + { type: "mousePressed", x: point.x, y: point.y, button, clickCount, modifiers }, + point.session, + ); + await this.cdp.send( + "Input.dispatchMouseEvent", + { type: "mouseReleased", x: point.x, y: point.y, button, clickCount, modifiers }, + point.session, + ); + } + } + + private async hover(action: CuaActionBrowserHover): Promise { + const targetId = await this.resolveTarget(action.tab_id); + const session = await this.attach(targetId); + const point = await this.resolvePoint(action, targetId, session); + await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: point.x, y: point.y }, point.session); + } + + private async drag(action: CuaActionBrowserDrag): Promise { + const session = await this.session(tabOf(action)); + await this.cdp.send("Input.dispatchMouseEvent", { type: "mousePressed", x: action.from.x, y: action.from.y, button: "left", clickCount: 1 }, session); + await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: action.to.x, y: action.to.y, button: "left" }, session); + await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseReleased", x: action.to.x, y: action.to.y, button: "left", clickCount: 1 }, session); + } + + private async fill(action: CuaActionBrowserFill): Promise { + const targetId = await this.resolveTarget(action.tab_id); + const entry = this.resolveRef(action.ref, targetId); + const session = await this.refSession(entry); + const objectId = await this.resolveObject(entry, action.ref, session); + const { exceptionDetails } = await this.cdp.send<{ exceptionDetails?: { exception?: { description?: string } } }>( + "Runtime.callFunctionOn", + { + objectId, + functionDeclaration: FILL_FUNCTION, + arguments: [{ value: action.value }], + }, + session, + ); + if (exceptionDetails) { + const description = exceptionDetails.exception?.description ?? "element rejected the value"; + const message = description.split("\n", 1)[0]?.replace(/^[A-Za-z]*(?:Error|Exception): /, "") || "element rejected the value"; + throw new Error(`browser_fill failed: ${message}`); + } + } + + private async scrollTo(action: CuaActionBrowserScrollTo): Promise { + const targetId = await this.resolveTarget(action.tab_id); + const entry = this.resolveRef(action.ref, targetId); + await this.scrollIntoView(entry, action.ref, await this.refSession(entry)); + } + + private async scroll(action: CuaActionBrowserScroll): Promise { + const session = await this.session(tabOf(action)); + const notches = action.amount ?? 3; + const delta = Math.trunc(notches) * SCROLL_NOTCH_PX; + const deltaX = action.direction === "left" ? -delta : action.direction === "right" ? delta : 0; + const deltaY = action.direction === "up" ? -delta : action.direction === "down" ? delta : 0; + await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseWheel", x: action.x, y: action.y, deltaX, deltaY }, session); + } + + private async key(action: CuaActionBrowserKey): Promise { + const session = await this.session(tabOf(action)); + const repeat = Math.min(Math.max(1, Math.trunc(action.repeat ?? 1)), 100); + const chords = action.text.trim().split(/\s+/).filter(Boolean); + for (let iteration = 0; iteration < repeat; iteration += 1) { + for (const chord of chords) { + await this.dispatchChord(chord, session); + } + } + } + + private async dispatchChord(chord: string, session: string): Promise { + const parts = chord.split("+").filter(Boolean); + const keyPart = parts[parts.length - 1] ?? ""; + const modifierParts = parts.slice(0, -1); + const modifiers = modifierBits(modifierParts); + const key = resolveKey(keyPart); + const base = { key: key.key, code: key.code, windowsVirtualKeyCode: key.keyCode, modifiers }; + await this.cdp.send("Input.dispatchKeyEvent", { type: key.text ? "keyDown" : "rawKeyDown", ...base, ...(key.text ? { text: key.text } : {}) }, session); + await this.cdp.send("Input.dispatchKeyEvent", { type: "keyUp", ...base }, session); + } + + private async navigate(action: CuaActionBrowserNavigate): Promise { + const targetId = await this.resolveTarget(action.tab_id); + const session = await this.attach(targetId); + const direction = action.url.trim().toLowerCase(); + if (direction === "back" || direction === "forward") { + const history = await this.cdp.send<{ currentIndex: number; entries: Array<{ id: number; url: string }> }>( + "Page.getNavigationHistory", + {}, + session, + ); + const entry = history.entries[history.currentIndex + (direction === "back" ? -1 : 1)]; + if (!entry) throw new Error(`cannot go ${direction}: no history entry`); + await this.selfNavigate(targetId, () => this.cdp.send("Page.navigateToHistoryEntry", { entryId: entry.id }, session)); + this.invalidateRefs(targetId); + return `Navigated ${direction}.\n${await this.tabContext(targetId)}`; + } + const url = normalizeGotoUrl(action.url); + if (!url) throw new Error("invalid url"); + const { errorText } = await this.selfNavigate(targetId, () => + this.cdp.send<{ errorText?: string }>("Page.navigate", { url }, session), + ); + if (errorText) { + this.selfNavigations.delete(targetId); + throw new Error(`navigation to ${url} failed: ${errorText}`); + } + this.invalidateRefs(targetId); + return `Navigated to ${url}.\n${await this.tabContext(targetId)}`; + } + + /** + * Run a navigation command with the self-navigation flag armed. The flag + * must be set before the command (frameNavigated can arrive first), and + * must not survive a rejected command — it would swallow the next real + * navigation's invalidation. + */ + private async selfNavigate(targetId: string, command: () => Promise): Promise { + this.selfNavigations.add(targetId); + try { + return await command(); + } catch (err) { + this.selfNavigations.delete(targetId); + throw err; + } + } + + /** URL of the active tab. */ + async currentUrl(): Promise { + const targetId = await this.resolveTarget(); + const targets = await this.cdp.pageTargets(); + return targets.find((target) => target.targetId === targetId)?.url ?? ""; + } + + private async listTabs(): Promise { + const targets = await this.cdp.pageTargets(); + if (targets.length === 0) return "No open tabs."; + return targets.map((target) => `tab_id ${shortTabId(target.targetId)}: ${JSON.stringify(target.title)} (${target.url})`).join("\n"); + } + + private async newTab(): Promise { + const targetId = await this.cdp.createTarget("about:blank"); + this.activeTargetId = targetId; + return `Opened tab_id ${shortTabId(targetId)}.\n${await this.tabContext(targetId)}`; + } + + private async evaluate(code: string, tabId?: string): Promise { + const session = await this.session(tabId); + const { result, exceptionDetails } = await this.cdp.send<{ + result: { value?: unknown; description?: string; type?: string }; + exceptionDetails?: { exception?: { description?: string } }; + }>("Runtime.evaluate", { expression: code, returnByValue: true, awaitPromise: true }, session); + if (exceptionDetails) throw new Error(`browser_evaluate failed: ${exceptionDetails.exception?.description ?? "evaluation threw"}`); + if (result.value === undefined) return result.description ?? String(result.type ?? "undefined"); + return typeof result.value === "string" ? result.value : JSON.stringify(result.value); + } + + private async pageText(tabId?: string): Promise { + return this.evaluate("document.body ? document.body.innerText : ''", tabId); + } + + private async tabContext(executedOn: string): Promise { + const targets = await this.cdp.pageTargets(); + const lines = targets.map((target) => ` • tab_id ${shortTabId(target.targetId)}: ${JSON.stringify(target.title)} (${target.url})`); + return [`Tab Context:`, `- Executed on tab_id: ${shortTabId(executedOn)}`, `- Available tabs:`, ...lines].join("\n"); + } + + private async resolvePoint( + action: CuaActionBrowserClick | CuaActionBrowserHover, + targetId: string, + session: string, + ): Promise<{ x: number; y: number; session: string }> { + if (action.ref !== undefined) { + const entry = this.resolveRef(action.ref, targetId); + const refSession = await this.refSession(entry); + await this.scrollIntoView(entry, action.ref, refSession); + const { model } = await this.cdp.send<{ model: { content: number[] } }>( + "DOM.getBoxModel", + { backendNodeId: entry.backendNodeId }, + refSession, + ); + const quad = model.content; + // Box-model quads are main-viewport coordinates even through an OOPIF's + // session, so input always dispatches on the page target's session. + return { x: (quad[0]! + quad[4]!) / 2, y: (quad[1]! + quad[5]!) / 2, session }; + } + if (typeof action.x === "number" && typeof action.y === "number") return { x: action.x, y: action.y, session }; + throw new Error("page target required: pass a ref or viewport coordinates"); + } + + private async scrollIntoView(entry: RefEntry, ref: string, session: string): Promise { + try { + await this.cdp.send("DOM.scrollIntoViewIfNeeded", { backendNodeId: entry.backendNodeId }, session); + } catch (err) { + await this.healRef(ref, entry, err); + await this.cdp.send("DOM.scrollIntoViewIfNeeded", { backendNodeId: entry.backendNodeId }, session); + } + } + + private async resolveObject(entry: RefEntry, ref: string, session: string): Promise { + try { + const { object } = await this.cdp.send<{ object: { objectId: string } }>( + "DOM.resolveNode", + { backendNodeId: entry.backendNodeId }, + session, + ); + return object.objectId; + } catch (err) { + await this.healRef(ref, entry, err); + const { object } = await this.cdp.send<{ object: { objectId: string } }>( + "DOM.resolveNode", + { backendNodeId: entry.backendNodeId }, + session, + ); + return object.objectId; + } + } + + private async healRef(ref: string, entry: RefEntry, cause: unknown): Promise { + const { nodes } = await this.frameAxTree(entry.frameId, entry.targetId, await this.refSession(entry)); + this.healEntry(ref, entry, nodes, cause); + } + + /** + * Re-resolve a stale entry by its (role, name, nth) triple against a fresh + * AX tree. Heals only when the fresh role+name cohort has the same size as + * at mint time, so the nth position still identifies the same element. + */ + private healEntry(ref: string, entry: RefEntry, nodes: AXNode[], cause?: unknown): AXNode { + const candidates = nodes.filter( + (node) => + !node.ignored && + node.backendDOMNodeId !== undefined && + (node.role?.value ?? "") === entry.role && + (node.name?.value ?? "") === entry.name, + ); + const match = + (entry.role || entry.name) && candidates.length === entry.cohort && entry.nth < candidates.length + ? candidates[entry.nth] + : undefined; + if (!match) throw staleRefError(ref, cause); + entry.backendNodeId = match.backendDOMNodeId!; + return match; + } + + private mintRef(node: AXNode, ctx: RenderContext): string { + const role = node.role?.value ?? ""; + const name = node.name?.value ?? ""; + this.refCounter += 1; + const ref = `e${this.refCounter}`; + this.refs.set(ref, { + backendNodeId: node.backendDOMNodeId!, + targetId: ctx.targetId, + frameId: ctx.frameKey, + sessionId: ctx.sessionId, + generation: ctx.generation, + role, + name, + nth: ctx.nthIndex.index.get(node.nodeId) ?? 0, + cohort: ctx.nthIndex.cohorts.get(cohortKey(role, name)) ?? 1, + }); + return ref; + } + + /** + * Session for a ref's DOM/Input calls. Imported refs (see + * {@link importRefState}) carry no live session and rebind here: the + * frame's own session for OOPIFs when auto-attach has surfaced it, the + * page session otherwise. + */ + private async refSession(entry: RefEntry): Promise { + if (!entry.sessionId) { + entry.sessionId = this.frameSessions.get(entry.frameId) ?? (await this.attach(entry.targetId)); + } + return entry.sessionId; + } + + private resolveRef(ref: string, targetId: string): RefEntry { + const entry = this.refs.get(ref); + // Entries are deleted eagerly on invalidation; the generation check only + // guards refs resolved while a navigation event is still in flight. + if (!entry || entry.targetId !== targetId || entry.generation !== this.generation(entry.frameId)) { + throw staleRefError(ref); + } + return entry; + } + + private generation(targetId: string): number { + return this.generations.get(targetId) ?? 0; + } + + private invalidateRefs(targetId: string): void { + this.generations.set(targetId, this.generation(targetId) + 1); + for (const [ref, entry] of this.refs) { + if (entry.targetId === targetId) this.refs.delete(ref); + } + } + + private invalidateFrame(frameKey: string): void { + let tracked = this.generations.has(frameKey) || this.frameSessions.has(frameKey); + for (const [ref, entry] of this.refs) { + if (entry.frameId === frameKey) { + this.refs.delete(ref); + tracked = true; + } + } + // Frames we never referenced don't get a generation entry, or pages with + // rotating ad iframes would grow the map without bound. + if (tracked) this.generations.set(frameKey, this.generation(frameKey) + 1); + } + + private dropTarget(targetId: string): void { + this.generations.delete(targetId); + this.selfNavigations.delete(targetId); + this.lastSnapshots.delete(targetId); + this.frameSessions.delete(targetId); + this.frameTargets.delete(targetId); + for (const [ref, entry] of this.refs) { + if (entry.targetId === targetId || entry.frameId === targetId) { + if (entry.frameId !== targetId) this.generations.delete(entry.frameId); + this.refs.delete(ref); + } + } + } + + /** SPAs can mint refs indefinitely without ever navigating; bound per-target growth by evicting the oldest. */ + private pruneRefs(targetId: string): void { + const owned: string[] = []; + for (const [ref, entry] of this.refs) { + if (entry.targetId === targetId) owned.push(ref); + } + for (const ref of owned.slice(0, Math.max(0, owned.length - REF_LIMIT_PER_TARGET))) this.refs.delete(ref); + } + + private drainDialogNotes(): string | undefined { + if (this.dialogNotes.length === 0) return undefined; + const text = this.dialogNotes.join("\n"); + this.dialogNotes.length = 0; + return text; + } + + private async session(tabId?: string): Promise { + return this.attach(await this.resolveTarget(tabId)); + } + + private async attach(targetId: string): Promise { + const session = await this.cdp.attachToTarget(targetId); + if (!this.targetsBySession.has(session)) { + this.targetsBySession.set(session, targetId); + await this.cdp.send("Page.enable", {}, session); + await this.cdp.send("Target.setAutoAttach", { autoAttach: true, flatten: true, waitForDebuggerOnStart: false }, session); + } + return session; + } + + private async resolveTarget(tabId?: string): Promise { + const targets = await this.cdp.pageTargets(); + if (targets.length === 0) throw new Error("no open browser tabs"); + if (tabId) { + const match = targets.find((target) => shortTabId(target.targetId) === tabId || target.targetId === tabId); + if (!match) throw new Error(`unknown tab_id "${tabId}". Call list_tabs for current tabs.`); + this.activeTargetId = match.targetId; + return match.targetId; + } + if (this.activeTargetId && targets.some((target) => target.targetId === this.activeTargetId)) { + return this.activeTargetId; + } + this.activeTargetId = targets[0]!.targetId; + return this.activeTargetId; + } +} + +function tabOf(action: { tab_id?: string }): string | undefined { + return action.tab_id; +} + +function staleRefError(ref: string, cause?: unknown): Error { + return new Error(`ref ${ref} is stale or not on the current page. ${STALE_REF_HINT}`, cause === undefined ? undefined : { cause }); +} + +/** Index each ref-eligible node by its position among nodes with the same role and name, in tree order. */ +function buildNthIndex(nodes: AXNode[]): NthIndex { + const cohorts = new Map(); + const index = new Map(); + for (const node of nodes) { + if (node.ignored || node.backendDOMNodeId === undefined) continue; + const key = cohortKey(node.role?.value ?? "", node.name?.value ?? ""); + const nth = cohorts.get(key) ?? 0; + cohorts.set(key, nth + 1); + index.set(node.nodeId, nth); + } + return { index, cohorts }; +} + +function cohortKey(role: string, name: string): string { + return `${role}\u0000${name}`; +} + +/** Merge a run of two or more consecutive StaticText siblings (text split by inline markup) into one node. */ +function staticTextRun(tree: Map, childIds: string[], start: number): { node: AXNode; end: number } | undefined { + let end = start; + const parts: string[] = []; + while (end < childIds.length) { + const node = tree.get(childIds[end]!); + if (!node || node.ignored || node.role?.value !== "StaticText") break; + const text = node.name?.value ?? ""; + if (text) parts.push(text); + end += 1; + } + if (end - start < 2) return undefined; + const first = tree.get(childIds[start]!)!; + return { node: { ...first, name: { value: parts.join(" ") }, childIds: [] }, end: end - 1 }; +} + +function collectStates(node: AXNode): string[] { + const states: string[] = []; + for (const property of node.properties ?? []) { + const value = property.value?.value; + switch (property.name) { + case "checked": + case "pressed": + case "expanded": + // False is meaningful here: it distinguishes an unchecked checkbox or + // collapsed disclosure from an element without the state at all. + if (value === true || value === "true") states.push(property.name); + else if (value === false || value === "false") states.push(`${property.name}=false`); + else if (value === "mixed") states.push(`${property.name}=mixed`); + break; + case "disabled": + case "selected": + case "required": + if (value === true || value === "true") states.push(property.name); + break; + case "level": + if (typeof value === "number") states.push(`level=${value}`); + break; + } + } + const value = node.value?.value; + if (value !== undefined && value !== "" && String(value) !== (node.name?.value ?? "")) { + states.push(`value=${JSON.stringify(String(value))}`); + } + return states; +} + +function shortTabId(targetId: string): string { + return targetId.slice(0, 10).toUpperCase(); +} + +function tokenize(value: string): string[] { + return value + .toLowerCase() + .split(/[^a-z0-9]+/) + .filter((token) => token.length > 1); +} + +function overlapScore(query: string[], candidate: string[]): number { + if (query.length === 0 || candidate.length === 0) return 0; + const set = new Set(candidate); + return query.reduce((score, token) => score + (set.has(token) ? 1 : 0), 0); +} + +function modifierBits(modifiers: readonly string[] | undefined): number { + let bits = 0; + for (const modifier of modifiers ?? []) { + switch (modifier.trim().toLowerCase()) { + case "alt": + case "option": + bits |= 1; + break; + case "ctrl": + case "control": + bits |= 2; + break; + case "meta": + case "cmd": + case "command": + case "super": + bits |= 4; + break; + case "shift": + bits |= 8; + break; + } + } + return bits; +} + +interface ResolvedKey { + key: string; + code: string; + keyCode: number; + text?: string; +} + +const NAMED_KEYS: Record = { + enter: { key: "Enter", code: "Enter", keyCode: 13, text: "\r" }, + return: { key: "Enter", code: "Enter", keyCode: 13, text: "\r" }, + tab: { key: "Tab", code: "Tab", keyCode: 9 }, + escape: { key: "Escape", code: "Escape", keyCode: 27 }, + esc: { key: "Escape", code: "Escape", keyCode: 27 }, + backspace: { key: "Backspace", code: "Backspace", keyCode: 8 }, + delete: { key: "Delete", code: "Delete", keyCode: 46 }, + space: { key: " ", code: "Space", keyCode: 32, text: " " }, + up: { key: "ArrowUp", code: "ArrowUp", keyCode: 38 }, + down: { key: "ArrowDown", code: "ArrowDown", keyCode: 40 }, + left: { key: "ArrowLeft", code: "ArrowLeft", keyCode: 37 }, + right: { key: "ArrowRight", code: "ArrowRight", keyCode: 39 }, + arrowup: { key: "ArrowUp", code: "ArrowUp", keyCode: 38 }, + arrowdown: { key: "ArrowDown", code: "ArrowDown", keyCode: 40 }, + arrowleft: { key: "ArrowLeft", code: "ArrowLeft", keyCode: 37 }, + arrowright: { key: "ArrowRight", code: "ArrowRight", keyCode: 39 }, + home: { key: "Home", code: "Home", keyCode: 36 }, + end: { key: "End", code: "End", keyCode: 35 }, + pageup: { key: "PageUp", code: "PageUp", keyCode: 33 }, + page_up: { key: "PageUp", code: "PageUp", keyCode: 33 }, + pagedown: { key: "PageDown", code: "PageDown", keyCode: 34 }, + page_down: { key: "PageDown", code: "PageDown", keyCode: 34 }, +}; + +function resolveKey(value: string): ResolvedKey { + const named = NAMED_KEYS[value.toLowerCase()]; + if (named) return named; + if (value.length === 1) { + const upper = value.toUpperCase(); + const code = /[a-z]/i.test(value) ? `Key${upper}` : /[0-9]/.test(value) ? `Digit${value}` : ""; + return { key: value, code, keyCode: upper.charCodeAt(0), text: value }; + } + return { key: value, code: value, keyCode: 0 }; +} + +const FILL_FUNCTION = `function(value) { + const el = this; + const tag = el.tagName ? el.tagName.toLowerCase() : ""; + if (tag === "select") { + const str = String(value); + const options = Array.from(el.options); + const match = options.find((o) => o.value === str) ?? options.find((o) => (o.label || o.textContent || "").trim() === str); + if (!match) throw new Error("no option matches " + JSON.stringify(str)); + el.value = match.value; + } else if (el.type === "checkbox" || el.type === "radio") { + el.checked = Boolean(value); + } else if (tag === "input" || tag === "textarea") { + const proto = Object.getPrototypeOf(el); + const setter = Object.getOwnPropertyDescriptor(proto, "value"); + if (setter && setter.set) setter.set.call(el, String(value)); + else el.value = String(value); + } else if (el.isContentEditable) { + el.textContent = String(value); + } else { + throw new Error("element is not a form control"); + } + el.focus(); + el.dispatchEvent(new Event("input", { bubbles: true })); + el.dispatchEvent(new Event("change", { bubbles: true })); +}`; + +const INTERACTIVE_ROLES: ReadonlySet = new Set([ + "button", + "link", + "textbox", + "searchbox", + "checkbox", + "radio", + "combobox", + "listbox", + "option", + "menuitem", + "menuitemcheckbox", + "menuitemradio", + "slider", + "spinbutton", + "switch", + "tab", + "treeitem", +]); + +const SKIPPED_ROLES: ReadonlySet = new Set(["none", "generic", "InlineTextBox", "LineBreak", "StaticText"]); + +const FRAME_ROLES: ReadonlySet = new Set(["Iframe", "IframePresentational"]); + +/** Non-interactive roles that get refs when named, so scroll_to / ref-scoped snapshots can target them. */ +const CONTENT_ROLES: ReadonlySet = new Set([ + "heading", + "cell", + "gridcell", + "columnheader", + "rowheader", + "row", + "listitem", + "article", + "region", + "main", + "navigation", + "banner", + "contentinfo", + "complementary", + "tabpanel", + "figure", + "image", +]); + +const CURSOR_SCAN_GROUP = "cua-cursor-scan"; + +const CURSOR_POINTER_SCAN = `(() => { + const matches = []; + if (!document.body) return matches; + for (const el of document.body.querySelectorAll("*")) { + if (matches.length >= 100) break; + if (el.closest("a, button, input, select, textarea, summary")) continue; + if (getComputedStyle(el).cursor !== "pointer") continue; + const parent = el.parentElement; + if (parent && getComputedStyle(parent).cursor === "pointer") continue; + const rect = el.getBoundingClientRect(); + if (rect.width === 0 || rect.height === 0) continue; + matches.push(el); + } + return matches; +})()`; + + diff --git a/packages/agent/src/translator/cdp.ts b/packages/agent/src/translator/cdp.ts new file mode 100644 index 00000000..c38604ae --- /dev/null +++ b/packages/agent/src/translator/cdp.ts @@ -0,0 +1,151 @@ +/** + * Minimal raw-CDP client for the browser action plane. + * + * Connects to a Kernel browser's `cdp_ws_url` over a plain WebSocket and + * speaks the DevTools JSON-RPC protocol directly — no Playwright and no + * driver dependency. Only what the browser executor needs: command dispatch on + * the browser connection and on attached page sessions. + */ + +interface PendingCommand { + resolve(result: unknown): void; + reject(error: Error): void; +} + +export interface CdpEventMessage { + method: string; + params: Record; + sessionId?: string; +} + +export interface CdpTargetInfo { + targetId: string; + type: string; + title: string; + url: string; + attached?: boolean; +} + +export type CdpEventListener = (event: CdpEventMessage) => void; + +export class CdpConnection { + private socket?: WebSocket; + private opening?: Promise; + private nextId = 1; + private readonly pending = new Map(); + private readonly listeners = new Set(); + private readonly sessionsByTarget = new Map(); + + constructor(private readonly wsUrl: string) {} + + onEvent(listener: CdpEventListener): void { + this.listeners.add(listener); + } + + async send>(method: string, params: Record = {}, sessionId?: string): Promise { + const socket = await this.connect(); + const id = this.nextId++; + const message = JSON.stringify({ id, method, params, ...(sessionId ? { sessionId } : {}) }); + return new Promise((resolve, reject) => { + this.pending.set(id, { resolve: resolve as (result: unknown) => void, reject }); + // A non-OPEN socket silently discards send() per the WebSocket spec, + // which would leave this command pending forever. + if (socket.readyState !== WebSocket.OPEN) { + this.pending.delete(id); + reject(new Error(`CDP connection closed before ${method} could be sent`)); + return; + } + try { + socket.send(message); + } catch (err) { + this.pending.delete(id); + reject(err instanceof Error ? err : new Error(String(err))); + } + }); + } + + /** List page targets (tabs) on the browser connection. */ + async pageTargets(): Promise { + const { targetInfos } = await this.send<{ targetInfos: CdpTargetInfo[] }>("Target.getTargets"); + return targetInfos.filter((target) => target.type === "page"); + } + + /** Attach to a page target (flat protocol) and cache the session. */ + async attachToTarget(targetId: string): Promise { + const cached = this.sessionsByTarget.get(targetId); + if (cached) return cached; + const { sessionId } = await this.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true }); + this.sessionsByTarget.set(targetId, sessionId); + return sessionId; + } + + async createTarget(url: string): Promise { + const { targetId } = await this.send<{ targetId: string }>("Target.createTarget", { url }); + return targetId; + } + + close(): void { + this.socket?.close(); + this.socket = undefined; + this.opening = undefined; + this.sessionsByTarget.clear(); + this.rejectPending(new Error("CDP connection closed")); + } + + private rejectPending(error: Error): void { + for (const pending of this.pending.values()) pending.reject(error); + this.pending.clear(); + } + + private connect(): Promise { + if (this.socket && this.socket.readyState === WebSocket.OPEN) return Promise.resolve(this.socket); + this.opening ??= new Promise((resolve, reject) => { + const socket = new WebSocket(this.wsUrl); + socket.addEventListener("open", () => { + this.socket = socket; + resolve(socket); + }); + socket.addEventListener("error", () => { + this.opening = undefined; + reject(new Error(`CDP connection to ${this.wsUrl} failed`)); + }); + socket.addEventListener("close", () => { + this.socket = undefined; + this.opening = undefined; + this.sessionsByTarget.clear(); + this.rejectPending(new Error("CDP connection closed")); + }); + socket.addEventListener("message", (event) => { + this.handleMessage(typeof event.data === "string" ? event.data : String(event.data)); + }); + }); + return this.opening; + } + + private handleMessage(data: string): void { + let message: { id?: number; result?: unknown; error?: { message?: string }; method?: string; params?: Record; sessionId?: string }; + try { + message = JSON.parse(data); + } catch { + return; + } + if (typeof message.id === "number") { + const pending = this.pending.get(message.id); + if (!pending) return; + this.pending.delete(message.id); + if (message.error) pending.reject(new Error(message.error.message ?? "CDP command failed")); + else pending.resolve(message.result ?? {}); + return; + } + if (typeof message.method === "string") { + if (message.method === "Target.detachedFromTarget") { + const detached = message.params?.sessionId; + for (const [targetId, sessionId] of this.sessionsByTarget) { + if (sessionId === detached) this.sessionsByTarget.delete(targetId); + } + } + const event: CdpEventMessage = { method: message.method, params: message.params ?? {}, sessionId: message.sessionId }; + for (const listener of this.listeners) listener(event); + } + } +} diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts index 26d309c4..5610d31b 100644 --- a/packages/agent/src/translator/translator.ts +++ b/packages/agent/src/translator/translator.ts @@ -1,6 +1,7 @@ import type Kernel from "@onkernel/sdk"; import type { BrowserCreateResponse, BrowserRetrieveResponse } from "@onkernel/sdk/resources/browsers"; import { + isCuaBrowserAction, normalizeGotoUrl, type ComputerToolCoordinateSystem, type CuaAction, @@ -13,11 +14,14 @@ import { type CuaActionScroll, type CuaActionTypeText, type CuaActionWait, + type CuaActionZoom, + type CuaBrowserAction, type CuaDragMouseButton, type CuaMouseButton, type CuaScreenshotSpec, } from "@onkernel/cua-ai"; import sharp from "sharp"; +import { BrowserExecutor } from "./browser"; import { isKernelModifierKey, normalizeKernelKey, normalizeKernelKeyCombo } from "./keys"; import type { BatchExecutionResult } from "./types"; @@ -28,6 +32,8 @@ export interface InternalComputerTranslatorOptions { client: Kernel; coordinateSystem?: ComputerToolCoordinateSystem; screenshot?: CuaScreenshotSpec; + /** Browser executor factory, overridable for tests. Defaults to a raw-CDP executor on the browser's cdp_ws_url. */ + createBrowserExecutor?: (cdpWsUrl: string) => BrowserExecutor; } export class InternalComputerTranslator { @@ -36,6 +42,9 @@ export class InternalComputerTranslator { private readonly coordinateSystem: ComputerToolCoordinateSystem; private readonly screenshotSpec?: CuaScreenshotSpec; private readonly viewport: { width: number; height: number }; + private readonly cdpWsUrl?: string; + private readonly browserExecutorFactory: (cdpWsUrl: string) => BrowserExecutor; + private browserExecutor?: BrowserExecutor; constructor(opts: InternalComputerTranslatorOptions) { this.sessionId = opts.browser.session_id; @@ -43,6 +52,23 @@ export class InternalComputerTranslator { this.coordinateSystem = opts.coordinateSystem ?? { type: "pixel" }; this.screenshotSpec = opts.screenshot; this.viewport = opts.browser.viewport ?? { width: 1920, height: 1080 }; + this.cdpWsUrl = opts.browser.cdp_ws_url; + this.browserExecutorFactory = opts.createBrowserExecutor ?? ((cdpWsUrl) => new BrowserExecutor(cdpWsUrl)); + } + + /** Release held resources: closes the browser executor's CDP connection if one was opened. */ + dispose(): void { + this.browserExecutor?.close(); + this.browserExecutor = undefined; + } + + /** The browser-plane executor, connected lazily over the browser's CDP websocket. */ + browser(): BrowserExecutor { + if (!this.browserExecutor) { + if (!this.cdpWsUrl) throw new Error("browser has no cdp_ws_url; browser actions are unavailable"); + this.browserExecutor = this.browserExecutorFactory(this.cdpWsUrl); + } + return this.browserExecutor; } async screenshotRaw(): Promise { @@ -106,11 +132,20 @@ export class InternalComputerTranslator { }; for (const action of actions) { + if (isCuaBrowserAction(action)) { + await flush(); + result.readResults.push(...(await this.browser().execute(action))); + continue; + } switch (action.type) { case "screenshot": await flush(); result.readResults.push({ type: "screenshot", ...(await this.screenshot()) }); break; + case "zoom": + await flush(); + result.readResults.push({ type: "screenshot", ...(await this.zoom(action)) }); + break; case "url": await flush(); result.readResults.push({ type: "url", url: await this.currentUrl() }); @@ -133,6 +168,17 @@ export class InternalComputerTranslator { pending.push(keypress(["Alt", "Right"])); break; default: + // Native computer mappings may omit click coordinates, meaning + // "at the current cursor position" — resolve before batching. + if ( + (action.type === "click" || action.type === "mouse_down" || action.type === "mouse_up") && + (action.x === undefined || action.y === undefined) + ) { + await flush(); + const position = await this.currentMousePosition(); + pending.push(this.toSdkAction({ ...action, x: action.x ?? position.x, y: action.y ?? position.y })); + break; + } pending.push(this.toSdkAction(action)); break; } @@ -142,12 +188,27 @@ export class InternalComputerTranslator { return result; } + /** Crop the OS screenshot to a region; coordinates stay in the full-screenshot frame. */ + async zoom(action: CuaActionZoom): Promise<{ data: Buffer; mimeType: string }> { + const screenshot = await this.screenshot(); + const [x0, y0, x1, y1] = action.region; + const left = Math.max(0, Math.trunc(Math.min(x0, x1))); + const top = Math.max(0, Math.trunc(Math.min(y0, y1))); + const width = Math.max(1, Math.trunc(Math.abs(x1 - x0))); + const height = Math.max(1, Math.trunc(Math.abs(y1 - y0))); + const data = await sharp(screenshot.data).extract({ left, top, width, height }).png().toBuffer(); + return { data, mimeType: "image/png" }; + } + private toSdkAction( - action: Exclude, + action: Exclude, ): KernelBatchAction { switch (action.type) { case "click": - return this.clickAction(action, { button: mouseButton(action.button) }); + return this.clickAction(action, { + button: mouseButton(action.button), + ...(action.num_clicks !== undefined && action.num_clicks > 1 ? { num_clicks: Math.trunc(action.num_clicks) } : {}), + }); case "double_click": return this.clickAction(action, { num_clicks: 2 }); case "mouse_down": @@ -175,7 +236,7 @@ export class InternalComputerTranslator { action: CuaActionClick | CuaActionDoubleClick | CuaActionMouseDown | CuaActionMouseUp, extra: { button?: CuaMouseButton; num_clicks?: number; click_type?: "down" | "up" }, ): KernelBatchAction { - const point = this.toViewportPoint(action.x, action.y); + const point = this.toViewportPoint(action.x ?? 0, action.y ?? 0); return { type: "click_mouse", click_mouse: { diff --git a/packages/agent/src/translator/types.ts b/packages/agent/src/translator/types.ts index e9aaca37..55f793d7 100644 --- a/packages/agent/src/translator/types.ts +++ b/packages/agent/src/translator/types.ts @@ -1,7 +1,8 @@ export type BatchReadResult = | { type: "screenshot"; data: Buffer; mimeType: string } | { type: "url"; url: string } - | { type: "cursor_position"; x: number; y: number }; + | { type: "cursor_position"; x: number; y: number } + | { type: "browser_text"; label: string; text: string }; export interface BatchExecutionResult { readResults: BatchReadResult[]; diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index 686231c0..d7d2273e 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { createAssistantMessageEventStream, type AssistantMessage } from "@earendil-works/pi-ai"; import { resolveCuaRuntimeSpec } from "@onkernel/cua-ai"; import type Kernel from "@onkernel/sdk"; @@ -92,7 +92,7 @@ describe("CuaAgent", () => { }, }); - expect(agent.state.tools.map((item) => item.name)).toEqual([...runtime.toolExecutors.map((item) => item.definition.name), "custom"]); + expect(agent.state.tools.map((item) => item.name)).toEqual([...runtime.toolExecutors.map((item) => item.definition.name), "computer_use_extra", "custom"]); }); it("always keeps provider CUA tools when adding extra tools", () => { @@ -109,7 +109,7 @@ describe("CuaAgent", () => { }, }); - expect(agent.state.tools.map((item) => item.name)).toEqual([...runtime.toolExecutors.map((item) => item.definition.name), "custom"]); + expect(agent.state.tools.map((item) => item.name)).toEqual([...runtime.toolExecutors.map((item) => item.definition.name), "computer_use_extra", "custom"]); expect(agent.state.systemPrompt).toBe("Use the browser carefully."); }); @@ -124,15 +124,14 @@ describe("CuaAgent", () => { }); expect(runtime.toolDefinitions.map((tool) => tool.name)).toContain(ANTHROPIC_BATCH_TOOL_NAME); - expect(agent.state.tools.map((tool) => tool.name)).toEqual(runtime.toolExecutors.map((tool) => tool.definition.name)); + expect(agent.state.tools.map((tool) => tool.name)).toEqual([...runtime.toolExecutors.map((tool) => tool.definition.name), "computer_use_extra"]); }); - it("synthesizes navigation tools when requested", () => { + it("synthesizes navigation tools by default", () => { const runtime = resolveCuaRuntimeSpec("openai:gpt-5.5"); const agent = new CuaAgent({ browser, client, - computerUseExtra: true, initialState: { model: "openai:gpt-5.5", }, @@ -157,6 +156,7 @@ describe("CuaAgent", () => { expect(agent.state.tools.map((tool) => tool.name)).toEqual([ ...runtime.toolExecutors.map((tool) => tool.definition.name), + "computer_use_extra", "playwright_execute", ]); }); @@ -175,7 +175,41 @@ describe("CuaAgent", () => { expect(agent.state.model.id).toBe(runtime.model.id); expect(agent.state.systemPrompt).toBe(runtime.defaultSystemPrompt); - expect(agent.state.tools).toHaveLength(runtime.toolExecutors.length); + expect(agent.state.tools).toHaveLength(runtime.toolExecutors.length + 1); + }); + + it("switches action planes through setMode", () => { + const agent = new CuaAgent({ + browser, + client, + initialState: { + model: "anthropic:claude-opus-4-5", + }, + }); + expect(agent.getMode()).toBe("computer"); + expect(agent.state.tools.map((tool) => tool.name)).toContain("click"); + + agent.setMode("browser"); + + expect(agent.getMode()).toBe("browser"); + const names = agent.state.tools.map((tool) => tool.name); + expect(names).toContain("snapshot"); + expect(names).not.toContain("move"); + expect(agent.state.systemPrompt).toBe(resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "browser" }).defaultSystemPrompt); + }); + + it("rejects setMode conflicting with a configured native tool", () => { + const agent = new CuaAgent({ + browser, + client, + nativeTool: { type: "browser_20260701" }, + initialState: { + model: "anthropic:claude-opus-4-5", + }, + }); + expect(agent.getMode()).toBe("browser"); + expect(() => agent.setMode("computer")).toThrow(/requires mode "browser"/); + expect(agent.getMode()).toBe("browser"); }); it("keeps extra tools and caller-owned system prompt when state.model changes", () => { @@ -193,7 +227,7 @@ describe("CuaAgent", () => { agent.state.model = "google:gemini-3-flash-preview"; const runtime = resolveCuaRuntimeSpec("google:gemini-3-flash-preview"); - expect(agent.state.tools.map((item) => item.name)).toEqual([...runtime.toolExecutors.map((item) => item.definition.name), "custom"]); + expect(agent.state.tools.map((item) => item.name)).toEqual([...runtime.toolExecutors.map((item) => item.definition.name), "computer_use_extra", "custom"]); expect(agent.state.systemPrompt).toBe("custom prompt"); }); @@ -265,7 +299,6 @@ describe("CuaAgent", () => { client: screenshotClient, streamFn, extraTools: [createCustomTool("custom_tool")], - computerUseExtra: true, initialState: { model: "yutori:n1.5-latest", }, @@ -308,7 +341,7 @@ describe("CuaAgent", () => { const update = await agent.prepareNextTurn?.(undefined); expect(update?.model?.id).toBe(runtime.model.id); - expect(update?.context?.tools).toHaveLength(runtime.toolExecutors.length); + expect(update?.context?.tools).toHaveLength(runtime.toolExecutors.length + 1); await expect(agent.prepareNextTurn?.(undefined)).resolves.toBeUndefined(); }); @@ -389,7 +422,195 @@ describe("CuaAgentHarness", () => { await harness.setModel("google:gemini-3-flash-preview"); expect(harness.getModel().id).toBe(runtime.model.id); - expect(harness.getTools()).toHaveLength(runtime.toolExecutors.length); + expect(harness.getTools()).toHaveLength(runtime.toolExecutors.length + 1); + }); + + it("switches action planes through setMode", async () => { + const harness = new CuaAgentHarness({ + ...(await createHarnessServices()), + browser, + client, + model: "anthropic:claude-opus-4-5", + }); + expect(harness.getMode()).toBe("computer"); + + await harness.setMode("hybrid"); + + expect(harness.getMode()).toBe("hybrid"); + const names = harness.getTools().map((tool) => tool.name); + expect(names).toContain("computer_click"); + expect(names).toContain("browser_snapshot"); + }); + + it("setMode keeps the requested activation state of surviving tools", async () => { + const harness = new CuaAgentHarness({ + ...(await createHarnessServices()), + browser, + client, + model: "anthropic:claude-opus-4-5", + extraTools: [createCustomTool()], + }); + const withoutCustom = harness + .getTools() + .map((tool) => tool.name) + .filter((name) => name !== "custom"); + await harness.setActiveTools(withoutCustom); + + await harness.setMode("browser"); + + const active = harness.getActiveTools().map((tool) => tool.name); + expect(active).toContain("snapshot"); + expect(active).not.toContain("custom"); + }); + + it("setModel after setMode keeps the mode's active tool subset", async () => { + const harness = new CuaAgentHarness({ + ...(await createHarnessServices()), + browser, + client, + model: "anthropic:claude-opus-4-5", + extraTools: [createCustomTool()], + }); + const withoutCustom = harness + .getTools() + .map((tool) => tool.name) + .filter((name) => name !== "custom"); + await harness.setActiveTools(withoutCustom); + await harness.setMode("browser"); + + await harness.setModel("anthropic:claude-opus-4-7"); + + const active = harness.getActiveTools().map((tool) => tool.name); + expect(active).toContain("snapshot"); + expect(active).not.toContain("custom"); + }); + + it("a failed mode switch keeps the pre-switch runtime and its live translator", async () => { + const { env, session } = await createHarnessServices(); + let failWrites = false; + const flakySession = new Proxy(session, { + get(target, prop, receiver) { + if (prop === "appendActiveToolsChange" && failWrites) { + return () => Promise.reject(new Error("session write failed")); + } + return Reflect.get(target, prop, receiver); + }, + }); + const harness = new CuaAgentHarness({ + env, + session: flakySession, + browser, + client, + model: "anthropic:claude-opus-4-5", + }); + const runtime = (harness as unknown as { runtime: { translator: unknown } }).runtime; + const translatorBefore = runtime.translator; + + failWrites = true; + await expect(harness.setMode("browser")).rejects.toThrow("session write failed"); + + // The exposed tools wrap this translator; rollback must not have + // disposed or replaced it. + expect(runtime.translator).toBe(translatorBefore); + expect(harness.getMode()).toBe("computer"); + + failWrites = false; + await harness.setMode("browser"); + expect(harness.getMode()).toBe("browser"); + expect(harness.getTools().map((tool) => tool.name)).toContain("snapshot"); + }); + + it("setMode keeps the translator and its CDP-backed state alive", async () => { + const harness = new CuaAgentHarness({ + ...(await createHarnessServices()), + browser, + client, + model: "anthropic:claude-opus-4-5", + }); + const runtime = (harness as unknown as { runtime: { translator: unknown } }).runtime; + const translator = runtime.translator; + + await harness.setMode("browser"); + await harness.setMode("hybrid"); + + expect(runtime.translator).toBe(translator); + }); + + it("setModel keeps the translator when the provider translator config is unchanged", async () => { + const harness = new CuaAgentHarness({ + ...(await createHarnessServices()), + browser, + client, + model: "anthropic:claude-opus-4-5", + }); + const runtime = (harness as unknown as { runtime: { translator: unknown } }).runtime; + const translator = runtime.translator; + + await harness.setModel("anthropic:claude-opus-4-7"); + expect(runtime.translator).toBe(translator); + + // Gemini uses a normalized coordinate system: the translator must be rebuilt. + await harness.setModel("google:gemini-3-flash-preview"); + expect(runtime.translator).not.toBe(translator); + }); + + it("an overlapping model switch disposes the superseded pending translator", async () => { + const { env, session } = await createHarnessServices(); + let gate: Promise | undefined; + const gatedSession = new Proxy(session, { + get(target, prop, receiver) { + const value = Reflect.get(target, prop, receiver); + if (prop === "appendActiveToolsChange" && gate) { + const pending = gate; + gate = undefined; + return async (...args: unknown[]) => { + await pending; + return (value as (...a: unknown[]) => Promise).apply(target, args); + }; + } + return typeof value === "function" ? (value as (...a: unknown[]) => unknown).bind(target) : value; + }, + }); + const harness = new CuaAgentHarness({ + env, + session: gatedSession, + browser, + client, + model: "anthropic:claude-opus-4-5", + }); + const runtime = (harness as unknown as { runtime: { translator: { dispose(): void } } }).runtime; + const original = vi.spyOn(runtime.translator, "dispose"); + + let release!: () => void; + gate = new Promise((resolve) => { + release = resolve; + }); + const first = harness.setModel("google:gemini-3-flash-preview"); + const superseded = vi.spyOn(runtime.translator, "dispose"); + + const second = harness.setModel("openai:gpt-5.5"); + release(); + await Promise.all([first, second]); + + // Neither the original nor the superseded pending translator may leak. + expect(original).toHaveBeenCalled(); + expect(superseded).toHaveBeenCalled(); + expect(harness.getModel().id).toBe("gpt-5.5"); + }); + + it("treats a repeated setMode as a no-op", async () => { + const harness = new CuaAgentHarness({ + ...(await createHarnessServices()), + browser, + client, + model: "anthropic:claude-opus-4-5", + }); + const before = harness.getTools(); + + await harness.setMode("computer"); + + // Same tool instances: the translator and its CDP state were not replaced. + expect(harness.getTools()[0]).toBe(before[0]); }); it("appends extraTools in harness construction", async () => { @@ -405,6 +626,7 @@ describe("CuaAgentHarness", () => { expect(harness.getTools().map((item) => item.name)).toEqual([ ...runtime.toolExecutors.map((item) => item.definition.name), + "computer_use_extra", "custom", ]); }); @@ -437,7 +659,7 @@ describe("CuaAgentHarness", () => { await harness.setModel("google:gemini-3-flash-preview"); expect(harness.getTools()).toHaveLength( - resolveCuaRuntimeSpec("google:gemini-3-flash-preview").toolExecutors.length, + resolveCuaRuntimeSpec("google:gemini-3-flash-preview").toolExecutors.length + 1, ); expect(harness.getActiveTools().map((tool) => tool.name)).toEqual(["click", "screenshot"]); diff --git a/packages/agent/test/cdp.test.ts b/packages/agent/test/cdp.test.ts new file mode 100644 index 00000000..6fe84fd8 --- /dev/null +++ b/packages/agent/test/cdp.test.ts @@ -0,0 +1,69 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { CdpConnection } from "../src/translator/cdp"; + +class FakeSocket { + static instances: FakeSocket[] = []; + readyState = 0; + sent: string[] = []; + private listeners = new Map void>>(); + + constructor(public url: string) { + FakeSocket.instances.push(this); + queueMicrotask(() => { + this.readyState = 1; + this.emit("open", {}); + }); + } + + addEventListener(type: string, listener: (event: unknown) => void): void { + const list = this.listeners.get(type) ?? []; + list.push(listener); + this.listeners.set(type, list); + } + + emit(type: string, event: unknown): void { + for (const listener of this.listeners.get(type) ?? []) listener(event); + } + + send(message: string): void { + // Mirrors the WebSocket spec: a non-OPEN socket discards silently. + if (this.readyState !== 1) return; + this.sent.push(message); + } + + close(): void { + this.readyState = 3; + this.emit("close", {}); + } +} + +afterEach(() => { + vi.unstubAllGlobals(); + FakeSocket.instances = []; +}); + +describe("CdpConnection send", () => { + it("rejects instead of hanging when the socket closed between connect and send", async () => { + vi.stubGlobal("WebSocket", Object.assign(FakeSocket, { OPEN: 1 })); + const cdp = new CdpConnection("wss://fake.test/cdp"); + + const first = cdp.send("Target.getTargets"); + await vi.waitFor(() => expect(FakeSocket.instances[0]!.sent).toHaveLength(1)); + const socket = FakeSocket.instances[0]!; + socket.emit("message", { data: JSON.stringify({ id: 1, result: { targetInfos: [] } }) }); + await first; + + // Close silently (no close event yet): send() would be discarded. + socket.readyState = 3; + await expect(cdp.send("Target.getTargets")).rejects.toThrow(/closed before/); + }); + + it("rejects in-flight commands when the connection closes", async () => { + vi.stubGlobal("WebSocket", Object.assign(FakeSocket, { OPEN: 1 })); + const cdp = new CdpConnection("wss://fake.test/cdp"); + const pending = cdp.send("Target.getTargets"); + await Promise.resolve(); + FakeSocket.instances[0]!.close(); + await expect(pending).rejects.toThrow(/closed/); + }); +}); diff --git a/packages/agent/test/tool-exhaustiveness.test.ts b/packages/agent/test/tool-exhaustiveness.test.ts index af8c1ba9..a0a67ed8 100644 --- a/packages/agent/test/tool-exhaustiveness.test.ts +++ b/packages/agent/test/tool-exhaustiveness.test.ts @@ -24,7 +24,7 @@ describe("Cua tool executor coverage", () => { it("instantiates one executor per provider execution adapter", () => { const toolExecutors = resolveCuaRuntimeSpec("openai:gpt-5.5").toolExecutors; const tools = createCuaComputerTools({ browser, client, toolExecutors }); - expect(tools.map((tool) => tool.name).sort()).toEqual(toolExecutors.map((tool) => tool.definition.name).sort()); + expect(tools.map((tool) => tool.name).sort()).toEqual([...toolExecutors.map((tool) => tool.definition.name), "computer_use_extra"].sort()); }); it("executes Yutori local canonical action tools", async () => { diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts new file mode 100644 index 00000000..b345b62f --- /dev/null +++ b/packages/agent/test/translator-browser.test.ts @@ -0,0 +1,830 @@ +import type Kernel from "@onkernel/sdk"; +import sharp from "sharp"; +import { describe, expect, it } from "vitest"; +import type { CuaBrowserAction } from "@onkernel/cua-ai"; +import { BrowserExecutor } from "../src/translator/browser"; +import type { CdpConnection } from "../src/translator/cdp"; +import { buildCuaComputerTools } from "../src/tools"; +import { InternalComputerTranslator, type KernelBrowser } from "../src/translator/translator"; +import type { BatchReadResult } from "../src/translator/types"; + +const browser = { session_id: "browser_123", cdp_ws_url: "wss://example.test/cdp" } as KernelBrowser; + +function createClient() { + const batches: unknown[] = []; + const client = { + browsers: { + computer: { + batch: async (_id: string, body: { actions: unknown[] }) => { + batches.push(body.actions); + }, + captureScreenshot: async () => new Response(new Uint8Array(await sharp({ create: { width: 100, height: 80, channels: 3, background: "#fff" } }).png().toBuffer())), + getMousePosition: async () => ({ x: 42, y: 24 }), + }, + }, + } as unknown as Kernel; + return { batches, client }; +} + +function createFakeBrowserExecutor() { + const executed: CuaBrowserAction[] = []; + const executor = { + execute: async (action: CuaBrowserAction): Promise => { + executed.push(action); + if (action.type === "browser_text") return [{ type: "browser_text", label: "text", text: "hello" }]; + return []; + }, + screenshot: async () => ({ data: Buffer.from("png"), mimeType: "image/png" }), + } as unknown as BrowserExecutor; + return { executed, executor }; +} + +describe("InternalComputerTranslator browser plane", () => { + it("dispatches browser actions to the browser executor, flushing pending OS input first", async () => { + const { batches, client } = createClient(); + const { executed, executor } = createFakeBrowserExecutor(); + const translator = new InternalComputerTranslator({ browser, client, createBrowserExecutor: () => executor }); + + const result = await translator.executeBatch([ + { type: "click", x: 1, y: 2 }, + { type: "browser_text" }, + { type: "browser_click", ref: "e3" }, + ]); + + expect(batches).toHaveLength(1); + expect(executed.map((action) => action.type)).toEqual(["browser_text", "browser_click"]); + expect(result.readResults).toEqual([{ type: "browser_text", label: "text", text: "hello" }]); + }); + + it("errors on browser actions when the browser has no cdp_ws_url", async () => { + const { client } = createClient(); + const translator = new InternalComputerTranslator({ browser: { session_id: "b" } as KernelBrowser, client }); + await expect(translator.executeBatch([{ type: "browser_text" }])).rejects.toThrow(/cdp_ws_url/); + }); +}); + +describe("InternalComputerTranslator computer additions", () => { + it("crops the OS screenshot for zoom, staying in the screenshot frame", async () => { + const { client } = createClient(); + const translator = new InternalComputerTranslator({ browser, client }); + const result = await translator.executeBatch([{ type: "zoom", region: [10, 10, 60, 40] }]); + const read = result.readResults[0]!; + if (read.type !== "screenshot") throw new Error("expected screenshot read result"); + const metadata = await sharp(read.data).metadata(); + expect(metadata.width).toBe(50); + expect(metadata.height).toBe(30); + }); + + it("passes num_clicks through and resolves missing click coordinates from the cursor", async () => { + const { batches, client } = createClient(); + const translator = new InternalComputerTranslator({ browser, client }); + await translator.executeBatch([ + { type: "click", x: 1, y: 2, num_clicks: 3 }, + { type: "click" }, + ]); + expect(batches.flat()).toEqual([ + { type: "click_mouse", click_mouse: { x: 1, y: 2, button: "left", num_clicks: 3 } }, + { type: "click_mouse", click_mouse: { x: 42, y: 24, button: "left" } }, + ]); + }); +}); + +interface FakeCdpEvent { + method: string; + params: Record; + sessionId?: string; +} + +interface SentCommand { + method: string; + params: Record; + sessionId?: string; +} + +function createFakeCdp(initialNodes: unknown[] = []) { + const sent: SentCommand[] = []; + const listeners: Array<(event: FakeCdpEvent) => void> = []; + let nodes = initialNodes as Array<{ backendDOMNodeId?: number }>; + let cursorBackendIds: number[] = []; + const sessionTrees = new Map>(); + const frameTrees = new Map>(); + const iframeFrameIds = new Map(); + const autoAttachFrames: Array<{ targetId: string; sessionId: string }> = []; + const failMethods = new Set(); + const emit = (event: FakeCdpEvent) => { + for (const listener of listeners) listener(event); + }; + const treeFor = (sessionId?: string, frameId?: unknown) => { + if (typeof frameId === "string" && frameTrees.has(frameId)) return frameTrees.get(frameId)!; + if (sessionId && sessionTrees.has(sessionId)) return sessionTrees.get(sessionId)!; + return nodes; + }; + const requireBackendId = (id: unknown, sessionId?: string) => { + const candidates = sessionId && sessionTrees.has(sessionId) ? [sessionTrees.get(sessionId)!] : [nodes, ...frameTrees.values()]; + if (!candidates.some((tree) => tree.some((node) => node.backendDOMNodeId === id))) throw new Error("No node with given id found"); + }; + const fake = { + onEvent: (listener: (event: FakeCdpEvent) => void) => { + listeners.push(listener); + }, + send: async (method: string, params: Record = {}, sessionId?: string) => { + sent.push({ method, params, sessionId }); + if (failMethods.has(method)) throw new Error(`${method} rejected`); + switch (method) { + case "Accessibility.getFullAXTree": + return { nodes: treeFor(sessionId, params.frameId) }; + case "Target.setAutoAttach": + for (const frame of autoAttachFrames.splice(0)) { + emit({ + method: "Target.attachedToTarget", + params: { sessionId: frame.sessionId, targetInfo: { targetId: frame.targetId, type: "iframe" } }, + sessionId, + }); + } + return {}; + case "DOM.scrollIntoViewIfNeeded": + requireBackendId(params.backendNodeId, sessionId); + return {}; + case "DOM.getBoxModel": + requireBackendId(params.backendNodeId, sessionId); + return { model: { content: [0, 0, 10, 0, 10, 10, 0, 10] } }; + case "DOM.resolveNode": + requireBackendId(params.backendNodeId, sessionId); + return { object: { objectId: "node-obj" } }; + case "Runtime.evaluate": + if (params.returnByValue === false) return { result: { objectId: "cursor-scan" } }; + return { result: { value: "hello" } }; + case "Runtime.getProperties": + return { + result: [ + ...cursorBackendIds.map((id, index) => ({ name: String(index), value: { objectId: `el-${id}` } })), + { name: "length", value: {} }, + ], + }; + case "DOM.describeNode": + if (typeof params.backendNodeId === "number") { + return { node: { backendNodeId: params.backendNodeId, frameId: iframeFrameIds.get(params.backendNodeId) } }; + } + return { node: { backendNodeId: Number(String(params.objectId).slice(3)) } }; + default: + return {}; + } + }, + pageTargets: async () => [{ targetId: "TARGET-1", type: "page", title: "Page", url: "https://a.test/" }], + attachToTarget: async () => "session-1", + createTarget: async () => "TARGET-2", + close: () => {}, + }; + const setNodes = (next: unknown[]) => { + nodes = next as Array<{ backendDOMNodeId?: number }>; + }; + const setCursorBackendIds = (ids: number[]) => { + cursorBackendIds = ids; + }; + const setSessionTree = (sessionId: string, tree: unknown[]) => { + sessionTrees.set(sessionId, tree as Array<{ backendDOMNodeId?: number }>); + }; + const setFrameTree = (frameId: string, tree: unknown[]) => { + frameTrees.set(frameId, tree as Array<{ backendDOMNodeId?: number }>); + }; + const setIframeFrame = (backendNodeId: number, frameId: string) => { + iframeFrameIds.set(backendNodeId, frameId); + }; + const addAutoAttachFrame = (frame: { targetId: string; sessionId: string }) => { + autoAttachFrames.push(frame); + }; + const failOn = (method: string) => { + failMethods.add(method); + }; + return { + sent, + emit, + setNodes, + setCursorBackendIds, + setSessionTree, + setFrameTree, + setIframeFrame, + addAutoAttachFrame, + failOn, + cdp: fake as unknown as CdpConnection, + }; +} + +interface AXNodeSpec { + nodeId: string; + role?: string; + name?: string; + value?: unknown; + properties?: Array<{ name: string; value?: unknown }>; + backendDOMNodeId?: number; + parentId?: string; + childIds?: string[]; +} + +function ax(spec: AXNodeSpec) { + return { + nodeId: spec.nodeId, + parentId: spec.parentId, + childIds: spec.childIds, + backendDOMNodeId: spec.backendDOMNodeId, + role: spec.role !== undefined ? { value: spec.role } : undefined, + name: spec.name !== undefined ? { value: spec.name } : undefined, + value: spec.value !== undefined ? { value: spec.value } : undefined, + properties: spec.properties?.map((property) => ({ name: property.name, value: { value: property.value } })), + }; +} + +function refsOf(executor: BrowserExecutor): Map { + return (executor as unknown as { refs: Map }).refs; +} + +async function snapshotText(executor: BrowserExecutor, action: Record = {}): Promise { + const results = await executor.execute({ type: "browser_snapshot", ...action } as CuaBrowserAction); + const read = results[0]!; + if (read.type !== "browser_text") throw new Error("expected browser_text read result"); + return read.text; +} + +const BUTTON_TREE = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 42, parentId: "1" }), +]; + +describe("BrowserExecutor ref lifecycle", () => { + it("prunes stale refs when a navigation bumps the generation", async () => { + const { cdp } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + expect(refsOf(executor).size).toBe(1); + await executor.execute({ type: "browser_navigate", url: "https://b.test" } as CuaBrowserAction); + expect(refsOf(executor).size).toBe(0); + }); + + it("does not let a rejected navigate suppress the next real navigation's invalidation", async () => { + const { cdp, emit, failOn } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + expect(refsOf(executor).size).toBe(1); + + failOn("Page.navigate"); + await expect(executor.execute({ type: "browser_navigate", url: "https://b.test" } as CuaBrowserAction)).rejects.toThrow(/rejected/); + + // A page-initiated navigation right after the failed command must still invalidate. + emit({ method: "Page.frameNavigated", params: { frame: { id: "F0" } }, sessionId: "session-1" }); + expect(refsOf(executor).size).toBe(0); + }); + + it("invalidates refs on main-frame frameNavigated but not on subframe navigation", async () => { + const { cdp, emit, sent } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toContain('button "Save" [e1]'); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "F2", parentId: "F1" } }, sessionId: "session-1" }); + await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + expect(sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent")).toBe(true); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "F1" } }, sessionId: "session-1" }); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + expect(refsOf(executor).size).toBe(0); + }); + + it("does not let a same-document navigation suppress the next real navigation's invalidation", async () => { + const { cdp, emit } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await executor.execute({ type: "browser_navigate", url: "https://a.test/#section" } as CuaBrowserAction); + emit({ method: "Page.navigatedWithinDocument", params: { frameId: "TARGET-1", url: "https://a.test/#section" }, sessionId: "session-1" }); + await snapshotText(executor); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "TARGET-1" } }, sessionId: "session-1" }); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + }); + + it("does not double-bump the generation for its own navigate", async () => { + const { cdp, emit } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await executor.execute({ type: "browser_navigate", url: "https://b.test" } as CuaBrowserAction); + const text = await snapshotText(executor); + expect(text).toContain('button "Save" [e1]'); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "F1" } }, sessionId: "session-1" }); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).resolves.toEqual([]); + }); +}); + +describe("BrowserExecutor snapshot rendering", () => { + it("indents by rendered depth so skipped wrappers neither indent nor consume the depth budget", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "generic", parentId: "1", childIds: ["3"] }), + ax({ nodeId: "3", role: "generic", parentId: "2", childIds: ["4"] }), + ax({ nodeId: "4", role: "button", name: "Save", backendDOMNodeId: 42, parentId: "3" }), + ]; + const { cdp } = createFakeCdp(tree); + const executor = new BrowserExecutor(cdp); + expect(await snapshotText(executor)).toBe('RootWebArea "Page"\n button "Save" [e1]'); + expect(await snapshotText(executor, { depth: 1 })).toBe('RootWebArea "Page"\n button "Save" [e2]'); + }); + + it("treats treeitem as interactive and the bogus textarea role as non-interactive", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3"] }), + ax({ nodeId: "2", role: "treeitem", name: "Reports", backendDOMNodeId: 10, parentId: "1" }), + ax({ nodeId: "3", role: "textarea", name: "Notes", backendDOMNodeId: 11, parentId: "1" }), + ]; + const { cdp } = createFakeCdp(tree); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toContain('treeitem "Reports" [e1]'); + expect(text).toContain('textarea "Notes"'); + expect(text).not.toContain('textarea "Notes" ['); + }); + + it("renders node states in a compact bracket after the ref", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3", "4", "5"] }), + ax({ + nodeId: "2", + role: "checkbox", + name: "Terms", + backendDOMNodeId: 10, + parentId: "1", + properties: [{ name: "checked", value: "true" }, { name: "required", value: true }], + }), + ax({ nodeId: "3", role: "checkbox", name: "Maybe", backendDOMNodeId: 11, parentId: "1", properties: [{ name: "checked", value: "mixed" }] }), + ax({ + nodeId: "4", + role: "button", + name: "Save", + backendDOMNodeId: 12, + parentId: "1", + properties: [{ name: "disabled", value: true }, { name: "expanded", value: false }], + }), + ax({ nodeId: "5", role: "textbox", name: "Email", backendDOMNodeId: 13, parentId: "1", value: "a@b.c" }), + ]; + const { cdp } = createFakeCdp(tree); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toContain('checkbox "Terms" [e1] [checked, required]'); + expect(text).toContain('checkbox "Maybe" [e2] [checked=mixed]'); + expect(text).toContain('button "Save" [e3] [disabled, expanded=false]'); + expect(text).toContain('textbox "Email" [e4] [value="a@b.c"]'); + }); + + it("renders false checked state, expanded, pressed, selected, and heading level", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3", "4", "5"] }), + ax({ nodeId: "2", role: "radio", name: "Solo", backendDOMNodeId: 10, parentId: "1", properties: [{ name: "checked", value: "false" }] }), + ax({ nodeId: "3", role: "button", name: "Bold", backendDOMNodeId: 11, parentId: "1", properties: [{ name: "pressed", value: "true" }] }), + ax({ + nodeId: "4", + role: "tab", + name: "Overview", + backendDOMNodeId: 12, + parentId: "1", + properties: [{ name: "selected", value: true }, { name: "expanded", value: true }], + }), + ax({ nodeId: "5", role: "heading", name: "Pricing", backendDOMNodeId: 13, parentId: "1", properties: [{ name: "level", value: 2 }] }), + ]; + const { cdp } = createFakeCdp(tree); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toContain('radio "Solo" [e1] [checked=false]'); + expect(text).toContain('button "Bold" [e2] [pressed]'); + expect(text).toContain('tab "Overview" [e3] [selected, expanded]'); + expect(text).toContain('heading "Pricing" [e4] [level=2]'); + }); + + it("merges consecutive StaticText siblings into one line", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3", "4"] }), + ax({ nodeId: "2", role: "StaticText", name: "Fast", parentId: "1" }), + ax({ nodeId: "3", role: "StaticText", name: "browsers", parentId: "1" }), + ax({ nodeId: "4", role: "button", name: "Go", backendDOMNodeId: 42, parentId: "1" }), + ]; + const { cdp } = createFakeCdp(tree); + const executor = new BrowserExecutor(cdp); + expect(await snapshotText(executor)).toBe('RootWebArea "Page"\n StaticText "Fast browsers"\n button "Go" [e1]'); + }); + + it("scopes a snapshot to a named content role's ref", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "4"] }), + ax({ nodeId: "2", role: "navigation", name: "Menu", backendDOMNodeId: 30, parentId: "1", childIds: ["3"] }), + ax({ nodeId: "3", role: "link", name: "Home", backendDOMNodeId: 31, parentId: "2" }), + ax({ nodeId: "4", role: "button", name: "Save", backendDOMNodeId: 32, parentId: "1" }), + ]; + const { cdp } = createFakeCdp(tree); + const executor = new BrowserExecutor(cdp); + const full = await snapshotText(executor); + expect(full).toContain('navigation "Menu" [e1]'); + const scoped = await snapshotText(executor, { ref: "e1" }); + expect(scoped).toContain('navigation "Menu"'); + expect(scoped).toContain('link "Home"'); + expect(scoped).not.toContain('button "Save"'); + }); + + it("skips StaticText duplicating the parent name and collapses wrappers without losing text", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "4", "7"] }), + ax({ nodeId: "2", role: "heading", name: "Title", parentId: "1", childIds: ["3"] }), + ax({ nodeId: "3", role: "StaticText", name: "Title", parentId: "2" }), + ax({ nodeId: "4", role: "link", name: "Docs", backendDOMNodeId: 20, parentId: "1", childIds: ["5"] }), + ax({ nodeId: "5", role: "generic", parentId: "4", childIds: ["6"] }), + ax({ nodeId: "6", role: "StaticText", name: "Docs", parentId: "5" }), + ax({ nodeId: "7", role: "StaticText", name: "Standalone", parentId: "1" }), + ]; + const { cdp } = createFakeCdp(tree); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toBe(['RootWebArea "Page"', ' heading "Title"', ' link "Docs" [e1]', ' StaticText "Standalone"'].join("\n")); + expect(text.split("\n")).toHaveLength(4); + }); +}); + +describe("BrowserExecutor stale-ref self-healing", () => { + it("heals a ref whose backend node moved when exactly one node matches the role/name triple", async () => { + const { cdp, sent, setNodes } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + setNodes([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), + ]); + await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + expect(sent.some((cmd) => cmd.method === "DOM.scrollIntoViewIfNeeded" && cmd.params.backendNodeId === 99)).toBe(true); + expect(sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent")).toBe(true); + }); + + it("heals a duplicate ref by position when the cohort size is unchanged", async () => { + const { cdp, sent, setNodes } = createFakeCdp([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3"] }), + ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 42, parentId: "1" }), + ax({ nodeId: "3", role: "button", name: "Save", backendDOMNodeId: 43, parentId: "1" }), + ]); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + setNodes([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3"] }), + ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), + ax({ nodeId: "3", role: "button", name: "Save", backendDOMNodeId: 100, parentId: "1" }), + ]); + await executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction); + expect(sent.some((cmd) => cmd.method === "DOM.scrollIntoViewIfNeeded" && cmd.params.backendNodeId === 100)).toBe(true); + }); + + it("heals a stale ref on browser_fill and retries the resolve", async () => { + const { cdp, sent, setNodes } = createFakeCdp([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "textbox", name: "Email", backendDOMNodeId: 42, parentId: "1" }), + ]); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + setNodes([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "textbox", name: "Email", backendDOMNodeId: 99, parentId: "1" }), + ]); + await executor.execute({ type: "browser_fill", ref: "e1", value: "a@b.c" } as CuaBrowserAction); + expect(sent.some((cmd) => cmd.method === "DOM.resolveNode" && cmd.params.backendNodeId === 99)).toBe(true); + expect(sent.some((cmd) => cmd.method === "Runtime.callFunctionOn")).toBe(true); + }); + + it("refuses to heal when multiple nodes match the stored role and name", async () => { + const { cdp, setNodes } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + setNodes([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3"] }), + ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), + ax({ nodeId: "3", role: "button", name: "Save", backendDOMNodeId: 100, parentId: "1" }), + ]); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + }); + + it("refuses to heal a duplicate ref when the cohort shrank", async () => { + const { cdp, setNodes } = createFakeCdp([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3"] }), + ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 42, parentId: "1" }), + ax({ nodeId: "3", role: "button", name: "Save", backendDOMNodeId: 43, parentId: "1" }), + ]); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + setNodes([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), + ]); + await expect(executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction)).rejects.toThrow(/stale/); + }); +}); + +describe("BrowserExecutor fill", () => { + const FILL_TREE = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "textbox", name: "Email", backendDOMNodeId: 42, parentId: "1" }), + ]; + + it("focuses the element it fills before dispatching input events", async () => { + const { cdp, sent } = createFakeCdp(FILL_TREE); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + await executor.execute({ type: "browser_fill", ref: "e1", value: "a@b.c" } as CuaBrowserAction); + const call = sent.find((cmd) => cmd.method === "Runtime.callFunctionOn"); + const declaration = call?.params.functionDeclaration as string; + const fillFn = new Function(`return (${declaration})`)() as (value: unknown) => void; + const events: string[] = []; + const el = { + tagName: "INPUT", + type: "text", + value: "", + isContentEditable: false, + focus: () => events.push("focus"), + dispatchEvent: (event: Event) => events.push(event.type), + }; + fillFn.call(el, "hello"); + expect(el.value).toBe("hello"); + expect(events).toEqual(["focus", "input", "change"]); + }); + + it.each([ + ["Error: element is not a form control"], + ["TypeError: element is not a form control"], + ])("trims fill exception %j to a single line without the prefix or stack", async (firstLine) => { + const { cdp } = createFakeCdp(FILL_TREE); + const inner = cdp as unknown as { send: (method: string, params?: Record, sessionId?: string) => Promise }; + const wrapped = { + ...cdp, + send: async (method: string, params?: Record, sessionId?: string) => { + if (method === "Runtime.callFunctionOn") { + return { + exceptionDetails: { + exception: { description: `${firstLine}\n at HTMLAnchorElement. (:20:9)` }, + }, + }; + } + return inner.send(method, params, sessionId); + }, + } as unknown as CdpConnection; + const executor = new BrowserExecutor(wrapped); + await snapshotText(executor); + await expect(executor.execute({ type: "browser_fill", ref: "e1", value: "x" } as CuaBrowserAction)).rejects.toThrow( + /^browser_fill failed: element is not a form control$/, + ); + }); +}); + +describe("BrowserExecutor cursor-pointer hints", () => { + const POINTER_TREE = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "generic", name: "Buy now", backendDOMNodeId: 77, parentId: "1" }), + ]; + + it("marks cursor:pointer elements as clickable hints in every snapshot", async () => { + const { cdp, sent, setCursorBackendIds } = createFakeCdp(POINTER_TREE); + setCursorBackendIds([77]); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toContain('generic "Buy now" [e1] [cursor:pointer]'); + expect(sent.some((cmd) => cmd.method === "DOM.describeNode")).toBe(true); + expect(sent.some((cmd) => cmd.method === "Runtime.releaseObjectGroup")).toBe(true); + }); +}); + +describe("BrowserExecutor dialog guard", () => { + it("dismisses confirm/prompt dialogs and surfaces the message on the next action", async () => { + const { cdp, emit, sent } = createFakeCdp(); + const executor = new BrowserExecutor(cdp); + await executor.execute({ type: "browser_text" } as CuaBrowserAction); + + emit({ method: "Page.javascriptDialogOpening", params: { type: "confirm", message: "Delete item?" }, sessionId: "session-1" }); + const handled = sent.find((cmd) => cmd.method === "Page.handleJavaScriptDialog"); + expect(handled).toEqual({ method: "Page.handleJavaScriptDialog", params: { accept: false }, sessionId: "session-1" }); + + const results = await executor.execute({ type: "browser_text" } as CuaBrowserAction); + expect(results).toEqual([ + { type: "browser_text", label: "text", text: "hello" }, + { type: "browser_text", label: "dialog", text: 'Dismissed a JavaScript confirm dialog (answered No/cancel): "Delete item?"' }, + ]); + }); + + it("accepts alert and beforeunload dialogs so navigation can proceed", async () => { + const { cdp, emit, sent } = createFakeCdp(); + const executor = new BrowserExecutor(cdp); + await executor.execute({ type: "browser_text" } as CuaBrowserAction); + + emit({ method: "Page.javascriptDialogOpening", params: { type: "beforeunload", message: "" }, sessionId: "session-1" }); + emit({ method: "Page.javascriptDialogOpening", params: { type: "alert", message: "Saved!" }, sessionId: "session-1" }); + const handled = sent.filter((cmd) => cmd.method === "Page.handleJavaScriptDialog"); + expect(handled.map((cmd) => cmd.params)).toEqual([{ accept: true }, { accept: true }]); + + const results = await executor.execute({ type: "browser_text" } as CuaBrowserAction); + expect(results[1]).toEqual({ + type: "browser_text", + label: "dialog", + text: 'Accepted a beforeunload dialog so navigation could proceed: ""\nAcknowledged a JavaScript alert dialog: "Saved!"', + }); + }); +}); + +describe("BrowserExecutor snapshot diffing", () => { + const UNCHANGED = "Page unchanged since the last snapshot; previous element refs are still valid."; + + it("returns a short unchanged notice for an identical re-snapshot and the full tree after a change", async () => { + const { cdp, setNodes } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + expect(await snapshotText(executor)).toContain('button "Save" [e1]'); + expect(await snapshotText(executor)).toBe(UNCHANGED); + await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + setNodes([ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "button", name: "Delete", backendDOMNodeId: 43, parentId: "1" }), + ]); + expect(await snapshotText(executor)).toContain('button "Delete" [e2]'); + }); + + it("returns the full tree when the params differ from the previous snapshot", async () => { + const { cdp } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + expect(await snapshotText(executor, { filter: "interactive" })).toContain('button "Save" [e2]'); + }); +}); + +describe("BrowserExecutor iframe stitching", () => { + it("stitches a same-process iframe subtree indented under its iframe node", async () => { + const tree = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), + ax({ nodeId: "2", role: "Iframe", backendDOMNodeId: 50, parentId: "1" }), + ]; + const { cdp, emit, setFrameTree, setIframeFrame } = createFakeCdp(tree); + setIframeFrame(50, "FRAME-SP"); + setFrameTree("FRAME-SP", [ + ax({ nodeId: "f1", role: "RootWebArea", name: "Embed", childIds: ["f2"] }), + ax({ nodeId: "f2", role: "button", name: "Inside", backendDOMNodeId: 60, parentId: "f1" }), + ]); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toBe(['RootWebArea "Page"', " Iframe [e1]", ' RootWebArea "Embed"', ' button "Inside" [e2]'].join("\n")); + await executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-SP", parentId: "F0" } }, sessionId: "session-1" }); + await expect(executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction)).rejects.toThrow(/stale/); + }); + + const OOPIF_PAGE = [ + ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2", "3"] }), + ax({ nodeId: "2", role: "button", name: "Top", backendDOMNodeId: 40, parentId: "1" }), + ax({ nodeId: "3", role: "Iframe", backendDOMNodeId: 50, parentId: "1" }), + ]; + const OOPIF_CHILD = [ + ax({ nodeId: "f1", role: "RootWebArea", name: "Widget", childIds: ["f2"] }), + ax({ nodeId: "f2", role: "button", name: "Pay", backendDOMNodeId: 70, parentId: "f1" }), + ]; + const setupOopif = () => { + const fake = createFakeCdp(OOPIF_PAGE); + fake.setIframeFrame(50, "FRAME-OOP"); + fake.addAutoAttachFrame({ targetId: "FRAME-OOP", sessionId: "session-oop" }); + fake.setSessionTree("session-oop", OOPIF_CHILD); + return fake; + }; + + it("resolves an OOPIF ref's node through the child session but dispatches input on the page session", async () => { + const { cdp, sent } = setupOopif(); + const executor = new BrowserExecutor(cdp); + const text = await snapshotText(executor); + expect(text).toContain('button "Top" [e1]'); + expect(text).toContain(' button "Pay" [e3]'); + + await executor.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction); + const scrolled = sent.find((cmd) => cmd.method === "DOM.scrollIntoViewIfNeeded" && cmd.params.backendNodeId === 70); + expect(scrolled?.sessionId).toBe("session-oop"); + const pressed = sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); + expect(pressed?.sessionId).toBe("session-1"); + }); + + it("invalidates a frame target's refs when a subframe inside it navigates", async () => { + const { cdp, emit } = setupOopif(); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-INNER", parentId: "FRAME-OOP" } }, sessionId: "session-oop" }); + await expect(executor.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); + }); + + it("finds elements inside stitched iframes", async () => { + const { cdp, sent } = setupOopif(); + const executor = new BrowserExecutor(cdp); + + const results = await executor.execute({ type: "browser_find", query: "pay button" } as CuaBrowserAction); + const text = (results[0] as { text: string }).text; + expect(text).toContain('button "Pay" [e'); + + const ref = /\[(e\d+)\]/.exec(text)![1]!; + await executor.execute({ type: "browser_click", ref } as CuaBrowserAction); + const resolved = sent.find((cmd) => cmd.method === "DOM.getBoxModel" && cmd.params.backendNodeId === 70); + expect(resolved?.sessionId).toBe("session-oop"); + }); + + it("invalidates only the child frame's refs when the child frame navigates", async () => { + const { cdp, emit } = setupOopif(); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-OOP" } }, sessionId: "session-oop" }); + await expect(executor.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + + const text = await snapshotText(executor); + expect(text).toContain('button "Pay" [e'); + }); +}); + +describe("navigation tool grounding frame", () => { + const navTool = (mode: "computer" | "browser" | "hybrid") => { + const { client, batches } = createClient(); + const { executor, executed } = createFakeBrowserExecutor(); + const translator = new InternalComputerTranslator({ browser, client, mode, createBrowserExecutor: () => executor }); + const tool = buildCuaComputerTools({ toolExecutors: [], mode }, translator).find((tool) => tool.name === "computer_use_extra")!; + return { tool, batches, executed }; + }; + + it("captures the viewport in browser mode and the OS display otherwise", async () => { + const viewportData = Buffer.from("png").toString("base64"); + + const browserResult = await navTool("browser").tool.execute("call_1", { action: "back" }); + const viewportImage = browserResult.content.find((block) => block.type === "image"); + expect(viewportImage).toMatchObject({ type: "image", data: viewportData }); + + const computerResult = await navTool("computer").tool.execute("call_2", { action: "back" }); + const osImage = computerResult.content.find((block) => block.type === "image"); + expect(osImage?.type).toBe("image"); + expect((osImage as { data: string }).data).not.toBe(viewportData); + }); + + it("navigates on the browser plane in browser and hybrid modes and the OS plane in computer mode", async () => { + const inBrowser = navTool("browser"); + await inBrowser.tool.execute("call_1", { action: "goto", url: "https://example.com" }); + await inBrowser.tool.execute("call_2", { action: "back" }); + expect(inBrowser.executed).toEqual([ + { type: "browser_navigate", url: "https://example.com" }, + { type: "browser_navigate", url: "back" }, + ]); + expect(inBrowser.batches).toEqual([]); + + const inHybrid = navTool("hybrid"); + await inHybrid.tool.execute("call_3", { action: "forward" }); + expect(inHybrid.executed).toEqual([{ type: "browser_navigate", url: "forward" }]); + expect(inHybrid.batches).toEqual([]); + + const inComputer = navTool("computer"); + await inComputer.tool.execute("call_4", { action: "back" }); + expect(inComputer.executed).toEqual([]); + expect(inComputer.batches).toHaveLength(1); + }); +}); + +describe("BrowserExecutor multi-click", () => { + it("dispatches one press/release cycle per click with incrementing clickCount", async () => { + const { cdp, sent } = createFakeCdp(BUTTON_TREE); + const executor = new BrowserExecutor(cdp); + await snapshotText(executor); + await executor.execute({ type: "browser_click", ref: "e1", num_clicks: 2 } as CuaBrowserAction); + + const mouse = sent.filter((cmd) => cmd.method === "Input.dispatchMouseEvent").map((cmd) => cmd.params); + expect(mouse.map((params) => [params.type, params.clickCount])).toEqual([ + ["mouseMoved", undefined], + ["mousePressed", 1], + ["mouseReleased", 1], + ["mousePressed", 2], + ["mouseReleased", 2], + ]); + }); +}); + +describe("BrowserExecutor ref state export/import", () => { + it("resolves refs imported from a previous executor against the same browser", async () => { + const first = new BrowserExecutor(createFakeCdp(BUTTON_TREE).cdp); + await snapshotText(first); + const state = first.exportRefState(); + + const { cdp, sent } = createFakeCdp(BUTTON_TREE); + const second = new BrowserExecutor(cdp); + second.importRefState(state); + await second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + const pressed = sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); + expect(pressed).toBeDefined(); + }); + + it("keeps minting unique refs after import and invalidates imported refs on navigation", async () => { + const first = new BrowserExecutor(createFakeCdp(BUTTON_TREE).cdp); + await snapshotText(first); + const state = first.exportRefState(); + + const { cdp, emit } = createFakeCdp(BUTTON_TREE); + const second = new BrowserExecutor(cdp); + second.importRefState(state); + expect(await snapshotText(second)).toContain('button "Save" [e2]'); + + emit({ method: "Page.frameNavigated", params: { frame: { id: "F0" } }, sessionId: "session-1" }); + await expect(second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + }); +}); diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 7f0dd654..591be5d5 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,33 @@ # Changelog +## 0.5.0 - 2026-07-09 + +Introduces action planes (modes) and Anthropic native computer-use tools. + +- New `mode` option (`"computer"` | `"browser"` | `"hybrid"`, exported as + `CuaMode`) on `resolveCuaRuntimeSpec`, `computerTools`, and the executor + builders selects which action plane(s) the model sees. `computer` is the + pre-modes default and stays byte-compatible. `browser` exposes the new + browser-plane canonical actions; `hybrid` exposes both planes deduplicated + to one tool per capability, with browser actions restricted to element refs + so the OS screenshot is the single coordinate frame. +- New browser-plane canonical actions (`CUA_BROWSER_ACTION_TYPES`): + `browser_snapshot`, `browser_find`, `browser_text`, `browser_click`, + `browser_fill`, `browser_scroll_to`, `browser_navigate`, + `browser_list_tabs`, `browser_new_tab`, `browser_screenshot`, + `browser_evaluate`, and friends, with per-mode tool naming + (`cuaToolNameForAction`), descriptions, schemas, and system prompts. +- New `nativeTool` option drives Anthropic models through their native + computer-use declarations: `computer_20260701` (computer mode, with + `enable_zoom`) and `browser_20260701` (browser mode) behind + `anthropic-beta: computer-use-2026-07-01`. +- JavaScript execution is on by default: `browser_evaluate` is part of the + default browser/hybrid action sets, and native `browser_20260701` + declarations default `enable_javascript_exec` to true (an explicit value on + the spec wins). Opt out by passing an explicit `actions` list or native + tool spec. +- New `zoom` computer action (cropped display inspection). + ## 0.4.0 - 2026-07-07 Breaking: adopts pi-ai 0.80's instance-based `Models` API and drops the diff --git a/packages/ai/package.json b/packages/ai/package.json index 2e5bc4d6..09ed707d 100644 --- a/packages/ai/package.json +++ b/packages/ai/package.json @@ -1,6 +1,6 @@ { "name": "@onkernel/cua-ai", - "version": "0.4.0", + "version": "0.5.0", "description": "Kernel-curated computer-use model access built on pi-ai", "license": "MIT", "type": "module", diff --git a/packages/ai/src/actions/browser.ts b/packages/ai/src/actions/browser.ts new file mode 100644 index 00000000..55589ac6 --- /dev/null +++ b/packages/ai/src/actions/browser.ts @@ -0,0 +1,326 @@ +import { Type, type TSchema } from "@earendil-works/pi-ai"; + +/** + * Browser-plane canonical actions. + * + * These execute over CDP against the browser itself: accessibility-tree + * reads with element references, element-targeted interaction, navigation, + * tabs, and viewport screenshots. Where a browser action takes coordinates + * (`browser_click`, `browser_hover`, `browser_drag`, `browser_scroll`), they are pixels + * in the browser viewport — a different frame from the computer-plane actions in + * `./computer`. Modes that expose both planes (hybrid) therefore restrict browser + * actions to element references so exactly one coordinate frame is live. + * + * Element references (`ref`) are snapshot-scoped opaque ids (`e12`) minted + * by `browser_snapshot` / `browser_find`; a stale ref is an error instructing the + * model to re-snapshot. + */ +export const CUA_BROWSER_ACTION_TYPES = [ + "browser_snapshot", + "browser_text", + "browser_find", + "browser_click", + "browser_hover", + "browser_drag", + "browser_fill", + "browser_scroll_to", + "browser_scroll", + "browser_type", + "browser_key", + "browser_navigate", + "browser_list_tabs", + "browser_new_tab", + "browser_screenshot", + "browser_evaluate", +] as const; + +export type CuaBrowserActionType = (typeof CUA_BROWSER_ACTION_TYPES)[number]; + +export interface CuaActionBrowserSnapshot { + type: "browser_snapshot"; + filter?: "all" | "interactive"; + ref?: string; + depth?: number; + tab_id?: string; +} + +export interface CuaActionBrowserText { + type: "browser_text"; + tab_id?: string; +} + +export interface CuaActionBrowserFind { + type: "browser_find"; + query: string; + tab_id?: string; +} + +export interface CuaActionBrowserClick { + type: "browser_click"; + ref?: string; + x?: number; + y?: number; + button?: "left" | "right" | "middle"; + num_clicks?: number; + modifiers?: string[]; + tab_id?: string; +} + +export interface CuaActionBrowserHover { + type: "browser_hover"; + ref?: string; + x?: number; + y?: number; + tab_id?: string; +} + +export interface CuaActionBrowserDrag { + type: "browser_drag"; + from: { x: number; y: number }; + to: { x: number; y: number }; + tab_id?: string; +} + +export interface CuaActionBrowserFill { + type: "browser_fill"; + ref: string; + value: string | number | boolean; + tab_id?: string; +} + +export interface CuaActionBrowserScrollTo { + type: "browser_scroll_to"; + ref: string; + tab_id?: string; +} + +export interface CuaActionBrowserScroll { + type: "browser_scroll"; + x: number; + y: number; + direction: "up" | "down" | "left" | "right"; + amount?: number; + tab_id?: string; +} + +export interface CuaActionBrowserType { + type: "browser_type"; + text: string; + tab_id?: string; +} + +export interface CuaActionBrowserKey { + type: "browser_key"; + text: string; + repeat?: number; + tab_id?: string; +} + +export interface CuaActionBrowserNavigate { + type: "browser_navigate"; + /** A URL, or the sentinels "back" / "forward" for history navigation. */ + url: string; + tab_id?: string; +} + +export interface CuaActionBrowserListTabs { + type: "browser_list_tabs"; +} + +export interface CuaActionBrowserNewTab { + type: "browser_new_tab"; +} + +export interface CuaActionBrowserScreenshot { + type: "browser_screenshot"; + /** Optional crop region, [x0, y0, x1, y1] in viewport pixels. */ + region?: [number, number, number, number]; + tab_id?: string; +} + +export interface CuaActionBrowserEvaluate { + type: "browser_evaluate"; + code: string; + tab_id?: string; +} + +export type CuaBrowserAction = + | CuaActionBrowserSnapshot + | CuaActionBrowserText + | CuaActionBrowserFind + | CuaActionBrowserClick + | CuaActionBrowserHover + | CuaActionBrowserDrag + | CuaActionBrowserFill + | CuaActionBrowserScrollTo + | CuaActionBrowserScroll + | CuaActionBrowserType + | CuaActionBrowserKey + | CuaActionBrowserNavigate + | CuaActionBrowserListTabs + | CuaActionBrowserNewTab + | CuaActionBrowserScreenshot + | CuaActionBrowserEvaluate; + +/** Options for building browser action schemas. */ +export interface CuaBrowserSchemaOptions { + /** + * Whether coordinate targeting is allowed on `browser_click` / `browser_hover` + * and whether `browser_drag` / `browser_scroll` are expressible at all. Browser + * mode allows viewport coordinates (they are the only frame); hybrid mode + * must disallow them so the OS screenshot frame stays the single live + * coordinate frame. + */ + coordinates: boolean; +} + +const TabId = () => Type.Optional(Type.String({ description: "Tab to act on. Defaults to the active tab." })); + +const RefProperty = () => Type.String({ description: "Element reference from browser_snapshot or browser_find, e.g. \"e12\"." }); + +export function createCuaBrowserActionSchemaByType(options: CuaBrowserSchemaOptions): Record { + const clickTarget: Record = options.coordinates + ? { + ref: Type.Optional(RefProperty()), + x: Type.Optional(Type.Number({ description: "Viewport x in pixels. Prefer ref targeting when available." })), + y: Type.Optional(Type.Number({ description: "Viewport y in pixels. Prefer ref targeting when available." })), + } + : { ref: RefProperty() }; + + return { + browser_snapshot: Type.Object( + { + type: Type.Literal("browser_snapshot"), + filter: Type.Optional(Type.Union([Type.Literal("all"), Type.Literal("interactive")])), + ref: Type.Optional(Type.String({ description: "Restrict the snapshot to the subtree rooted at this element reference." })), + depth: Type.Optional(Type.Number({ description: "Maximum tree depth (default 15)." })), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_text: Type.Object( + { + type: Type.Literal("browser_text"), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_find: Type.Object( + { + type: Type.Literal("browser_find"), + query: Type.String({ description: "Natural-language element description, e.g. \"the cookie consent accept button\"." }), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_click: Type.Object( + { + type: Type.Literal("browser_click"), + ...clickTarget, + button: Type.Optional(Type.Union([Type.Literal("left"), Type.Literal("right"), Type.Literal("middle")])), + num_clicks: Type.Optional(Type.Number()), + modifiers: Type.Optional(Type.Array(Type.String())), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_hover: Type.Object( + { + type: Type.Literal("browser_hover"), + ...clickTarget, + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_drag: Type.Object( + { + type: Type.Literal("browser_drag"), + from: Type.Object({ x: Type.Number(), y: Type.Number() }, { additionalProperties: false }), + to: Type.Object({ x: Type.Number(), y: Type.Number() }, { additionalProperties: false }), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_fill: Type.Object( + { + type: Type.Literal("browser_fill"), + ref: RefProperty(), + value: Type.Union([Type.String(), Type.Number(), Type.Boolean()], { + description: "Value to set. Use a boolean for checkboxes, an option value or label for selects.", + }), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_scroll_to: Type.Object( + { + type: Type.Literal("browser_scroll_to"), + ref: RefProperty(), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_scroll: Type.Object( + { + type: Type.Literal("browser_scroll"), + x: Type.Number({ description: "Viewport x anchor in pixels." }), + y: Type.Number({ description: "Viewport y anchor in pixels." }), + direction: Type.Union([Type.Literal("up"), Type.Literal("down"), Type.Literal("left"), Type.Literal("right")]), + amount: Type.Optional(Type.Number({ description: "Scroll-wheel notches (default 3)." })), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_type: Type.Object( + { + type: Type.Literal("browser_type"), + text: Type.String(), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_key: Type.Object( + { + type: Type.Literal("browser_key"), + text: Type.String({ description: "Key or chord, e.g. \"Return\", \"ctrl+a\"." }), + repeat: Type.Optional(Type.Number()), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_navigate: Type.Object( + { + type: Type.Literal("browser_navigate"), + url: Type.String({ description: "URL to navigate to, or \"back\" / \"forward\" for history navigation." }), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_list_tabs: Type.Object({ type: Type.Literal("browser_list_tabs") }, { additionalProperties: false }), + browser_new_tab: Type.Object({ type: Type.Literal("browser_new_tab") }, { additionalProperties: false }), + browser_screenshot: Type.Object( + { + type: Type.Literal("browser_screenshot"), + // Not Type.Tuple: tuples emit draft-07 `items: [...]`, which Anthropic's + // draft 2020-12 schema validation rejects. + region: Type.Optional( + Type.Array(Type.Number(), { + minItems: 4, + maxItems: 4, + description: "Optional crop region, [x0, y0, x1, y1] in viewport pixels.", + }), + ), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + browser_evaluate: Type.Object( + { + type: Type.Literal("browser_evaluate"), + code: Type.String({ description: "JavaScript to evaluate in the page context. The value of the last expression is returned." }), + tab_id: TabId(), + }, + { additionalProperties: false }, + ), + }; +} diff --git a/packages/ai/src/actions/computer.ts b/packages/ai/src/actions/computer.ts new file mode 100644 index 00000000..181269b0 --- /dev/null +++ b/packages/ai/src/actions/computer.ts @@ -0,0 +1,301 @@ +import { Type, type TSchema } from "@earendil-works/pi-ai"; + +/** + * Computer-plane canonical actions. + * + * These execute as real OS-level input against the Kernel browser VM (mouse, + * keyboard, display capture) — never CDP. All coordinates are pixels in the + * OS screenshot frame. The browser-plane vocabulary lives in `./browser` and is + * executed over CDP; the two planes never share a coordinate frame. + */ +export const CUA_COMPUTER_ACTION_TYPES = [ + "click", + "double_click", + "mouse_down", + "mouse_up", + "type", + "keypress", + "scroll", + "move", + "drag", + "wait", + "screenshot", + "zoom", + "goto", + "back", + "forward", + "url", + "cursor_position", +] as const; + +export type CuaComputerActionType = (typeof CUA_COMPUTER_ACTION_TYPES)[number]; + +/** + * The default computer-mode toolset. This is the pre-modes canonical action list: + * every computer action except `zoom`, which is only exposed by default in hybrid + * mode and by Anthropic's native computer tool (`enable_zoom`). + */ +export const CUA_DEFAULT_COMPUTER_ACTION_TYPES = CUA_COMPUTER_ACTION_TYPES.filter( + (action): action is Exclude => action !== "zoom", +); + +/** + * Mouse buttons accepted by click, mouse_down, and mouse_up actions. The + * executor coerces anything outside this set to "left". + */ +export type CuaMouseButton = "left" | "right" | "middle" | "back" | "forward"; + +/** + * Mouse buttons accepted by drag actions. The executor coerces anything + * outside this set to "left". + */ +export type CuaDragMouseButton = "left" | "right" | "middle"; + +export interface CuaActionClick { + type: "click"; + /** OS screenshot pixels. Omitted (native mappings only) means the current cursor position. */ + x?: number; + y?: number; + button?: CuaMouseButton; + hold_keys?: string[]; + num_clicks?: number; +} + +export interface CuaActionDoubleClick { + type: "double_click"; + x: number; + y: number; + hold_keys?: string[]; +} + +export interface CuaActionMouseDown { + type: "mouse_down"; + x?: number; + y?: number; + button?: CuaMouseButton; + hold_keys?: string[]; +} + +export interface CuaActionMouseUp { + type: "mouse_up"; + x?: number; + y?: number; + button?: CuaMouseButton; + hold_keys?: string[]; +} + +export interface CuaActionTypeText { + type: "type"; + text: string; +} + +export interface CuaActionKeypress { + type: "keypress"; + keys: string[]; + duration?: number; +} + +export interface CuaActionScroll { + type: "scroll"; + x?: number; + y?: number; + scroll_x?: number; + scroll_y?: number; + hold_keys?: string[]; +} + +export interface CuaActionMove { + type: "move"; + x: number; + y: number; +} + +export interface CuaActionDrag { + type: "drag"; + path: Array<{ x: number; y: number }>; + button?: CuaDragMouseButton; + hold_keys?: string[]; +} + +export interface CuaActionWait { + type: "wait"; + ms?: number; +} + +export interface CuaActionScreenshot { + type: "screenshot"; +} + +/** Crop of the most recent OS screenshot; region is [x0, y0, x1, y1] in OS screenshot pixels. */ +export interface CuaActionZoom { + type: "zoom"; + region: [number, number, number, number]; +} + +export interface CuaActionGoto { + type: "goto"; + url: string; +} + +export interface CuaActionBack { + type: "back"; +} + +export interface CuaActionForward { + type: "forward"; +} + +export interface CuaActionUrl { + type: "url"; +} + +export interface CuaActionCursorPosition { + type: "cursor_position"; +} + +export type CuaComputerAction = + | CuaActionClick + | CuaActionDoubleClick + | CuaActionMouseDown + | CuaActionMouseUp + | CuaActionTypeText + | CuaActionKeypress + | CuaActionScroll + | CuaActionMove + | CuaActionDrag + | CuaActionWait + | CuaActionScreenshot + | CuaActionZoom + | CuaActionGoto + | CuaActionBack + | CuaActionForward + | CuaActionUrl + | CuaActionCursorPosition; + +const PointSchema = Type.Object( + { + x: Type.Number(), + y: Type.Number(), + }, + { additionalProperties: false }, +); + +export const CUA_COMPUTER_ACTION_SCHEMA_BY_TYPE = { + click: Type.Object( + { + type: Type.Literal("click"), + x: Type.Number(), + y: Type.Number(), + button: Type.Optional(Type.String()), + hold_keys: Type.Optional(Type.Array(Type.String())), + num_clicks: Type.Optional(Type.Number()), + }, + { additionalProperties: false }, + ), + double_click: Type.Object( + { + type: Type.Literal("double_click"), + x: Type.Number(), + y: Type.Number(), + hold_keys: Type.Optional(Type.Array(Type.String())), + }, + { additionalProperties: false }, + ), + mouse_down: Type.Object( + { + type: Type.Literal("mouse_down"), + x: Type.Optional(Type.Number()), + y: Type.Optional(Type.Number()), + button: Type.Optional(Type.String()), + hold_keys: Type.Optional(Type.Array(Type.String())), + }, + { additionalProperties: false }, + ), + mouse_up: Type.Object( + { + type: Type.Literal("mouse_up"), + x: Type.Optional(Type.Number()), + y: Type.Optional(Type.Number()), + button: Type.Optional(Type.String()), + hold_keys: Type.Optional(Type.Array(Type.String())), + }, + { additionalProperties: false }, + ), + type: Type.Object( + { + type: Type.Literal("type"), + text: Type.String(), + }, + { additionalProperties: false }, + ), + keypress: Type.Object( + { + type: Type.Literal("keypress"), + keys: Type.Array(Type.String()), + duration: Type.Optional(Type.Number()), + }, + { additionalProperties: false }, + ), + scroll: Type.Object( + { + type: Type.Literal("scroll"), + x: Type.Optional(Type.Number()), + y: Type.Optional(Type.Number()), + scroll_x: Type.Optional(Type.Number()), + scroll_y: Type.Optional(Type.Number()), + hold_keys: Type.Optional(Type.Array(Type.String())), + }, + { additionalProperties: false }, + ), + move: Type.Object( + { + type: Type.Literal("move"), + x: Type.Number(), + y: Type.Number(), + }, + { additionalProperties: false }, + ), + drag: Type.Object( + { + type: Type.Literal("drag"), + path: Type.Array(PointSchema, { minItems: 2 }), + button: Type.Optional(Type.String()), + hold_keys: Type.Optional(Type.Array(Type.String())), + }, + { additionalProperties: false }, + ), + wait: Type.Object( + { + type: Type.Literal("wait"), + ms: Type.Optional(Type.Number()), + }, + { additionalProperties: false }, + ), + screenshot: Type.Object({ type: Type.Literal("screenshot") }, { additionalProperties: false }), + zoom: Type.Object( + { + type: Type.Literal("zoom"), + // Not Type.Tuple: tuples emit draft-07 `items: [...]`, which Anthropic's + // draft 2020-12 schema validation rejects. + region: Type.Array(Type.Number(), { + minItems: 4, + maxItems: 4, + description: "[x0, y0, x1, y1] crop region in OS screenshot pixels.", + }), + }, + { additionalProperties: false }, + ), + goto: Type.Object( + { + type: Type.Literal("goto"), + url: Type.String(), + }, + { additionalProperties: false }, + ), + back: Type.Object({ type: Type.Literal("back") }, { additionalProperties: false }), + forward: Type.Object({ type: Type.Literal("forward") }, { additionalProperties: false }), + url: Type.Object({ type: Type.Literal("url") }, { additionalProperties: false }), + cursor_position: Type.Object({ type: Type.Literal("cursor_position") }, { additionalProperties: false }), +} satisfies Record; + +export type CuaZoomRegion = CuaActionZoom["region"]; diff --git a/packages/ai/src/actions/index.ts b/packages/ai/src/actions/index.ts new file mode 100644 index 00000000..a2d164e5 --- /dev/null +++ b/packages/ai/src/actions/index.ts @@ -0,0 +1,47 @@ +import type { TSchema } from "@earendil-works/pi-ai"; +import { CUA_BROWSER_ACTION_TYPES, createCuaBrowserActionSchemaByType, type CuaBrowserAction, type CuaBrowserActionType, type CuaBrowserSchemaOptions } from "./browser"; +import { CUA_COMPUTER_ACTION_SCHEMA_BY_TYPE, CUA_COMPUTER_ACTION_TYPES, type CuaComputerAction, type CuaComputerActionType } from "./computer"; + +export * from "./browser"; +export * from "./computer"; + +/** Any canonical CUA action type, across the computer and browser planes. */ +export type CuaActionType = CuaComputerActionType | CuaBrowserActionType; + +/** Any canonical CUA action, across the computer and browser planes. */ +export type CuaAction = CuaComputerAction | CuaBrowserAction; + +/** Every canonical action type: the computer plane followed by the browser plane. */ +export const CUA_ALL_ACTION_TYPES: readonly CuaActionType[] = [...CUA_COMPUTER_ACTION_TYPES, ...CUA_BROWSER_ACTION_TYPES]; + +const COMPUTER_ACTION_TYPE_SET: ReadonlySet = new Set(CUA_COMPUTER_ACTION_TYPES); +const BROWSER_ACTION_TYPE_SET: ReadonlySet = new Set(CUA_BROWSER_ACTION_TYPES); + +/** Whether a canonical action type belongs to the computer plane. */ +export function isCuaComputerActionType(action: CuaActionType): action is CuaComputerActionType { + return COMPUTER_ACTION_TYPE_SET.has(action); +} + +/** Whether a canonical action type belongs to the browser plane. */ +export function isCuaBrowserActionType(action: CuaActionType): action is CuaBrowserActionType { + return BROWSER_ACTION_TYPE_SET.has(action); +} + +/** Whether a canonical action belongs to the browser plane. */ +export function isCuaBrowserAction(action: CuaAction): action is CuaBrowserAction { + return BROWSER_ACTION_TYPE_SET.has(action.type); +} + +/** Options for building canonical action schemas. */ +export interface CuaActionSchemaOptions { + /** browser-plane schema variants; see {@link CuaBrowserSchemaOptions}. Defaults to coordinates allowed. */ + browser?: CuaBrowserSchemaOptions; +} + +/** Build the full action-type → schema map for a schema-options combination. */ +export function cuaActionSchemaByType(options: CuaActionSchemaOptions = {}): Record { + return { + ...CUA_COMPUTER_ACTION_SCHEMA_BY_TYPE, + ...createCuaBrowserActionSchemaByType(options.browser ?? { coordinates: true }), + }; +} diff --git a/packages/ai/src/modes.ts b/packages/ai/src/modes.ts new file mode 100644 index 00000000..cbc83f7d --- /dev/null +++ b/packages/ai/src/modes.ts @@ -0,0 +1,171 @@ +import { + CUA_BROWSER_ACTION_TYPES, + CUA_DEFAULT_COMPUTER_ACTION_TYPES, + isCuaComputerActionType, + type CuaActionSchemaOptions, + type CuaActionType, + type CuaBrowserActionType, + type CuaComputerActionType, +} from "./actions/index"; + +/** + * Which canonical action plane(s) a CUA agent exposes to the model. + * + * - `computer` — OS-level input only (mouse/keyboard/display against the + * VM). Today's default; coordinates are OS screenshot pixels. Pairs with + * Anthropic's native `computer_20260701` tool. + * - `browser` — page tools only, driven over CDP: accessibility snapshots + * with element refs, element-targeted interaction, navigation, tabs, and + * viewport screenshots. Coordinates, where used, are viewport pixels. + * Pairs with Anthropic's native `browser_20260701` tool. + * - `hybrid` — both planes, deduplicated to one tool per capability. + * Computer tools are prefixed `computer_`, browser tools keep their + * `browser_` prefix and accept element refs only, and the OS screenshot + * frame is the single live coordinate frame. + */ +export type CuaMode = "computer" | "browser" | "hybrid"; + +/** + * Computer actions exposed in hybrid mode: navigation reads/writes are + * excluded because they live on the browser plane (`browser_navigate`, + * `browser_list_tabs`), and `zoom` is included since the OS screenshot is + * hybrid's only capture. + */ +export const CUA_HYBRID_COMPUTER_ACTION_TYPES: readonly CuaComputerActionType[] = [ + "click", + "double_click", + "mouse_down", + "mouse_up", + "type", + "keypress", + "scroll", + "move", + "drag", + "wait", + "screenshot", + "zoom", + "cursor_position", +]; + +/** + * Browser actions exposed in hybrid mode: reads, element-targeted writes, + * and JavaScript evaluation. Pointer/keyboard capabilities (`browser_click` by coordinate, + * `browser_type`, `browser_key`, `browser_scroll`, `browser_hover`, `browser_drag`) and + * `browser_screenshot` are excluded — real OS input and the OS screenshot cover + * those, keeping one tool per capability and one coordinate frame. + */ +export const CUA_HYBRID_BROWSER_ACTION_TYPES: readonly CuaBrowserActionType[] = [ + "browser_snapshot", + "browser_text", + "browser_find", + "browser_click", + "browser_fill", + "browser_scroll_to", + "browser_navigate", + "browser_list_tabs", + "browser_new_tab", + "browser_evaluate", +]; + +/** Resolve the default canonical action set for a mode. */ +export function defaultActionsForMode(mode: CuaMode): readonly CuaActionType[] { + switch (mode) { + case "computer": + return CUA_DEFAULT_COMPUTER_ACTION_TYPES; + case "browser": + return [...CUA_BROWSER_ACTION_TYPES, "wait"]; + case "hybrid": + return [...CUA_HYBRID_COMPUTER_ACTION_TYPES, ...CUA_HYBRID_BROWSER_ACTION_TYPES]; + } +} + +/** Resolve the schema-building options for a mode; see {@link CuaActionSchemaOptions}. */ +export function schemaOptionsForMode(mode: CuaMode): CuaActionSchemaOptions { + // Hybrid restricts browser actions to element refs so the OS screenshot + // frame is the single live coordinate frame. Browser mode has no OS frame, + // so viewport coordinates are allowed there. + return { browser: { coordinates: mode !== "hybrid" } }; +} + +/** + * The model-facing tool name for a canonical action in a mode. + * + * - `computer`: canonical action ids as-is (`click`, `screenshot`, …). + * - `browser`: browser ids with the `browser_` prefix stripped (`snapshot`, + * `click`, …); the prefix only exists to disambiguate planes, and + * browser mode has one. + * - `hybrid`: computer ids prefixed `computer_`, browser ids kept as `browser_*`. + */ +export function cuaToolNameForAction(action: CuaActionType, mode: CuaMode): string { + switch (mode) { + case "computer": + if (!isCuaComputerActionType(action)) throw new Error(`browser action "${action}" is not available in computer mode`); + return action; + case "browser": + return isCuaComputerActionType(action) ? action : action.slice("browser_".length); + case "hybrid": + return isCuaComputerActionType(action) ? `computer_${action}` : action; + } +} + +const BROWSER_ACTION_DESCRIPTIONS: Record = { + browser_snapshot: + "Return an accessibility-tree snapshot of the page, including iframe content, with element references like [e12]. " + + "Use the refs to target elements in other page tools. Refs are only valid until the page changes; re-snapshot when told a ref is stale. " + + "If the page has not changed since your previous snapshot, a short unchanged notice is returned instead and earlier refs remain valid.", + browser_text: "Return the page's visible text content as plain text. Best for articles and text-heavy pages.", + browser_find: + "Find elements on the page (including iframe content) matching a natural-language description and return them with element references, like a filtered snapshot.", + browser_click: "Click an element. Prefer targeting by element reference from a snapshot.", + browser_hover: "Move the pointer over an element without clicking.", + browser_drag: "Drag from one viewport coordinate to another.", + browser_fill: "Set the value of a form element (input, textarea, select, checkbox) by element reference.", + browser_scroll_to: "Scroll an element into view by element reference.", + browser_scroll: "Scroll the page at a viewport position by wheel notches.", + browser_type: "Type a literal string at the current focus.", + browser_key: "Press a key or chord, e.g. \"Return\" or \"ctrl+a\".", + browser_navigate: "Navigate the page to a URL, or \"back\" / \"forward\" in history.", + browser_list_tabs: "List open tabs with each tab's id, title, and URL.", + browser_new_tab: "Open a new empty tab and return its tab id.", + browser_screenshot: "Capture the current browser viewport.", + browser_evaluate: "Execute JavaScript in the page context and return the value of the last expression.", +}; + +// Hybrid exposes both planes, so tool descriptions carry the arbitration +// rules the model needs: which plane is preferred for a capability and why +// (real OS input vs CDP), plus the single-coordinate-frame statement. +const HYBRID_COMPUTER_DESCRIPTION_OVERRIDES: Partial> = { + click: + "Click at a coordinate in OS screenshot pixels using real OS-level input. " + + "Preferred over browser_click when the target is visible in the screenshot — OS input is indistinguishable from a human user.", + screenshot: "Capture the display. This is the only screenshot tool; all coordinates refer to this image's pixels.", + zoom: "Return a cropped view of the current display for closer inspection. Coordinates in later actions still refer to the full screenshot, not the crop.", + scroll: "Scroll with the OS-level mouse wheel at a coordinate in OS screenshot pixels.", + type: "Type a literal string with OS-level keyboard input at the current focus.", + keypress: "Press keys with OS-level keyboard input.", +}; + +const HYBRID_BROWSER_DESCRIPTION_OVERRIDES: Partial> = { + browser_click: + "Click an element by reference from a browser_snapshot. Dispatched via CDP, which protected sites may detect — " + + "prefer computer_click when the element is visible in the screenshot; use browser_click for elements that are hard to hit by coordinate.", + browser_snapshot: + "Return an accessibility-tree snapshot of the page, including iframe content, with element references like [e12]. " + + "This is the high-fidelity way to read page structure — prefer it over screenshots for reading and locating elements. " + + "Refs are only valid until the page changes; re-snapshot when told a ref is stale. " + + "If the page has not changed since your previous snapshot, a short unchanged notice is returned instead and earlier refs remain valid.", +}; + +/** The model-facing tool description for a canonical action in a mode. */ +export function cuaToolDescriptionForAction(action: CuaActionType, mode: CuaMode): string { + if (isCuaComputerActionType(action)) { + if (mode === "hybrid") { + return HYBRID_COMPUTER_DESCRIPTION_OVERRIDES[action] ?? `Execute one ${action} computer action using real OS-level input.`; + } + return `Execute one ${action} computer action.`; + } + if (mode === "hybrid") { + return HYBRID_BROWSER_DESCRIPTION_OVERRIDES[action] ?? BROWSER_ACTION_DESCRIPTIONS[action]; + } + return BROWSER_ACTION_DESCRIPTIONS[action]; +} diff --git a/packages/ai/src/native-tools.ts b/packages/ai/src/native-tools.ts new file mode 100644 index 00000000..99c2dbeb --- /dev/null +++ b/packages/ai/src/native-tools.ts @@ -0,0 +1,115 @@ +import type { Api, Model } from "@earendil-works/pi-ai"; +import type { CuaMode } from "./modes"; + +/** + * Anthropic's native computer-use tool (`anthropic-beta: + * computer-use-2026-07-01`). Server-defined: the declaration below is sent + * verbatim in `tools[]` and Anthropic fixes the input schema. Maps to CUA's + * `computer` mode — actions arrive as OS-level input in screenshot-pixel + * coordinates. + */ +export interface AnthropicComputerNativeTool { + type: "computer_20260701"; + /** Tool name in the request; defaults to "computer". */ + name?: string; + /** Adds a `zoom` action returning a cropped screenshot region. Default false. */ + enable_zoom?: boolean; + /** X11 display number for multi-display environments. */ + display_number?: number; + /** Prompt-caching breakpoint. */ + cache_control?: { type: "ephemeral" }; +} + +/** + * Anthropic's native browser tool (`anthropic-beta: browser-use-2026-07-01`, + * proposed — the tool version and schema may change before release). Maps to + * CUA's `browser` mode — page reads by element reference plus pointer actions in + * viewport-pixel coordinates. + */ +export interface AnthropicBrowserNativeTool { + type: "browser_20260701"; + /** Tool name in the request; defaults to "browser". */ + name?: string; + /** Adds a `javascript_exec` action running arbitrary JS in the page. Default false. */ + enable_javascript_exec?: boolean; + /** Prompt-caching breakpoint. */ + cache_control?: { type: "ephemeral" }; +} + +/** + * A provider-native computer-use tool declaration. + * + * Pass one to `resolveCuaRuntimeSpec` (or `CuaAgent`/`CuaAgentHarness`) to + * drive the model through its provider-defined tool schema instead of CUA's + * canonical function tools. Each native tool pairs with exactly one + * {@link CuaMode}; execution is identical either way — native tool calls are + * translated to the same canonical actions the mode uses. + */ +export type CuaNativeToolSpec = AnthropicComputerNativeTool | AnthropicBrowserNativeTool; + +export type CuaNativeToolType = CuaNativeToolSpec["type"]; + +interface NativeToolInfo { + mode: Extract; + provider: "anthropic"; + betaHeader: string; + defaultName: string; +} + +const NATIVE_TOOL_INFO: Record = { + computer_20260701: { mode: "computer", provider: "anthropic", betaHeader: "computer-use-2026-07-01", defaultName: "computer" }, + browser_20260701: { mode: "browser", provider: "anthropic", betaHeader: "browser-use-2026-07-01", defaultName: "browser" }, +}; + +/** The {@link CuaMode} a native tool requires. */ +export function modeForNativeTool(spec: CuaNativeToolSpec): CuaMode { + return NATIVE_TOOL_INFO[spec.type].mode; +} + +/** The `anthropic-beta` header value a native tool requires. */ +export function betaHeaderForNativeTool(spec: CuaNativeToolSpec): string { + return NATIVE_TOOL_INFO[spec.type].betaHeader; +} + +/** The tool name a native tool declares, defaulting per tool type. */ +export function nativeToolName(spec: CuaNativeToolSpec): string { + return spec.name ?? NATIVE_TOOL_INFO[spec.type].defaultName; +} + +/** A validated native tool: the spec plus everything derived from it. */ +export interface ResolvedCuaNativeTool { + spec: CuaNativeToolSpec; + /** The declaration sent verbatim in the provider `tools[]` array. */ + declaration: Record; + /** Tool name the model's tool_use blocks arrive under. */ + name: string; + /** Required `anthropic-beta` header value. */ + betaHeader: string; + mode: CuaMode; +} + +/** + * Validate a native tool spec against the resolved model and mode, and + * derive the request-facing pieces. Throws when the model's provider does + * not serve the tool or the mode conflicts with the tool's plane (mirroring + * the API, which rejects e.g. `browser_20260701` outside a one-frame + * browser-only request). + */ +export function resolveNativeTool(spec: CuaNativeToolSpec, model: Model, mode: CuaMode): ResolvedCuaNativeTool { + const info = NATIVE_TOOL_INFO[spec.type]; + if (!info) throw new Error(`unknown native tool type "${(spec as { type: string }).type}"`); + if (model.provider !== info.provider) { + throw new Error(`native tool "${spec.type}" requires an ${info.provider} model paired with mode "${info.mode}"; got provider "${model.provider}"`); + } + if (mode !== info.mode) { + throw new Error(`native tool "${spec.type}" requires mode "${info.mode}"; got "${mode}"`); + } + const name = nativeToolName(spec); + return { + spec, + declaration: { ...spec, name }, + name, + betaHeader: info.betaHeader, + mode, + }; +} diff --git a/packages/ai/src/providers.ts b/packages/ai/src/providers.ts index 29492684..9a5d1c18 100644 --- a/packages/ai/src/providers.ts +++ b/packages/ai/src/providers.ts @@ -13,6 +13,7 @@ import { import { builtinModels } from "@earendil-works/pi-ai/providers/all"; import { cuaApiKeyEnvVarsForProvider } from "./api-keys"; import { cuaOverrideModels } from "./models"; +import { ANTHROPIC_NATIVE_API_BETA_HEADERS, withAnthropicBetaHeader } from "./providers/anthropic/native"; import { OPENAI_CUA_RESPONSES_API, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; import { streamSimpleTzafonResponses, streamTzafonResponses, TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; @@ -25,6 +26,10 @@ import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from ". * {@link getCuaModel} routes OpenAI models to, threading * `previous_response_id`; every other api falls through to pi's builtin * provider. + * - `anthropic` intercepts the native computer/browser tool apis that + * `resolveCuaRuntimeSpec` routes models with a `nativeTool` to, dispatching + * them to pi's builtin `anthropic-messages` transport with the tool's + * `anthropic-beta` header merged in. * - `google` resolves its API key from `GOOGLE_API_KEY` or `GEMINI_API_KEY` * (pi's builtin only reads `GEMINI_API_KEY`). * - `tzafon` and `yutori` are CUA-only providers pi does not ship. @@ -36,6 +41,8 @@ export function createCuaModels(options?: CreateModelsOptions): MutableModels { const models = builtinModels(options); const openai = models.getProvider("openai"); if (openai) models.setProvider(withOpenAICuaResponses(openai)); + const anthropic = models.getProvider("anthropic"); + if (anthropic) models.setProvider(withAnthropicNativeTools(anthropic)); const google = models.getProvider("google"); if (google) models.setProvider(withGoogleEnvKeys(google)); models.setProvider(tzafonProvider()); @@ -74,6 +81,27 @@ function withOpenAICuaResponses(base: Provider): Provider { }; } +// Native-tool runs route Anthropic models to a CUA-owned api id (see +// resolveCuaRuntimeSpec) so the required `anthropic-beta` header can be +// injected here; the request otherwise flows through pi's builtin +// anthropic-messages transport. +function withAnthropicNativeTools(base: Provider): Provider { + const toBuiltin = (model: Model): Model => ({ ...model, api: "anthropic-messages" as Model["api"] }); + return { + ...base, + stream: (model: Model, context: Context, options?: StreamOptions) => { + const beta = ANTHROPIC_NATIVE_API_BETA_HEADERS[model.api]; + return beta ? base.stream(toBuiltin(model), context, withAnthropicBetaHeader(options, beta)) : base.stream(model, context, options); + }, + streamSimple: (model: Model, context: Context, options?: SimpleStreamOptions) => { + const beta = ANTHROPIC_NATIVE_API_BETA_HEADERS[model.api]; + return beta + ? base.streamSimple(toBuiltin(model), context, withAnthropicBetaHeader(options, beta)) + : base.streamSimple(model, context, options); + }, + }; +} + function withGoogleEnvKeys(base: Provider): Provider { return { ...base, auth: { ...base.auth, apiKey: envApiKeyAuth("Google API key", cuaApiKeyEnvVarsForProvider("google")) } }; } diff --git a/packages/ai/src/providers/anthropic/actions.ts b/packages/ai/src/providers/anthropic/actions.ts index af0a1fc4..af51bda8 100644 --- a/packages/ai/src/providers/anthropic/actions.ts +++ b/packages/ai/src/providers/anthropic/actions.ts @@ -2,14 +2,18 @@ import type { Tool, TSchema } from "@earendil-works/pi-ai"; import { CUA_BATCH_TOOL_DESCRIPTION, CUA_BATCH_TOOL_NAME, + CUA_BROWSER_ACTION_TYPES, createCuaActionSchema, createCuaActionToolExecutors, createCuaActionToolDefinitions, createCuaBatchToolExecutor, createCuaBatchToolDefinition, + defaultActionsForMode, + isCuaBrowserActionType, type ComputerToolsOptions, type CuaAction, type CuaActionType, + type CuaMode, type CuaToolExecutorSpec, } from "../common"; @@ -34,13 +38,14 @@ export const ANTHROPIC_CUA_ACTION_TYPES = [ "drag", "wait", "screenshot", + "zoom", "goto", "cursor_position", ] as const satisfies readonly CuaActionType[]; -type AnthropicCanonicalActionType = (typeof ANTHROPIC_CUA_ACTION_TYPES)[number]; +type AnthropicCanonicalActionType = (typeof ANTHROPIC_CUA_ACTION_TYPES)[number] | (typeof CUA_BROWSER_ACTION_TYPES)[number]; -const ANTHROPIC_CANONICAL_ACTION_TYPE_SET: ReadonlySet = new Set(ANTHROPIC_CUA_ACTION_TYPES); +const ANTHROPIC_CANONICAL_ACTION_TYPE_SET: ReadonlySet = new Set([...ANTHROPIC_CUA_ACTION_TYPES, ...CUA_BROWSER_ACTION_TYPES]); /** Name of the batch tool included by default in Anthropic computer-use tools. */ export const ANTHROPIC_BATCH_TOOL_NAME = CUA_BATCH_TOOL_NAME; @@ -59,8 +64,15 @@ export interface AnthropicComputerToolsOptions extends ComputerToolsOptions { /** Canonical CUA action shape supported by Anthropic browser computer-use tools. */ export type AnthropicAction = Extract; -function resolveAnthropicActions(actions: readonly CuaActionType[] | undefined): readonly AnthropicCanonicalActionType[] { - const resolved = actions ?? ANTHROPIC_CUA_ACTION_TYPES; +function resolveAnthropicActions(options: AnthropicComputerToolsOptions): readonly AnthropicCanonicalActionType[] { + const mode = options.mode ?? "computer"; + const resolved = + options.actions ?? + (mode === "computer" + ? ANTHROPIC_CUA_ACTION_TYPES.filter((action) => action !== "zoom") + : defaultActionsForMode(mode).filter( + (action) => isCuaBrowserActionType(action) || isAnthropicCanonicalAction(action), + )); const supported: AnthropicCanonicalActionType[] = []; const unsupported: CuaActionType[] = []; for (const action of resolved) { @@ -76,8 +88,8 @@ function isAnthropicCanonicalAction(action: CuaActionType): action is AnthropicC } /** Build the TypeBox schema for Anthropic-supported canonical browser actions. */ -export function createActionSchema(actions?: readonly CuaActionType[]): TSchema { - return createCuaActionSchema(resolveAnthropicActions(actions)); +export function createActionSchema(actions?: readonly CuaActionType[], mode: CuaMode = "computer"): TSchema { + return createCuaActionSchema(resolveAnthropicActions({ actions, mode }), mode); } /** @@ -89,12 +101,14 @@ export function createActionSchema(actions?: readonly CuaActionType[]): TSchema * batch tool by default; pass `excludeBatch: true` to omit it. */ export function computerTools(options: AnthropicComputerToolsOptions = {}): Tool[] { - const actions = resolveAnthropicActions(options.actions); - const tools = createCuaActionToolDefinitions(actions); + const mode = options.mode ?? "computer"; + const actions = resolveAnthropicActions(options); + const tools = createCuaActionToolDefinitions(actions, mode); if (!options.excludeBatch) { tools.push(createCuaBatchToolDefinition(actions, { name: ANTHROPIC_BATCH_TOOL_NAME, description: ANTHROPIC_BATCH_TOOL_DESCRIPTION, + mode, })); } return tools; @@ -102,12 +116,14 @@ export function computerTools(options: AnthropicComputerToolsOptions = {}): Tool /** Build the local execution adapters used by CuaAgent and CuaAgentHarness. */ export function computerToolExecutors(options: AnthropicComputerToolsOptions = {}): CuaToolExecutorSpec[] { - const actions = resolveAnthropicActions(options.actions); - const executors = createCuaActionToolExecutors(actions); + const mode = options.mode ?? "computer"; + const actions = resolveAnthropicActions(options); + const executors = createCuaActionToolExecutors(actions, mode); if (!options.excludeBatch) { executors.push(createCuaBatchToolExecutor(actions, { name: ANTHROPIC_BATCH_TOOL_NAME, description: ANTHROPIC_BATCH_TOOL_DESCRIPTION, + mode, })); } return executors; diff --git a/packages/ai/src/providers/anthropic/index.ts b/packages/ai/src/providers/anthropic/index.ts index f51b1ae8..ff056198 100644 --- a/packages/ai/src/providers/anthropic/index.ts +++ b/packages/ai/src/providers/anthropic/index.ts @@ -1,5 +1,5 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import type { ComputerToolCoordinateSystem, CuaPayloadHook, CuaProviderModule } from "../common"; +import type { ComputerToolCoordinateSystem, CuaMode, CuaPayloadHook, CuaProviderModule } from "../common"; import { computerToolExecutors, computerTools } from "./actions"; export { @@ -14,6 +14,17 @@ export type { AnthropicComputerToolsOptions, AnthropicComputerToolsOptions as ComputerToolsOptions, } from "./actions"; +export { + ANTHROPIC_NATIVE_API_BETA_HEADERS, + ANTHROPIC_NATIVE_BROWSER_MESSAGES_API, + ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API, + createNativeToolOnPayload, + mapNativeBrowserInput, + mapNativeComputerInput, + nativeApiForToolType, + nativeToolExecutors, + withAnthropicBetaHeader, +} from "./native"; // Anthropic's quickstart uses pixel coordinates for both its computer and // browser tools. @@ -24,8 +35,18 @@ export function coordinateSystem(): ComputerToolCoordinateSystem { export const ANTHROPIC_COMPUTER_INSTRUCTIONS = `You control a Kernel cloud browser through individual browser tools. Use keyboard navigation where possible, and request screenshots when you need to inspect state.`; -export function buildAnthropicSystemPrompt(opts: { suffix?: string } = {}): string { - return [ANTHROPIC_COMPUTER_INSTRUCTIONS, opts.suffix].filter(Boolean).join("\n\n"); +export const ANTHROPIC_BROWSER_INSTRUCTIONS = `You control a Kernel cloud browser through page tools. Prefer reading the page with snapshot or find and targeting elements by reference; use screenshots when you need to inspect visual state. Element references go stale when the page changes — re-snapshot when told so.`; + +export const ANTHROPIC_HYBRID_INSTRUCTIONS = `You control a Kernel cloud browser through two kinds of tools: computer_* tools perform real OS-level input (coordinates are pixels in the most recent computer_screenshot), and browser_* tools read and act on the page itself by element reference. Prefer browser_snapshot/browser_find for reading and locating, and computer_* input for interaction; use browser_* interaction for elements that are hard to hit by coordinate.`; + +export function buildAnthropicSystemPrompt(opts: { suffix?: string; mode?: CuaMode } = {}): string { + const base = + opts.mode === "browser" + ? ANTHROPIC_BROWSER_INSTRUCTIONS + : opts.mode === "hybrid" + ? ANTHROPIC_HYBRID_INSTRUCTIONS + : ANTHROPIC_COMPUTER_INSTRUCTIONS; + return [base, opts.suffix].filter(Boolean).join("\n\n"); } export const anthropicAdaptiveThinkingOnPayload: CuaPayloadHook = (payload, model) => { diff --git a/packages/ai/src/providers/anthropic/native.ts b/packages/ai/src/providers/anthropic/native.ts new file mode 100644 index 00000000..a1f2b18c --- /dev/null +++ b/packages/ai/src/providers/anthropic/native.ts @@ -0,0 +1,311 @@ +import { Type, type Api, type Model, type StreamOptions, type Tool } from "@earendil-works/pi-ai"; +import type { CuaAction, CuaMouseButton } from "../../actions/index"; +import type { ResolvedCuaNativeTool } from "../../native-tools"; +import type { CuaPayloadContext, CuaPayloadHook, CuaToolExecutorSpec } from "../common"; + +/** + * pi-ai api ids CUA routes Anthropic models to when a native tool is + * configured. The registered provider dispatches these to pi's builtin + * `anthropic-messages` transport with the tool's `anthropic-beta` header + * merged in (see `createCuaModels`). + */ +export const ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API = "anthropic-cua-native-computer-messages"; +export const ANTHROPIC_NATIVE_BROWSER_MESSAGES_API = "anthropic-cua-native-browser-messages"; + +export const ANTHROPIC_NATIVE_API_BETA_HEADERS: Record = { + [ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API]: "computer-use-2026-07-01", + [ANTHROPIC_NATIVE_BROWSER_MESSAGES_API]: "browser-use-2026-07-01", +}; + +export function nativeApiForToolType(type: ResolvedCuaNativeTool["spec"]["type"]): string { + return type === "computer_20260701" ? ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API : ANTHROPIC_NATIVE_BROWSER_MESSAGES_API; +} + +/** Merge a native tool's `anthropic-beta` header into stream options. */ +export function withAnthropicBetaHeader(options: T | undefined, beta: string): T { + const headers = { ...(options?.headers ?? {}) }; + headers["anthropic-beta"] = headers["anthropic-beta"] ? `${headers["anthropic-beta"]},${beta}` : beta; + return { ...(options ?? {}), headers } as T; +} + +// The native tool's input schema is Anthropic-defined and validated +// server-side; the local placeholder schema stays permissive and the +// executor validates during mapping. +const NativeActionSchema = Type.Object({ action: Type.String() }, { additionalProperties: true }); + +/** + * Build the single execution adapter for a native Anthropic tool: tool calls + * arrive under the native tool's name with an `action`-discriminated input, + * and map onto the same canonical actions the tool's mode uses. + */ +export function nativeToolExecutors(resolved: ResolvedCuaNativeTool): CuaToolExecutorSpec[] { + const definition: Tool = { + name: resolved.name, + description: `Anthropic native ${resolved.spec.type} tool.`, + parameters: NativeActionSchema, + }; + const toActions = + resolved.spec.type === "computer_20260701" + ? (args: unknown) => mapNativeComputerInput(asNativeInput(args)) + : (args: unknown) => mapNativeBrowserInput(asNativeInput(args)); + return [{ definition, toActions }]; +} + +/** + * Payload hook for native tool requests: replaces the local placeholder + * function tool (matched by name) with the Anthropic-defined declaration. + * Other tools in the payload (e.g. `playwright_execute`, caller extras) are + * left in place. + */ +export function createNativeToolOnPayload(resolved: ResolvedCuaNativeTool): CuaPayloadHook { + return (payload: unknown, _model: Model, _context?: CuaPayloadContext) => { + if (!payload || typeof payload !== "object") return undefined; + const current = payload as { tools?: unknown }; + if (!Array.isArray(current.tools)) return undefined; + const tools = current.tools.map((tool) => + tool && typeof tool === "object" && (tool as { name?: unknown }).name === resolved.name ? resolved.declaration : tool, + ); + return { ...(payload as Record), tools }; + }; +} + +interface NativeInput { + action: string; + [key: string]: unknown; +} + +function asNativeInput(args: unknown): NativeInput { + if (args && typeof args === "object" && typeof (args as { action?: unknown }).action === "string") { + return args as NativeInput; + } + throw new Error("invalid native tool parameters: expected an object with an \"action\" field"); +} + +const MAX_KEY_REPEAT = 100; + +/** Map one `computer_20260701` tool input onto canonical computer-plane actions. */ +export function mapNativeComputerInput(input: NativeInput): CuaAction[] { + switch (input.action) { + case "screenshot": + return [{ type: "screenshot" }]; + case "left_click": + return [click(input, "left")]; + case "right_click": + return [click(input, "right")]; + case "middle_click": + return [click(input, "middle")]; + case "double_click": + return [{ ...click(input, "left"), num_clicks: 2 }]; + case "triple_click": + return [{ ...click(input, "left"), num_clicks: 3 }]; + case "left_click_drag": { + const start = coordinate(input.start_coordinate, "start_coordinate"); + const end = coordinate(input.coordinate, "coordinate"); + return [{ type: "drag", path: [start, end], ...holdKeys(input.text) }]; + } + case "mouse_move": + return [{ type: "move", ...coordinate(input.coordinate, "coordinate") }]; + case "left_mouse_down": + return [{ type: "mouse_down" }]; + case "left_mouse_up": + return [{ type: "mouse_up" }]; + case "scroll": { + const point = input.coordinate === undefined ? {} : coordinate(input.coordinate, "coordinate"); + return [{ type: "scroll", ...point, ...scrollDeltas(input.scroll_direction, input.scroll_amount), ...holdKeys(input.text) }]; + } + case "type": + return [{ type: "type", text: text(input) }]; + case "key": { + const repeat = clampRepeat(input.repeat); + return Array.from({ length: repeat }, () => ({ type: "keypress" as const, keys: [text(input)] })); + } + case "hold_key": + // Canonical keypress duration is milliseconds; the native tool speaks seconds. + return [{ type: "keypress", keys: [text(input)], duration: durationSeconds(input) * 1000 }]; + case "wait": + return [{ type: "wait", ms: durationSeconds(input) * 1000 }]; + case "cursor_position": + return [{ type: "cursor_position" }]; + case "zoom": + return [{ type: "zoom", region: region(input.region) }]; + default: + throw new Error(`unsupported computer_20260701 action "${input.action}"`); + } +} + +/** Map one `browser_20260701` tool input onto canonical browser-plane actions. */ +export function mapNativeBrowserInput(input: NativeInput): CuaAction[] { + const tab = tabId(input); + switch (input.action) { + case "navigate": + return [{ type: "browser_navigate", url: requireString(input.url, "url"), ...tab }]; + case "list_tabs": + return [{ type: "browser_list_tabs" }]; + case "new_tab": + return [{ type: "browser_new_tab" }]; + case "read_page": + return [ + { + type: "browser_snapshot", + ...(input.filter === "interactive" || input.filter === "all" ? { filter: input.filter } : {}), + ...(typeof input.depth === "number" ? { depth: input.depth } : {}), + ...(typeof input.ref === "string" ? { ref: input.ref } : {}), + ...tab, + }, + ]; + case "get_page_text": + return [{ type: "browser_text", ...tab }]; + case "find": + return [{ type: "browser_find", query: requireString(input.query, "query"), ...tab }]; + case "form_input": + return [{ type: "browser_fill", ref: refTarget(input.target), value: fillValue(input.value), ...tab }]; + case "scroll_to": + return [{ type: "browser_scroll_to", ref: refTarget(input.target), ...tab }]; + case "screenshot": + return [{ type: "browser_screenshot", ...tab }]; + case "zoom": + return [{ type: "browser_screenshot", region: region(input.region), ...tab }]; + case "left_click": + return [{ type: "browser_click", ...pageTarget(input.target), ...modifiers(input.modifiers), ...tab }]; + case "right_click": + return [{ type: "browser_click", ...pageTarget(input.target), button: "right", ...modifiers(input.modifiers), ...tab }]; + case "double_click": + return [{ type: "browser_click", ...pageTarget(input.target), num_clicks: 2, ...modifiers(input.modifiers), ...tab }]; + case "triple_click": + return [{ type: "browser_click", ...pageTarget(input.target), num_clicks: 3, ...modifiers(input.modifiers), ...tab }]; + case "hover": + return [{ type: "browser_hover", ...pageTarget(input.target), ...tab }]; + case "left_click_drag": + return [{ type: "browser_drag", from: coordinateTarget(input.from, "from"), to: coordinateTarget(input.target, "target"), ...tab }]; + case "scroll": + return [ + { + type: "browser_scroll", + ...coordinateTarget(input.target, "target"), + direction: scrollDirection(input.scroll_direction), + ...(typeof input.scroll_amount === "number" ? { amount: input.scroll_amount } : {}), + ...tab, + }, + ]; + case "type": + return [{ type: "browser_type", text: text(input), ...tab }]; + case "key": { + const repeat = clampRepeat(input.repeat); + return Array.from({ length: repeat }, () => ({ type: "browser_key" as const, text: text(input), ...tab })); + } + case "wait": + return [{ type: "wait", ms: durationSeconds(input) * 1000 }]; + case "javascript_exec": + return [{ type: "browser_evaluate", code: text(input), ...tab }]; + default: + throw new Error(`unsupported browser_20260701 action "${input.action}"`); + } +} + +function click(input: NativeInput, button: CuaMouseButton): CuaAction & { type: "click" } { + const point = input.coordinate === undefined ? {} : coordinate(input.coordinate, "coordinate"); + return { type: "click", ...point, button, ...holdKeys(input.text) }; +} + +function coordinate(value: unknown, field: string): { x: number; y: number } { + if (Array.isArray(value) && value.length === 2 && typeof value[0] === "number" && typeof value[1] === "number") { + return { x: value[0], y: value[1] }; + } + throw new Error(`invalid ${field}: expected [x, y]`); +} + +function region(value: unknown): [number, number, number, number] { + if (Array.isArray(value) && value.length === 4 && value.every((entry) => typeof entry === "number")) { + return value as [number, number, number, number]; + } + throw new Error("invalid region: expected [x0, y0, x1, y1]"); +} + +function holdKeys(value: unknown): { hold_keys?: string[] } { + return typeof value === "string" && value.trim() ? { hold_keys: value.split("+").map((key) => key.trim()) } : {}; +} + +function modifiers(value: unknown): { modifiers?: string[] } { + return typeof value === "string" && value.trim() ? { modifiers: value.split("+").map((key) => key.trim()) } : {}; +} + +function scrollDeltas(direction: unknown, amount: unknown): { scroll_x?: number; scroll_y?: number } { + const notches = typeof amount === "number" && Number.isFinite(amount) ? amount : 3; + const delta = Math.trunc(notches) * 120; + switch (direction) { + case "up": + return { scroll_y: -delta }; + case "down": + return { scroll_y: delta }; + case "left": + return { scroll_x: -delta }; + case "right": + return { scroll_x: delta }; + default: + throw new Error(`invalid scroll_direction "${String(direction)}"`); + } +} + +function scrollDirection(value: unknown): "up" | "down" | "left" | "right" { + if (value === "up" || value === "down" || value === "left" || value === "right") return value; + throw new Error(`invalid scroll_direction "${String(value)}"`); +} + +function text(input: NativeInput): string { + return requireString(input.text, "text"); +} + +function requireString(value: unknown, field: string): string { + if (typeof value !== "string") throw new Error(`invalid ${field}: expected a string`); + return value; +} + +function durationSeconds(input: NativeInput): number { + const value = input.duration; + if (typeof value !== "number" || !Number.isFinite(value) || value < 0) throw new Error("invalid duration: expected a non-negative number"); + return Math.min(value, 100); +} + +function clampRepeat(value: unknown): number { + if (value === undefined) return 1; + if (typeof value !== "number" || !Number.isInteger(value) || value < 1) throw new Error("invalid repeat: expected an integer ≥ 1"); + return Math.min(value, MAX_KEY_REPEAT); +} + +interface RefOrCoordinateTarget { + ref?: string; + x?: number; + y?: number; +} + +function pageTarget(value: unknown): RefOrCoordinateTarget { + const target = value as { type?: unknown; ref?: unknown; x?: unknown; y?: unknown } | undefined; + if (target && typeof target === "object") { + if (target.type === "ref" && typeof target.ref === "string") return { ref: target.ref }; + if (target.type === "coordinate" && typeof target.x === "number" && typeof target.y === "number") { + return { x: target.x, y: target.y }; + } + } + throw new Error("invalid target: expected {type: \"ref\", ref} or {type: \"coordinate\", x, y}"); +} + +function refTarget(value: unknown): string { + const target = pageTarget(value); + if (target.ref === undefined) throw new Error("invalid target: this action requires a ref target"); + return target.ref; +} + +function coordinateTarget(value: unknown, field: string): { x: number; y: number } { + const target = pageTarget(value); + if (target.x === undefined || target.y === undefined) throw new Error(`invalid ${field}: expected a coordinate target`); + return { x: target.x, y: target.y }; +} + +function fillValue(value: unknown): string | number | boolean { + if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") return value; + throw new Error("invalid value: expected string, number, or boolean"); +} + +function tabId(input: NativeInput): { tab_id?: string } { + return typeof input.tab_id === "string" ? { tab_id: input.tab_id } : {}; +} diff --git a/packages/ai/src/providers/common.ts b/packages/ai/src/providers/common.ts index 31723d0f..3b667703 100644 --- a/packages/ai/src/providers/common.ts +++ b/packages/ai/src/providers/common.ts @@ -9,291 +9,50 @@ import { type TSchema, type Tool, } from "@earendil-works/pi-ai"; +import { CUA_DEFAULT_COMPUTER_ACTION_TYPES, cuaActionSchemaByType, type CuaAction, type CuaActionType } from "../actions/index"; +import { cuaToolDescriptionForAction, cuaToolNameForAction, defaultActionsForMode, schemaOptionsForMode, type CuaMode } from "../modes"; +import type { ResolvedCuaNativeTool } from "../native-tools"; import type { CuaModelRef, CuaProvider } from "../models"; -export const CUA_ACTION_TYPES = [ - "click", - "double_click", - "mouse_down", - "mouse_up", - "type", - "keypress", - "scroll", - "move", - "drag", - "wait", - "screenshot", - "goto", - "back", - "forward", - "url", - "cursor_position", -] as const; - -export type CuaActionType = (typeof CUA_ACTION_TYPES)[number]; +export * from "../actions/index"; +export * from "../modes"; +export * from "../native-tools"; /** - * Mouse buttons accepted by click, mouse_down, and mouse_up actions. The - * executor coerces anything outside this set to "left". + * The default computer-mode action set: every computer-plane action except `zoom`. + * The full canonical vocabulary is split by plane into + * {@link CUA_COMPUTER_ACTION_TYPES} and {@link CUA_BROWSER_ACTION_TYPES}. */ -export type CuaMouseButton = "left" | "right" | "middle" | "back" | "forward"; - -/** - * Mouse buttons accepted by drag actions. The executor coerces anything - * outside this set to "left". - */ -export type CuaDragMouseButton = "left" | "right" | "middle"; - -export interface CuaActionClick { - type: "click"; - x: number; - y: number; - button?: CuaMouseButton; - hold_keys?: string[]; -} - -export interface CuaActionDoubleClick { - type: "double_click"; - x: number; - y: number; - hold_keys?: string[]; -} - -export interface CuaActionMouseDown { - type: "mouse_down"; - x: number; - y: number; - button?: CuaMouseButton; - hold_keys?: string[]; -} - -export interface CuaActionMouseUp { - type: "mouse_up"; - x: number; - y: number; - button?: CuaMouseButton; - hold_keys?: string[]; -} - -export interface CuaActionTypeText { - type: "type"; - text: string; -} - -export interface CuaActionKeypress { - type: "keypress"; - keys: string[]; - duration?: number; -} - -export interface CuaActionScroll { - type: "scroll"; - x?: number; - y?: number; - scroll_x?: number; - scroll_y?: number; - hold_keys?: string[]; -} - -export interface CuaActionMove { - type: "move"; - x: number; - y: number; -} - -export interface CuaActionDrag { - type: "drag"; - path: Array<{ x: number; y: number }>; - button?: CuaDragMouseButton; - hold_keys?: string[]; -} - -export interface CuaActionWait { - type: "wait"; - ms?: number; -} - -export interface CuaActionScreenshot { - type: "screenshot"; -} - -export interface CuaActionGoto { - type: "goto"; - url: string; -} - -export interface CuaActionBack { - type: "back"; -} - -export interface CuaActionForward { - type: "forward"; -} - -export interface CuaActionUrl { - type: "url"; -} - -export interface CuaActionCursorPosition { - type: "cursor_position"; -} - -export type CuaAction = - | CuaActionClick - | CuaActionDoubleClick - | CuaActionMouseDown - | CuaActionMouseUp - | CuaActionTypeText - | CuaActionKeypress - | CuaActionScroll - | CuaActionMove - | CuaActionDrag - | CuaActionWait - | CuaActionScreenshot - | CuaActionGoto - | CuaActionBack - | CuaActionForward - | CuaActionUrl - | CuaActionCursorPosition; - -const PointSchema = Type.Object( - { - x: Type.Number(), - y: Type.Number(), - }, - { additionalProperties: false }, -); - -const CUA_ACTION_SCHEMA_BY_TYPE = { - click: Type.Object( - { - type: Type.Literal("click"), - x: Type.Number(), - y: Type.Number(), - button: Type.Optional(Type.String()), - hold_keys: Type.Optional(Type.Array(Type.String())), - }, - { additionalProperties: false }, - ), - double_click: Type.Object( - { - type: Type.Literal("double_click"), - x: Type.Number(), - y: Type.Number(), - hold_keys: Type.Optional(Type.Array(Type.String())), - }, - { additionalProperties: false }, - ), - mouse_down: Type.Object( - { - type: Type.Literal("mouse_down"), - x: Type.Number(), - y: Type.Number(), - button: Type.Optional(Type.String()), - hold_keys: Type.Optional(Type.Array(Type.String())), - }, - { additionalProperties: false }, - ), - mouse_up: Type.Object( - { - type: Type.Literal("mouse_up"), - x: Type.Number(), - y: Type.Number(), - button: Type.Optional(Type.String()), - hold_keys: Type.Optional(Type.Array(Type.String())), - }, - { additionalProperties: false }, - ), - type: Type.Object( - { - type: Type.Literal("type"), - text: Type.String(), - }, - { additionalProperties: false }, - ), - keypress: Type.Object( - { - type: Type.Literal("keypress"), - keys: Type.Array(Type.String()), - duration: Type.Optional(Type.Number()), - }, - { additionalProperties: false }, - ), - scroll: Type.Object( - { - type: Type.Literal("scroll"), - x: Type.Optional(Type.Number()), - y: Type.Optional(Type.Number()), - scroll_x: Type.Optional(Type.Number()), - scroll_y: Type.Optional(Type.Number()), - hold_keys: Type.Optional(Type.Array(Type.String())), - }, - { additionalProperties: false }, - ), - move: Type.Object( - { - type: Type.Literal("move"), - x: Type.Number(), - y: Type.Number(), - }, - { additionalProperties: false }, - ), - drag: Type.Object( - { - type: Type.Literal("drag"), - path: Type.Array(PointSchema, { minItems: 2 }), - button: Type.Optional(Type.String()), - hold_keys: Type.Optional(Type.Array(Type.String())), - }, - { additionalProperties: false }, - ), - wait: Type.Object( - { - type: Type.Literal("wait"), - ms: Type.Optional(Type.Number()), - }, - { additionalProperties: false }, - ), - screenshot: Type.Object({ type: Type.Literal("screenshot") }, { additionalProperties: false }), - goto: Type.Object( - { - type: Type.Literal("goto"), - url: Type.String(), - }, - { additionalProperties: false }, - ), - back: Type.Object({ type: Type.Literal("back") }, { additionalProperties: false }), - forward: Type.Object({ type: Type.Literal("forward") }, { additionalProperties: false }), - url: Type.Object({ type: Type.Literal("url") }, { additionalProperties: false }), - cursor_position: Type.Object({ type: Type.Literal("cursor_position") }, { additionalProperties: false }), -} satisfies Record; +export const CUA_ACTION_TYPES = CUA_DEFAULT_COMPUTER_ACTION_TYPES; type ObjectSchemaWithProperties = TSchema & { properties: Record }; -function createCuaActionArgumentSchema(action: CuaActionType): TSchema { - const { type: _type, ...properties } = (CUA_ACTION_SCHEMA_BY_TYPE[action] as ObjectSchemaWithProperties).properties; +function createCuaActionArgumentSchema(action: CuaActionType, mode: CuaMode): TSchema { + const schemaByType = cuaActionSchemaByType(schemaOptionsForMode(mode)); + const { type: _type, ...properties } = (schemaByType[action] as ObjectSchemaWithProperties).properties; return Type.Object(properties, { additionalProperties: false }); } -export function createCuaActionSchema(actions: readonly CuaActionType[] = CUA_ACTION_TYPES): TSchema { +export function createCuaActionSchema(actions: readonly CuaActionType[] = CUA_ACTION_TYPES, mode: CuaMode = "computer"): TSchema { if (actions.length === 0) throw new Error("actions must include at least one CUA action type"); - if (actions.length === 1) return CUA_ACTION_SCHEMA_BY_TYPE[actions[0]!]; - return Type.Union(actions.map((action) => CUA_ACTION_SCHEMA_BY_TYPE[action])); + const schemaByType = cuaActionSchemaByType(schemaOptionsForMode(mode)); + if (actions.length === 1) return schemaByType[actions[0]!]; + return Type.Union(actions.map((action) => schemaByType[action])); } -export function createCuaActionToolDefinitions(actions: readonly CuaActionType[] = CUA_ACTION_TYPES): Tool[] { +export function createCuaActionToolDefinitions(actions: readonly CuaActionType[] = CUA_ACTION_TYPES, mode: CuaMode = "computer"): Tool[] { return actions.map((action) => ({ - name: action, - description: `Execute one ${action} computer action.`, - parameters: createCuaActionArgumentSchema(action), + name: cuaToolNameForAction(action, mode), + description: cuaToolDescriptionForAction(action, mode), + parameters: createCuaActionArgumentSchema(action, mode), })); } export const CuaActionSchema = createCuaActionSchema(); -export function createCuaBatchSchema(actions?: readonly CuaActionType[]): TSchema { +export function createCuaBatchSchema(actions?: readonly CuaActionType[], mode: CuaMode = "computer"): TSchema { return Type.Object({ - actions: Type.Array(createCuaActionSchema(actions), { description: "Ordered computer actions to execute." }), + actions: Type.Array(createCuaActionSchema(actions, mode), { description: "Ordered computer actions to execute." }), }); } @@ -358,6 +117,8 @@ export const CUA_PLAYWRIGHT_TOOL_DESCRIPTION = [ export interface ComputerToolsOptions { actions?: readonly CuaActionType[]; + /** Which action plane(s) to expose. Default "computer". */ + mode?: CuaMode; } export type ComputerToolCoordinateSystem = @@ -377,13 +138,25 @@ export type ComputerToolCoordinateSystem = * smaller set, such as `["click"]`. */ export function computerTools(options: ComputerToolsOptions = {}): Tool[] { - return createCuaActionToolDefinitions(options.actions); + return createCuaActionToolDefinitions(resolveModeActions(options), options.mode ?? "computer"); +} + +/** Resolve the action list for a tools-options object: explicit list, or the mode's default set. */ +export function resolveModeActions(options: ComputerToolsOptions = {}): readonly CuaActionType[] { + return options.actions ?? defaultActionsForMode(options.mode ?? "computer"); +} + +/** Guard for providers whose computer-use vocabulary only covers the computer plane. */ +export function assertComputerModeOnly(provider: CuaProvider, options: ComputerToolsOptions = {}): void { + const mode = options.mode ?? "computer"; + if (mode !== "computer") throw new Error(`provider "${provider}" does not support mode "${mode}" (computer only)`); } /** Build execution adapters for individual canonical CUA action tools. */ -export function createCuaActionToolExecutors(actions: readonly CuaActionType[] = CUA_ACTION_TYPES): CuaToolExecutorSpec[] { - return createCuaActionToolDefinitions(actions).map((definition) => { - const actionType = definition.name as CuaActionType; +export function createCuaActionToolExecutors(actions: readonly CuaActionType[] = CUA_ACTION_TYPES, mode: CuaMode = "computer"): CuaToolExecutorSpec[] { + const definitions = createCuaActionToolDefinitions(actions, mode); + return definitions.map((definition, index) => { + const actionType = actions[index]!; return { definition, toActions(args: unknown): CuaAction[] { @@ -414,19 +187,19 @@ export function normalizeGotoUrl(value: unknown): string | undefined { export function createCuaBatchToolDefinition( actions?: readonly CuaActionType[], - options: { name?: string; description?: string } = {}, + options: { name?: string; description?: string; mode?: CuaMode } = {}, ): Tool { return { name: options.name ?? CUA_BATCH_TOOL_NAME, description: options.description ?? CUA_BATCH_TOOL_DESCRIPTION, - parameters: createCuaBatchSchema(actions), + parameters: createCuaBatchSchema(actions, options.mode ?? "computer"), }; } /** Build an execution adapter for a batch tool whose input is `{ actions }`. */ export function createCuaBatchToolExecutor( actions?: readonly CuaActionType[], - options: { name?: string; description?: string } = {}, + options: { name?: string; description?: string; mode?: CuaMode } = {}, ): CuaToolExecutorSpec { const definition = createCuaBatchToolDefinition(actions, options); return { @@ -440,7 +213,7 @@ export function createCuaBatchToolExecutor( /** Build the provider's default CUA tool execution adapters. */ export function computerToolExecutors(options: ComputerToolsOptions = {}): CuaToolExecutorSpec[] { - return createCuaActionToolExecutors(options.actions); + return createCuaActionToolExecutors(resolveModeActions(options), options.mode ?? "computer"); } function isBatchInput(value: unknown): value is CuaBatchInput { @@ -530,17 +303,19 @@ export interface ResponseThreadingDelta { * Anchors on the most recent assistant turn: returns its `responseId` and the * messages after it (the delta). An errored or aborted turn may carry a * `responseId` captured from an incomplete response the server never stored, so - * its id is ignored. When the anchor has no usable `responseId`, or there is no - * assistant turn yet, returns every message and no id so the caller replays the - * full history, rather than chaining to a phantom id and pruning past it. + * its id is ignored. A turn produced by a different `api` (e.g. after a + * mid-session `-m` provider switch) is ignored too — its id would be foreign to + * the current provider. When the anchor has no usable `responseId`, or there is + * no assistant turn yet, returns every message and no id so the caller replays + * the full history, rather than chaining to a phantom id and pruning past it. */ -export function responseThreadingDelta(messages: readonly Message[]): ResponseThreadingDelta { +export function responseThreadingDelta(messages: readonly Message[], api: Api): ResponseThreadingDelta { for (let index = messages.length - 1; index >= 0; index -= 1) { const message = messages[index]!; if (message.role !== "assistant") continue; const assistant = message as AssistantMessage; const failed = assistant.stopReason === "error" || assistant.stopReason === "aborted"; - const responseId = failed ? undefined : assistant.responseId; + const responseId = failed || assistant.api !== api ? undefined : assistant.responseId; return responseId ? { previousResponseId: responseId, deltaMessages: messages.slice(index + 1) } : { deltaMessages: [...messages] }; } return { deltaMessages: [...messages] }; @@ -556,6 +331,10 @@ export function responseThreadingDelta(messages: readonly Message[]): ResponseTh export interface CuaRuntimeSpec { model: Model; provider: CuaProvider; + /** Which canonical action plane(s) this runtime exposes. */ + mode: CuaMode; + /** Present when the model is driven through a provider-native tool declaration. */ + nativeTool?: ResolvedCuaNativeTool; /** Provider-facing CUA tool definitions used for model requests. */ toolDefinitions: Tool[]; /** Local execution adapters that turn provider tool calls into canonical CUA actions. */ @@ -581,7 +360,7 @@ export interface CuaProviderModule { /** Coordinate convention emitted by this provider's tool calls. */ coordinateSystem(): ComputerToolCoordinateSystem; /** Provider-tuned baseline browser-control system prompt. */ - buildSystemPrompt(opts?: { suffix?: string }): string; + buildSystemPrompt(opts?: { suffix?: string; mode?: CuaMode }): string; /** Optional request-payload middleware for provider protocol quirks. */ onPayload?: CuaPayloadHook; /** Optional provider screenshot input policy. */ diff --git a/packages/ai/src/providers/gemini/index.ts b/packages/ai/src/providers/gemini/index.ts index 1a337c7e..87875f13 100644 --- a/packages/ai/src/providers/gemini/index.ts +++ b/packages/ai/src/providers/gemini/index.ts @@ -1,5 +1,5 @@ -import { computerToolExecutors, computerTools } from "../common"; -import type { ComputerToolCoordinateSystem, CuaProviderModule } from "../common"; +import { assertComputerModeOnly, computerToolExecutors, computerTools } from "../common"; +import type { ComputerToolCoordinateSystem, ComputerToolsOptions, CuaProviderModule } from "../common"; export { CUA_ACTION_TYPES as GEMINI_CUA_ACTION_TYPES, @@ -30,8 +30,17 @@ export function buildGeminiSystemPrompt(opts: { suffix?: string } = {}): string } export const providerModule = { - toolDefinitions: computerTools, - toolExecutors: computerToolExecutors, + // Gemini's computer-use coordinate convention is normalized 0-999, which + // only maps onto the computer plane today; browser-plane viewport coordinates are + // unvalidated for it. + toolDefinitions: (options?: ComputerToolsOptions) => { + assertComputerModeOnly("google", options); + return computerTools(options); + }, + toolExecutors: (options?: ComputerToolsOptions) => { + assertComputerModeOnly("google", options); + return computerToolExecutors(options); + }, coordinateSystem, buildSystemPrompt: buildGeminiSystemPrompt, } satisfies CuaProviderModule; diff --git a/packages/ai/src/providers/openai/index.ts b/packages/ai/src/providers/openai/index.ts index 21c293a8..324bbcde 100644 --- a/packages/ai/src/providers/openai/index.ts +++ b/packages/ai/src/providers/openai/index.ts @@ -1,4 +1,4 @@ -import type { ComputerToolCoordinateSystem, CuaProviderModule } from "../common"; +import type { ComputerToolCoordinateSystem, CuaMode, CuaProviderModule } from "../common"; import { computerToolExecutors, computerTools } from "../common"; export { @@ -31,8 +31,14 @@ export function coordinateSystem(): ComputerToolCoordinateSystem { export const OPENAI_COMPUTER_INSTRUCTIONS = `You control a Kernel cloud browser through individual browser tools. Use the available tools for browser interaction and request explicit url, cursor_position, or screenshot reads when you need updated state.`; -export function buildOpenAISystemPrompt(opts: { suffix?: string } = {}): string { - return [OPENAI_COMPUTER_INSTRUCTIONS, opts.suffix].filter(Boolean).join("\n\n"); +export const OPENAI_BROWSER_INSTRUCTIONS = `You control a Kernel cloud browser through page tools. Prefer reading the page with snapshot or find and targeting elements by reference; use screenshots when you need to inspect visual state. Element references go stale when the page changes — re-snapshot when told so.`; + +export const OPENAI_HYBRID_INSTRUCTIONS = `You control a Kernel cloud browser through two kinds of tools: computer_* tools perform real OS-level input (coordinates are pixels in the most recent computer_screenshot), and browser_* tools read and act on the page itself by element reference. Prefer browser_snapshot/browser_find for reading and locating, and computer_* input for interaction; use browser_* interaction for elements that are hard to hit by coordinate.`; + +export function buildOpenAISystemPrompt(opts: { suffix?: string; mode?: CuaMode } = {}): string { + const base = + opts.mode === "browser" ? OPENAI_BROWSER_INSTRUCTIONS : opts.mode === "hybrid" ? OPENAI_HYBRID_INSTRUCTIONS : OPENAI_COMPUTER_INSTRUCTIONS; + return [base, opts.suffix].filter(Boolean).join("\n\n"); } export const providerModule = { diff --git a/packages/ai/src/providers/openai/provider.ts b/packages/ai/src/providers/openai/provider.ts index cb8bc75d..bca616a3 100644 --- a/packages/ai/src/providers/openai/provider.ts +++ b/packages/ai/src/providers/openai/provider.ts @@ -29,7 +29,7 @@ export function threadRequest( context: Context, options: (ResponseThreadingOptions & { onPayload?: OnPayload }) | undefined, ): { context: Context; onPayload: OnPayload } { - const delta = responseThreadingEnabled(options) ? responseThreadingDelta(context.messages) : undefined; + const delta = responseThreadingEnabled(options) ? responseThreadingDelta(context.messages, OPENAI_CUA_RESPONSES_API) : undefined; const previousResponseId = delta?.previousResponseId; const messages = previousResponseId && delta ? delta.deltaMessages : context.messages; const onPayload: OnPayload = async (payload, model) => { diff --git a/packages/ai/src/providers/tzafon/index.ts b/packages/ai/src/providers/tzafon/index.ts index 4a5e029e..61065218 100644 --- a/packages/ai/src/providers/tzafon/index.ts +++ b/packages/ai/src/providers/tzafon/index.ts @@ -1,4 +1,11 @@ -import { computerToolExecutors, computerTools, type ComputerToolCoordinateSystem, type CuaProviderModule } from "../common"; +import { + assertComputerModeOnly, + computerToolExecutors, + computerTools, + type ComputerToolCoordinateSystem, + type ComputerToolsOptions, + type CuaProviderModule, +} from "../common"; import { tzafonComputerUseOnPayload } from "./provider"; export { @@ -44,8 +51,14 @@ export function buildTzafonSystemPrompt(opts: { suffix?: string } = {}): string } export const providerModule = { - toolDefinitions: computerTools, - toolExecutors: computerToolExecutors, + toolDefinitions: (options?: ComputerToolsOptions) => { + assertComputerModeOnly("tzafon", options); + return computerTools(options); + }, + toolExecutors: (options?: ComputerToolsOptions) => { + assertComputerModeOnly("tzafon", options); + return computerToolExecutors(options); + }, coordinateSystem, buildSystemPrompt: buildTzafonSystemPrompt, onPayload: tzafonComputerUseOnPayload, diff --git a/packages/ai/src/providers/tzafon/provider.ts b/packages/ai/src/providers/tzafon/provider.ts index db64cc90..95f31de9 100644 --- a/packages/ai/src/providers/tzafon/provider.ts +++ b/packages/ai/src/providers/tzafon/provider.ts @@ -76,7 +76,7 @@ export function buildTzafonRequestInput(model: Model, context: Context, opt max_output_tokens: options?.maxTokens ?? model.maxTokens, }; if (!responseThreadingEnabled(options)) return body; - const { previousResponseId, deltaMessages } = responseThreadingDelta(context.messages); + const { previousResponseId, deltaMessages } = responseThreadingDelta(context.messages, TZAFON_RESPONSES_API); if (!previousResponseId) return body; return { ...body, input: convertMessages(deltaMessages), previous_response_id: previousResponseId, store: true }; } diff --git a/packages/ai/src/providers/yutori/index.ts b/packages/ai/src/providers/yutori/index.ts index da05b4c3..1fa91f5d 100644 --- a/packages/ai/src/providers/yutori/index.ts +++ b/packages/ai/src/providers/yutori/index.ts @@ -1,4 +1,4 @@ -import type { ComputerToolCoordinateSystem, CuaProviderModule } from "../common"; +import { assertComputerModeOnly, type ComputerToolCoordinateSystem, type ComputerToolsOptions, type CuaProviderModule } from "../common"; import { computerToolExecutors } from "./actions"; import { yutoriCuaOnPayload } from "./provider"; @@ -63,8 +63,14 @@ export function buildYutoriSystemPrompt(opts: { suffix?: string } = {}): string } export const providerModule = { - toolDefinitions: () => [], - toolExecutors: computerToolExecutors, + toolDefinitions: (options?: ComputerToolsOptions) => { + assertComputerModeOnly("yutori", options); + return []; + }, + toolExecutors: (options?: ComputerToolsOptions) => { + assertComputerModeOnly("yutori", options); + return computerToolExecutors(options); + }, coordinateSystem, buildSystemPrompt: buildYutoriSystemPrompt, onPayload: yutoriCuaOnPayload, diff --git a/packages/ai/src/runtime-spec.ts b/packages/ai/src/runtime-spec.ts index 178de193..e8fbee66 100644 --- a/packages/ai/src/runtime-spec.ts +++ b/packages/ai/src/runtime-spec.ts @@ -1,12 +1,16 @@ +import type { Api, Model } from "@earendil-works/pi-ai"; import type { CuaProvider } from "./models"; import { getCuaModel, providerForModel, routeCuaApi } from "./models"; +import { modeForNativeTool, resolveNativeTool, type CuaNativeToolSpec } from "./native-tools"; import { providerModule as anthropic } from "./providers/anthropic/index"; +import { createNativeToolOnPayload, nativeApiForToolType, nativeToolExecutors } from "./providers/anthropic/native"; import { providerModule as gemini } from "./providers/gemini/index"; import { providerModule as openai } from "./providers/openai/index"; import { providerModule as tzafon } from "./providers/tzafon/index"; import { providerModule as yutori } from "./providers/yutori/index"; import type { ComputerToolsOptions, + CuaPayloadHook, CuaProviderModule, CuaRuntimeSpec, CuaRuntimeSpecInput, @@ -20,26 +24,76 @@ const PROVIDERS = { yutori, } satisfies Record; +/** Options accepted by {@link resolveCuaRuntimeSpec}. */ +export interface CuaRuntimeSpecOptions extends ComputerToolsOptions { + /** + * Drive the model through a provider-native tool declaration instead of + * CUA's canonical function tools. The native tool determines (and is + * validated against) the mode: `computer_20260701` requires `"computer"`, + * `browser_20260701` requires `"browser"`. When `mode` is omitted it is + * inferred from the native tool. + */ + nativeTool?: CuaNativeToolSpec; +} + /** * Resolve provider defaults from either a CUA model ref or a concrete model. * * Use the returned spec to build computer-use requests without hard-coding - * model-provider rules in your application. Pass `options` (e.g. - * `{ actions: ["click"] }`) to narrow the resolved tool definitions and - * executors to a supported subset. + * model-provider rules in your application. Pass `options` to select the + * action plane(s) (`mode`), narrow the exposed actions (`actions`), or drive + * an Anthropic model through its native tool schema (`nativeTool`). */ -export function resolveCuaRuntimeSpec(input: CuaRuntimeSpecInput, options?: ComputerToolsOptions): CuaRuntimeSpec { +export function resolveCuaRuntimeSpec(input: CuaRuntimeSpecInput, options: CuaRuntimeSpecOptions = {}): CuaRuntimeSpec { const model = typeof input === "string" ? getCuaModel(input) : routeCuaApi(input); const provider = providerForModel(model); const mod: CuaProviderModule = PROVIDERS[provider]; + const mode = options.mode ?? (options.nativeTool ? modeForNativeTool(options.nativeTool) : "computer"); + + if (options.nativeTool) { + const nativeTool = resolveNativeTool(withDefaultJavascriptExec(options.nativeTool), model, mode); + const nativeModel: Model = { ...model, api: nativeApiForToolType(nativeTool.spec.type) as Model["api"] }; + const executors = nativeToolExecutors(nativeTool); + return { + model: nativeModel, + provider, + mode, + nativeTool, + toolDefinitions: executors.map((executor) => executor.definition), + toolExecutors: executors, + defaultSystemPrompt: mod.buildSystemPrompt({ mode }), + coordinateSystem: mod.coordinateSystem(), + screenshot: mod.screenshot, + onPayload: composePayloadHooks(createNativeToolOnPayload(nativeTool), mod.onPayload), + }; + } + + const toolsOptions: ComputerToolsOptions = { ...options, mode }; return { model, provider, - toolDefinitions: mod.toolDefinitions(options), - toolExecutors: mod.toolExecutors(options), - defaultSystemPrompt: mod.buildSystemPrompt(), + mode, + toolDefinitions: mod.toolDefinitions(toolsOptions), + toolExecutors: mod.toolExecutors(toolsOptions), + defaultSystemPrompt: mod.buildSystemPrompt({ mode }), coordinateSystem: mod.coordinateSystem(), screenshot: mod.screenshot, onPayload: mod.onPayload, }; } + +// JavaScript execution is on by default, matching canonical browser mode +// where `browser_evaluate` is part of the default toolset. An explicit +// enable_javascript_exec on the spec wins. +function withDefaultJavascriptExec(spec: CuaNativeToolSpec): CuaNativeToolSpec { + if (spec.type !== "browser_20260701" || spec.enable_javascript_exec !== undefined) return spec; + return { ...spec, enable_javascript_exec: true }; +} + +function composePayloadHooks(first: CuaPayloadHook, second: CuaPayloadHook | undefined): CuaPayloadHook { + if (!second) return first; + return async (payload, model, context) => { + const afterFirst = (await first(payload, model, context)) ?? payload; + return (await second(afterFirst, model, context)) ?? afterFirst; + }; +} diff --git a/packages/ai/test/modes.test.ts b/packages/ai/test/modes.test.ts new file mode 100644 index 00000000..d9b2f030 --- /dev/null +++ b/packages/ai/test/modes.test.ts @@ -0,0 +1,118 @@ +import { describe, expect, it } from "vitest"; +import { + CUA_ACTION_TYPES, + CUA_BROWSER_ACTION_TYPES, + CUA_HYBRID_BROWSER_ACTION_TYPES, + CUA_HYBRID_COMPUTER_ACTION_TYPES, + anthropic, + computerTools, + cuaToolNameForAction, + defaultActionsForMode, + openai, + resolveCuaRuntimeSpec, +} from "../src/index"; + +describe("mode action sets", () => { + it("computer mode defaults to the legacy action set", () => { + expect(defaultActionsForMode("computer")).toEqual(CUA_ACTION_TYPES); + }); + + it("browser mode defaults to every browser action plus wait", () => { + const actions = defaultActionsForMode("browser"); + expect(actions).toContain("browser_snapshot"); + expect(actions).toContain("wait"); + expect(actions).toContain("browser_evaluate"); + expect(actions).not.toContain("click"); + }); + + it("hybrid mode dedupes to one tool per capability", () => { + const actions = defaultActionsForMode("hybrid"); + // Navigation lives on the browser plane. + expect(actions).not.toContain("goto"); + expect(actions).not.toContain("url"); + expect(actions).toContain("browser_navigate"); + // One screenshot: the OS display. + expect(actions).toContain("screenshot"); + expect(actions).toContain("zoom"); + expect(actions).not.toContain("browser_screenshot"); + // Pointer/keyboard stays OS-level. + expect(actions).not.toContain("browser_type"); + expect(actions).not.toContain("browser_scroll"); + expect(actions).toEqual([...CUA_HYBRID_COMPUTER_ACTION_TYPES, ...CUA_HYBRID_BROWSER_ACTION_TYPES]); + }); +}); + +describe("mode tool naming", () => { + it("computer mode keeps canonical action ids", () => { + expect(cuaToolNameForAction("click", "computer")).toBe("click"); + }); + + it("browser mode strips the browser_ prefix", () => { + expect(cuaToolNameForAction("browser_snapshot", "browser")).toBe("snapshot"); + expect(cuaToolNameForAction("browser_click", "browser")).toBe("click"); + expect(cuaToolNameForAction("wait", "browser")).toBe("wait"); + }); + + it("hybrid mode prefixes computer actions and keeps browser_ names", () => { + expect(cuaToolNameForAction("click", "hybrid")).toBe("computer_click"); + expect(cuaToolNameForAction("browser_click", "hybrid")).toBe("browser_click"); + }); + + it("computer mode rejects browser actions", () => { + expect(() => cuaToolNameForAction("browser_click", "computer")).toThrow(/not available in computer mode/); + }); +}); + +describe("mode tool schemas", () => { + it("browser mode click accepts refs or viewport coordinates", () => { + const tools = computerTools({ mode: "browser" }); + const click = tools.find((tool) => tool.name === "click")!; + expect(click.parameters.properties.ref).toBeDefined(); + expect(click.parameters.properties.x).toBeDefined(); + }); + + it("hybrid mode browser_click is ref-only, keeping one coordinate frame", () => { + const tools = computerTools({ mode: "hybrid" }); + const pageClick = tools.find((tool) => tool.name === "browser_click")!; + expect(pageClick.parameters.properties.ref).toBeDefined(); + expect(pageClick.parameters.properties.x).toBeUndefined(); + expect(pageClick.parameters.required).toContain("ref"); + }); + + it("browser mode exposes every default browser action under its unprefixed name", () => { + const tools = computerTools({ mode: "browser" }); + const names = tools.map((tool) => tool.name); + for (const action of CUA_BROWSER_ACTION_TYPES) { + expect(names).toContain(action.slice("browser_".length)); + } + }); +}); + +describe("mode runtime specs", () => { + it("resolves browser mode for anthropic with mode-specific prompt and tools", () => { + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "browser" }); + expect(spec.mode).toBe("browser"); + expect(spec.toolDefinitions.map((tool) => tool.name)).toContain("snapshot"); + expect(spec.defaultSystemPrompt).toBe(anthropic.buildAnthropicSystemPrompt({ mode: "browser" })); + }); + + it("resolves hybrid mode for openai", () => { + const spec = resolveCuaRuntimeSpec("openai:gpt-5.5", { mode: "hybrid" }); + expect(spec.mode).toBe("hybrid"); + const names = spec.toolDefinitions.map((tool) => tool.name); + expect(names).toContain("computer_click"); + expect(names).toContain("browser_snapshot"); + expect(spec.defaultSystemPrompt).toBe(openai.buildOpenAISystemPrompt({ mode: "hybrid" })); + }); + + it("rejects non-computer modes for computer-only providers", () => { + expect(() => resolveCuaRuntimeSpec("yutori:n1.5-latest", { mode: "browser" })).toThrow(/computer only/); + expect(() => resolveCuaRuntimeSpec("google:gemini-3-flash-preview", { mode: "hybrid" })).toThrow(/computer only/); + }); + + it("keeps computer mode byte-compatible with the pre-modes default", () => { + const before = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5"); + const after = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "computer" }); + expect(after.toolDefinitions.map((tool) => tool.name)).toEqual(before.toolDefinitions.map((tool) => tool.name)); + }); +}); diff --git a/packages/ai/test/native-tools.test.ts b/packages/ai/test/native-tools.test.ts new file mode 100644 index 00000000..dd118e92 --- /dev/null +++ b/packages/ai/test/native-tools.test.ts @@ -0,0 +1,181 @@ +import { describe, expect, it } from "vitest"; +import { + anthropic, + betaHeaderForNativeTool, + modeForNativeTool, + resolveCuaRuntimeSpec, + type CuaAction, +} from "../src/index"; +import { + ANTHROPIC_NATIVE_BROWSER_MESSAGES_API, + ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API, + mapNativeBrowserInput, + mapNativeComputerInput, +} from "../src/providers/anthropic/native"; + +describe("native tool validation", () => { + it("infers mode from the native tool", () => { + expect(modeForNativeTool({ type: "computer_20260701" })).toBe("computer"); + expect(modeForNativeTool({ type: "browser_20260701" })).toBe("browser"); + }); + + it("carries the beta header per tool", () => { + expect(betaHeaderForNativeTool({ type: "computer_20260701" })).toBe("computer-use-2026-07-01"); + expect(betaHeaderForNativeTool({ type: "browser_20260701" })).toBe("browser-use-2026-07-01"); + }); + + it("rejects a native tool with a conflicting mode", () => { + expect(() => resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "browser", nativeTool: { type: "computer_20260701" } })).toThrow( + /requires mode "computer"/, + ); + expect(() => + resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "hybrid", nativeTool: { type: "browser_20260701" } }), + ).toThrow(/requires mode "browser"/); + }); + + it("rejects native tools on non-anthropic models", () => { + expect(() => resolveCuaRuntimeSpec("openai:gpt-5.5", { nativeTool: { type: "computer_20260701" } })).toThrow( + /requires an anthropic model paired with mode "computer"/, + ); + }); +}); + +describe("native runtime specs", () => { + it("routes computer_20260701 to the native api with a single placeholder tool", () => { + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "computer_20260701", enable_zoom: true } }); + expect(spec.mode).toBe("computer"); + expect(spec.model.api).toBe(ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API); + expect(spec.nativeTool?.betaHeader).toBe("computer-use-2026-07-01"); + expect(spec.toolDefinitions.map((tool) => tool.name)).toEqual(["computer"]); + }); + + it("routes browser_20260701 to the native api under the default name", () => { + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "browser_20260701" } }); + expect(spec.mode).toBe("browser"); + expect(spec.model.api).toBe(ANTHROPIC_NATIVE_BROWSER_MESSAGES_API); + expect(spec.toolDefinitions.map((tool) => tool.name)).toEqual(["browser"]); + }); + + it("enables javascript exec on the native browser declaration unless the spec is explicit", () => { + const defaulted = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { + nativeTool: { type: "browser_20260701" }, + }); + expect(defaulted.nativeTool?.declaration.enable_javascript_exec).toBe(true); + + const explicit = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { + nativeTool: { type: "browser_20260701", enable_javascript_exec: false }, + }); + expect(explicit.nativeTool?.declaration.enable_javascript_exec).toBe(false); + }); + + it("swaps the placeholder tool for the native declaration in the payload", async () => { + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "computer_20260701", enable_zoom: true } }); + const payload = { + tools: [ + { name: "computer", description: "placeholder", input_schema: {} }, + { name: "playwright_execute", description: "keep me", input_schema: {} }, + ], + }; + const next = (await spec.onPayload?.(payload, spec.model)) as { tools: Array> }; + expect(next.tools[0]).toEqual({ type: "computer_20260701", name: "computer", enable_zoom: true }); + expect(next.tools[1]!.name).toBe("playwright_execute"); + }); +}); + +describe("computer_20260701 action mapping", () => { + it("maps clicks with coordinates, buttons, and modifier chords", () => { + expect(mapNativeComputerInput({ action: "left_click", coordinate: [10, 20], text: "ctrl+shift" })).toEqual([ + { type: "click", x: 10, y: 20, button: "left", hold_keys: ["ctrl", "shift"] }, + ]); + expect(mapNativeComputerInput({ action: "triple_click", coordinate: [1, 2] })).toEqual([ + { type: "click", x: 1, y: 2, button: "left", num_clicks: 3 }, + ]); + }); + + it("maps clicks without coordinates to the current cursor position", () => { + expect(mapNativeComputerInput({ action: "left_click" })).toEqual([{ type: "click", button: "left" }]); + }); + + it("maps hold_key seconds to keypress duration in milliseconds", () => { + expect(mapNativeComputerInput({ action: "hold_key", text: "a", duration: 2 })).toEqual([ + { type: "keypress", keys: ["a"], duration: 2000 }, + ]); + }); + + it("expands key repeat into repeated keypresses", () => { + expect(mapNativeComputerInput({ action: "key", text: "Down", repeat: 3 })).toEqual([ + { type: "keypress", keys: ["Down"] }, + { type: "keypress", keys: ["Down"] }, + { type: "keypress", keys: ["Down"] }, + ]); + }); + + it("maps scroll direction and wheel notches to deltas", () => { + expect(mapNativeComputerInput({ action: "scroll", coordinate: [5, 6], scroll_direction: "down", scroll_amount: 2 })).toEqual([ + { type: "scroll", x: 5, y: 6, scroll_y: 240 }, + ]); + }); + + it("maps drag, wait, and zoom", () => { + expect(mapNativeComputerInput({ action: "left_click_drag", start_coordinate: [1, 2], coordinate: [3, 4] })).toEqual([ + { type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] }, + ]); + expect(mapNativeComputerInput({ action: "wait", duration: 2 })).toEqual([{ type: "wait", ms: 2000 }]); + expect(mapNativeComputerInput({ action: "zoom", region: [0, 0, 10, 10] })).toEqual([{ type: "zoom", region: [0, 0, 10, 10] }]); + }); + + it("rejects unknown actions", () => { + expect(() => mapNativeComputerInput({ action: "warp" })).toThrow(/unsupported computer_20260701 action/); + }); +}); + +describe("browser_20260701 action mapping", () => { + it("maps browser reads", () => { + expect(mapNativeBrowserInput({ action: "read_page", filter: "interactive", depth: 5 })).toEqual([ + { type: "browser_snapshot", filter: "interactive", depth: 5 }, + ]); + expect(mapNativeBrowserInput({ action: "find", query: "search bar" })).toEqual([{ type: "browser_find", query: "search bar" }]); + expect(mapNativeBrowserInput({ action: "get_page_text", tab_id: "T1" })).toEqual([{ type: "browser_text", tab_id: "T1" }]); + }); + + it("maps ref and coordinate click targets", () => { + expect(mapNativeBrowserInput({ action: "left_click", target: { type: "ref", ref: "e7" } })).toEqual([ + { type: "browser_click", ref: "e7" }, + ]); + expect(mapNativeBrowserInput({ action: "left_click", target: { type: "coordinate", x: 4, y: 5 }, modifiers: "shift" })).toEqual([ + { type: "browser_click", x: 4, y: 5, modifiers: ["shift"] }, + ]); + }); + + it("requires ref targets on form_input and scroll_to", () => { + expect(mapNativeBrowserInput({ action: "form_input", target: { type: "ref", ref: "e7" }, value: "hi" })).toEqual([ + { type: "browser_fill", ref: "e7", value: "hi" }, + ]); + expect(() => mapNativeBrowserInput({ action: "scroll_to", target: { type: "coordinate", x: 1, y: 2 } })).toThrow(/requires a ref/); + }); + + it("maps navigation, tabs, zoom, and javascript_exec", () => { + expect(mapNativeBrowserInput({ action: "navigate", url: "back" })).toEqual([{ type: "browser_navigate", url: "back" }]); + expect(mapNativeBrowserInput({ action: "list_tabs" })).toEqual([{ type: "browser_list_tabs" }]); + expect(mapNativeBrowserInput({ action: "zoom", region: [1, 2, 3, 4] })).toEqual([ + { type: "browser_screenshot", region: [1, 2, 3, 4] }, + ]); + expect(mapNativeBrowserInput({ action: "javascript_exec", text: "document.title" })).toEqual([ + { type: "browser_evaluate", code: "document.title" }, + ]); + }); +}); + +describe("native tool executors", () => { + it("translate native tool calls through the runtime spec executors", () => { + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "browser_20260701" } }); + const executor = spec.toolExecutors[0]!; + const actions: CuaAction[] = executor.toActions({ action: "left_click", target: { type: "ref", ref: "e3" } }); + expect(actions).toEqual([{ type: "browser_click", ref: "e3" }]); + }); + + it("exports the anthropic namespace surface", () => { + expect(anthropic.mapNativeComputerInput).toBeTypeOf("function"); + expect(anthropic.ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API).toBe(ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API); + }); +}); diff --git a/packages/ai/test/openai-threading.test.ts b/packages/ai/test/openai-threading.test.ts index 7ba25602..d4821d06 100644 --- a/packages/ai/test/openai-threading.test.ts +++ b/packages/ai/test/openai-threading.test.ts @@ -96,6 +96,25 @@ describe("openai threadRequest", () => { expect(((await onPayload({}, model)) as Record).previous_response_id).toBeUndefined(); }); + it("never anchors previous_response_id on an assistant turn from a different api", async () => { + const ctx = multiTurnContext(); + // A mid-session -m provider switch leaves the prior provider's turn (and its foreign id) as the anchor. + ctx.messages.push({ + role: "assistant", + content: [{ type: "text", text: "done" }], + api: "anthropic-messages", + provider: "anthropic", + model: "claude-opus-4-8", + responseId: "msg_anthropic", + usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, + stopReason: "stop", + timestamp: 0, + }); + const { context, onPayload } = threadRequest(ctx, undefined); + expect(context).toBe(ctx); + expect(((await onPayload({}, model)) as Record).previous_response_id).toBeUndefined(); + }); + it("composes a caller onPayload on top of the threaded payload", async () => { const { onPayload } = threadRequest(multiTurnContext(), { onPayload: (payload) => ({ wrapped: payload }), diff --git a/packages/ai/test/provider-module.test.ts b/packages/ai/test/provider-module.test.ts index 17b17a16..88e81af8 100644 --- a/packages/ai/test/provider-module.test.ts +++ b/packages/ai/test/provider-module.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { anthropic, CUA_ACTION_TYPES, CUA_PROVIDERS, type CuaProvider, gemini, openai, tzafon, yutori } from "../src/index"; +import { anthropic, CUA_COMPUTER_ACTION_TYPES, CUA_PROVIDERS, type CuaProvider, gemini, openai, tzafon, yutori } from "../src/index"; import type { CuaProviderModule } from "../src/providers/common"; const MODULES: Record = { @@ -62,7 +62,7 @@ describe("provider namespaces export a uniform surface", () => { expect(Array.isArray(actionTypes), `${prefix}_CUA_ACTION_TYPES must be exported`).toBe(true); expect((actionTypes as unknown[]).length).toBeGreaterThan(0); for (const action of actionTypes as string[]) { - expect(CUA_ACTION_TYPES).toContain(action); + expect(CUA_COMPUTER_ACTION_TYPES).toContain(action); } expect(namespace[`${prefix}_COMPUTER_INSTRUCTIONS`], `${prefix}_COMPUTER_INSTRUCTIONS must be exported`).toBeTypeOf( diff --git a/packages/ai/test/tzafon-threading.test.ts b/packages/ai/test/tzafon-threading.test.ts index e7ba1f15..959b55c5 100644 --- a/packages/ai/test/tzafon-threading.test.ts +++ b/packages/ai/test/tzafon-threading.test.ts @@ -115,6 +115,25 @@ describe("buildTzafonRequestInput response threading", () => { expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); }); + it("replays full history when the latest assistant turn is from a different api", () => { + const context = multiTurnContext(); + context.messages.push({ + role: "assistant", + content: [{ type: "text", text: "done" }], + api: "anthropic-messages", + provider: "anthropic", + model: "claude-opus-4-8", + responseId: "msg_anthropic", + usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, + stopReason: "stop", + timestamp: 0, + }); + + const body = tzafon.buildTzafonRequestInput(model, context); + expect(body.previous_response_id).toBeUndefined(); + expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); + }); + // Off-path screenshot count scales with turn count; on-path stays constant at one. it("grows the payload per turn when off but stays flat when on", () => { const counts = (turns: number, disable: boolean) => { diff --git a/packages/cli/package.json b/packages/cli/package.json index ae251bb8..f8db9380 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "@onkernel/cua-cli", - "version": "0.2.0", + "version": "0.3.0", "description": "Kernel-cloud-browser computer-use TUI built on @onkernel/cua-agent and pi-tui", "license": "MIT", "type": "module", @@ -36,8 +36,8 @@ "dependencies": { "@earendil-works/pi-coding-agent": "0.80.3", "@earendil-works/pi-tui": "0.80.3", - "@onkernel/cua-agent": "0.4.0", - "@onkernel/cua-ai": "0.4.0", + "@onkernel/cua-agent": "0.5.0", + "@onkernel/cua-ai": "0.5.0", "@onkernel/sdk": "0.49.0" }, "devDependencies": { diff --git a/packages/cli/src/action/harness-runner.ts b/packages/cli/src/action/harness-runner.ts index da1f68da..bd883a8a 100644 --- a/packages/cli/src/action/harness-runner.ts +++ b/packages/cli/src/action/harness-runner.ts @@ -1,6 +1,5 @@ import type { AgentHarnessEvent, CuaAgentHarness, Session } from "@onkernel/cua-agent"; import type { AssistantMessage, ImageContent } from "@onkernel/cua-ai"; -import { writeFile } from "node:fs/promises"; import { stderr, stdout } from "node:process"; import { captureScreenshot, type CuaBrowserHandle } from "../harness-browser"; import { type ActionRequest, buildPrompt, DEFAULT_MAX_TURNS } from "./prompts"; @@ -15,53 +14,22 @@ export interface HarnessRunOptions { maxTurns?: number; } -export interface ScreenshotOutput { - out: string; // path or "-" for stdout -} - export interface RunActionResult { result: ActionResult; exitCode: number; } /** - * Run a single action subcommand against an existing harness + browser and - * return the parsed result plus exit code. The `screenshot` action is - * model-free — it captures directly through the SDK. All other actions - * drive the harness for at most `maxTurns` turns. + * Run a single model-mediated action subcommand against an existing + * harness + browser and return the parsed result plus exit code. Drives + * the harness for at most `maxTurns` turns. */ export async function runAction( req: ActionRequest, opts: HarnessRunOptions, - screenshot?: ScreenshotOutput, ): Promise { const startedAt = Date.now(); - if (req.action === "screenshot") { - const out = screenshot ?? { out: "screenshot.png" }; - const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id); - if (!png) { - const elapsed = Date.now() - startedAt; - const result: ActionResult = { - action: "screenshot", - status: "error", - text: "failed to capture screenshot", - elapsedMs: elapsed, - timestamp: Date.now(), - }; - return { result, exitCode: exitCodeFor(result) }; - } - if (out.out === "-") { - stdout.write(png); - } else { - await writeFile(out.out, png); - } - const elapsed = Date.now() - startedAt; - const result = parseResult("screenshot", "", [], elapsed); - result.text = out.out === "-" ? "(stdout)" : out.out; - return { result, exitCode: 0 }; - } - const prompt = buildPrompt(req); const maxTurns = req.maxTurns ?? opts.maxTurns ?? DEFAULT_MAX_TURNS; diff --git a/packages/cli/src/action/prompts.ts b/packages/cli/src/action/prompts.ts index 6772c0b8..4776f306 100644 --- a/packages/cli/src/action/prompts.ts +++ b/packages/cli/src/action/prompts.ts @@ -2,21 +2,12 @@ * Constrained one-shot prompts for the agent-friendly CLI subcommands. */ -export type ActionType = - | "click" - | "type" - | "open" - | "press" - | "screenshot" - | "url" - | "observe" - | "do"; +export type ModelActionType = "click" | "type" | "observe" | "do"; export interface ActionRequest { - action: ActionType; + action: ModelActionType; target?: string; text?: string; - keys?: string[]; maxTurns?: number; } @@ -31,26 +22,14 @@ export function buildPrompt(req: ActionRequest): string { if (!req.target) throw new Error("type action requires a target description"); if (!req.text) throw new Error("type action requires text to type"); return typePrompt(req.target, req.text); - case "open": { - const url = req.text || req.target; - if (!url) throw new Error("open action requires a URL"); - return openPrompt(url); - } - case "press": - if (!req.keys || req.keys.length === 0) throw new Error("press action requires at least one key"); - return pressPrompt(req.keys); case "observe": if (req.text) return observeWithQuestionPrompt(req.text); return observePrompt(); - case "url": - return urlPrompt(); case "do": { const instruction = req.text || req.target; if (!instruction) throw new Error("do action requires an instruction"); return instruction; } - case "screenshot": - throw new Error("screenshot action does not use a prompt"); } } @@ -69,16 +48,6 @@ If no matching element is visible on screen, respond with the text: NOT_FOUND: f Do not perform any other actions.`; } -function openPrompt(url: string): string { - return `Navigate the browser to this URL: ${url} -Use the goto action. Perform only this navigation, then stop.`; -} - -function pressPrompt(keys: string[]): string { - return `Press the following key(s): ${keys.join("+")} -Perform exactly this key press, then stop. Do not perform any other actions.`; -} - function observePrompt(): string { return `Look at the current screen and describe what you see. Be concise and factual. Do NOT perform any actions. Only observe and describe.`; @@ -88,7 +57,3 @@ function observeWithQuestionPrompt(question: string): string { return `Look at the current screen and answer this question: ${JSON.stringify(question)} Be concise and factual. Do NOT perform any actions. Only observe and respond.`; } - -function urlPrompt(): string { - return `Report the current page URL. Use the url action to read it. Do not perform any other actions.`; -} diff --git a/packages/cli/src/action/result.ts b/packages/cli/src/action/result.ts index 61b81e36..5b1983d4 100644 --- a/packages/cli/src/action/result.ts +++ b/packages/cli/src/action/result.ts @@ -1,4 +1,19 @@ -import type { ActionType } from "./prompts"; +import type { ModelActionType } from "./prompts"; + +/** Subcommands that execute directly against the browser/OS planes, no model involved. */ +export type DeterministicActionType = + | "open" + | "url" + | "snapshot" + | "text" + | "find" + | "fill" + | "press" + | "click" + | "tabs" + | "screenshot"; + +export type ActionType = ModelActionType | DeterministicActionType; export type Status = "ok" | "not_found" | "error" | "timeout"; @@ -23,7 +38,7 @@ export interface ActionResult { * and any action events captured during the run. */ export function parseResult( - action: ActionType, + action: ModelActionType, textOutput: string, actionEvents: ActionEventInfo[], elapsedMs: number, @@ -68,35 +83,11 @@ export function parseResult( } } - switch (action) { - case "observe": - result.text = trimmed; - break; - case "url": { - const url = extractFirstUrl(trimmed); - if (url) { - result.url = url; - } else if (trimmed) { - result.status = "error"; - result.text = trimmed; - } - break; - } - default: - break; - } + if (action === "observe") result.text = trimmed; return result; } -function extractFirstUrl(text: string): string | undefined { - const matches = text.match( - /(?:https?:\/\/\S+|about:blank|file:\/\/\S+|chrome:\/\/\S+|chrome-extension:\/\/\S+|edge:\/\/\S+|brave:\/\/\S+)/gi, - ); - if (!matches || matches.length === 0) return undefined; - return matches[matches.length - 1]!.replace(/[),.;!?]+$/, ""); -} - export function formatCompact(r: ActionResult): string { switch (r.status) { case "not_found": @@ -110,6 +101,7 @@ export function formatCompact(r: ActionResult): string { switch (r.action as ActionType) { case "click": if (r.coordinates) return `ok clicked (${r.coordinates[0]}, ${r.coordinates[1]})`; + if (r.text) return `ok clicked ${r.text}`; return "ok clicked"; case "type": return "ok typed"; @@ -117,12 +109,18 @@ export function formatCompact(r: ActionResult): string { return "ok"; case "press": return "ok pressed"; + case "fill": + return r.text ? `ok filled ${r.text}` : "ok filled"; case "observe": + case "snapshot": + case "text": + case "find": + case "tabs": return r.text ?? ""; case "url": return r.url ?? r.text ?? ""; case "screenshot": - return "ok"; + return r.text ?? "ok"; case "do": return r.text ?? "ok"; default: diff --git a/packages/cli/src/cli-executor.ts b/packages/cli/src/cli-executor.ts new file mode 100644 index 00000000..b11dab39 --- /dev/null +++ b/packages/cli/src/cli-executor.ts @@ -0,0 +1,334 @@ +import { InternalComputerTranslator, type BatchReadResult, type BrowserFindCandidate, type BrowserRefState } from "@onkernel/cua-agent"; +import { writeFile } from "node:fs/promises"; +import { stderr, stdout } from "node:process"; +import { emitCompact, type RunActionResult } from "./action/harness-runner"; +import { exitCodeFor, type ActionResult, type DeterministicActionType } from "./action/result"; +import { parseMode, parseNativeTool, provisionForFlags, requireKernelApiKey, type HarnessCliFlags } from "./cli-harness"; +import { readNamedSessionRefs, writeNamedSessionRefs } from "./harness-named-sessions"; +import { captureScreenshot, type CuaBrowserHandle } from "./harness-browser"; + +/** + * Model-free subcommand plane. These commands validate argv, attach to (or + * provision) a Kernel browser, and call the executor directly over CDP or + * the computer batch API — no LLM harness, no model API key. + */ + +export type DeterministicRequest = + | { action: "open"; url: string } + | { action: "url" } + | { action: "snapshot"; filter?: "interactive" } + | { action: "text" } + | { action: "find"; query: string } + | { action: "fill"; query: string; value: string } + | { action: "fill"; ref: string; value: string } + | { action: "press"; keys: string[] } + | { action: "click"; x: number; y: number } + | { action: "click"; ref: string } + | { action: "tabs" } + | { action: "screenshot"; out: string }; + +export const DETERMINISTIC_SUBCOMMANDS: ReadonlySet = new Set([ + "open", + "url", + "snapshot", + "text", + "find", + "fill", + "press", + "tabs", + "screenshot", +]); + +/** `cua click ` is deterministic; any other click argv is a model-mediated description. */ +export function isCoordinatePair(rest: string[]): boolean { + return rest.length === 2 && rest.every((token) => /^\d+$/.test(token)); +} + +/** An element ref minted by `cua snapshot` / `cua find`, e.g. `e12`. */ +export function isElementRef(token: string | undefined): token is string { + return token !== undefined && /^e\d+$/.test(token); +} + +/** Roles `cua fill` will target. Everything else is left to `click`/`type`. */ +const FILLABLE_ROLES: ReadonlySet = new Set([ + "textbox", + "searchbox", + "combobox", + "checkbox", + "radio", + "listbox", + "spinbutton", +]); + +/** Roles whose fill value is a checked state, not text. */ +const TOGGLE_ROLES: ReadonlySet = new Set(["checkbox", "radio"]); + +function parseToggleValue(raw: string): boolean { + const value = raw.trim().toLowerCase(); + if (["true", "1", "checked", "on"].includes(value)) return true; + if (["false", "0", "unchecked", "off"].includes(value)) return false; + throw new Error(`checkbox/radio value must be true|false|1|0|checked|unchecked|on|off, got ${JSON.stringify(raw)}`); +} + +/** + * Value for a ref-addressed fill, where the element's role is unknown until + * the browser resolves it. Toggle words become booleans — lossless for text + * controls (the page-side fill stringifies) and correct for checkboxes. + * "1"/"0" stay strings so select options and numeric inputs keep their value. + */ +function refFillValue(raw: string): string | boolean { + const value = raw.trim().toLowerCase(); + if (["true", "checked", "on"].includes(value)) return true; + if (["false", "unchecked", "off"].includes(value)) return false; + return raw; +} + +/** Resolve argv to a deterministic subcommand, or undefined when the model plane should handle it. */ +export function deterministicActionFor(first: string | undefined, rest: string[]): DeterministicActionType | undefined { + if (!first) return undefined; + if (DETERMINISTIC_SUBCOMMANDS.has(first)) return first as DeterministicActionType; + if (first === "click" && (isCoordinatePair(rest) || (rest.length === 1 && isElementRef(rest[0])))) return "click"; + return undefined; +} + +/** Parse and validate a deterministic subcommand's argv. Throws before any Kernel API call. */ +export function parseDeterministicArgs( + action: DeterministicActionType, + rest: string[], + flags: HarnessCliFlags, +): DeterministicRequest { + if (flags.filter !== undefined && action !== "snapshot") { + throw new Error("--filter only applies to cua snapshot"); + } + switch (action) { + case "open": { + const url = (rest[0] ?? "").trim(); + if (!url || rest.length > 1) throw new Error("usage: cua open "); + return { action, url }; + } + case "url": + if (rest.length > 0) throw new Error("usage: cua url"); + return { action }; + case "snapshot": { + if (rest.length > 0) throw new Error("usage: cua snapshot [--filter interactive]"); + const filter = flags.filter?.trim().toLowerCase(); + if (filter !== undefined && filter !== "interactive") { + throw new Error(`invalid --filter value "${flags.filter}"; expected: interactive`); + } + return { action, ...(filter === "interactive" ? { filter } : {}) }; + } + case "text": + if (rest.length > 0) throw new Error("usage: cua text"); + return { action }; + case "find": { + const query = rest.join(" ").trim(); + if (!query) throw new Error('usage: cua find ""'); + return { action, query }; + } + case "fill": { + const target = (rest[0] ?? "").trim(); + const value = rest[1]; + if (!target || value === undefined || rest.length > 2) { + throw new Error('usage: cua fill ""'); + } + if (isElementRef(target)) return { action, ref: target, value }; + return { action, query: target, value }; + } + case "press": { + const keys = rest.map((key) => key.trim()).filter((key) => key.length > 0); + if (keys.length === 0) throw new Error("usage: cua press [key...]"); + return { action, keys }; + } + case "click": { + if (rest.length === 1 && isElementRef(rest[0])) return { action, ref: rest[0] }; + if (!isCoordinatePair(rest)) throw new Error("usage: cua click | cua click "); + return { action, x: Number(rest[0]), y: Number(rest[1]) }; + } + case "tabs": + if (rest.length > 0) throw new Error("usage: cua tabs"); + return { action }; + case "screenshot": { + if (rest.length > 0) throw new Error("usage: cua screenshot [--out file|-]"); + return { action, out: flags.out ?? "screenshot.png" }; + } + } +} + +/** + * Persistence seam for element refs so they survive across invocations of + * the same named session. Absent for fresh (non `-s`) browsers, whose refs + * cannot outlive the browser anyway. + */ +export interface RefStateStore { + load(): Promise; + save(state: BrowserRefState): Promise; +} + +/** Run a deterministic subcommand end to end: parse, provision/attach, execute, print, tear down. */ +export async function runDeterministicCommand( + action: DeterministicActionType, + rest: string[], + flags: HarnessCliFlags, +): Promise { + const req = parseDeterministicArgs(action, rest, flags); + // Deterministic commands ignore the runtime mode, but an invalid value is + // still a usage error, same as the harness-backed entry points. + parseMode(flags.mode); + parseNativeTool(flags.nativeTool); + const { apiKey, baseUrl } = requireKernelApiKey(); + const provisioned = await provisionForFlags(flags, { kernelApiKey: apiKey, kernelBaseUrl: baseUrl }); + const name = flags.namedSession; + const refStore: RefStateStore | undefined = name + ? { + load: () => readNamedSessionRefs(name), + save: (state) => writeNamedSessionRefs(name, state), + } + : undefined; + return runDeterministicOnHandle(req, provisioned.handle, defaultTranslator, refStore); +} + +/** Execute a parsed request against a browser handle. Split from provisioning for tests. */ +export async function runDeterministicOnHandle( + req: DeterministicRequest, + handle: CuaBrowserHandle, + createTranslator: (handle: CuaBrowserHandle) => InternalComputerTranslator = defaultTranslator, + refStore?: RefStateStore, +): Promise { + const translator = createTranslator(handle); + try { + if (refStore) { + const state = await refStore.load(); + if (state) translator.browser().importRefState(state); + } + const res = await executeDeterministic(req, translator, handle); + return emitCompact(res); + } finally { + if (refStore) { + try { + await refStore.save(translator.browser().exportRefState()); + } catch (err) { + stderr.write(`[cua] cleanup warning: ${(err as Error).message}\n`); + } + } + try { + translator.dispose(); + } catch (err) { + stderr.write(`[cua] cleanup warning: ${(err as Error).message}\n`); + } + try { + await handle.close(); + } catch (err) { + stderr.write(`[cua] cleanup warning: ${(err as Error).message}\n`); + } + } +} + +function defaultTranslator(handle: CuaBrowserHandle): InternalComputerTranslator { + return new InternalComputerTranslator({ browser: handle.browser, client: handle.client }); +} + +async function executeDeterministic( + req: DeterministicRequest, + translator: InternalComputerTranslator, + handle: CuaBrowserHandle, +): Promise { + const startedAt = Date.now(); + const finish = (partial: Omit): RunActionResult => { + const result: ActionResult = { ...partial, elapsedMs: Date.now() - startedAt, timestamp: Date.now() }; + return { result, exitCode: exitCodeFor(result) }; + }; + try { + switch (req.action) { + case "open": + await translator.browser().execute({ type: "browser_navigate", url: req.url }); + return finish({ action: req.action, status: "ok" }); + case "url": + return finish({ action: req.action, status: "ok", url: await translator.browser().currentUrl() }); + case "snapshot": { + const reads = await translator.browser().execute({ type: "browser_snapshot", ...(req.filter ? { filter: req.filter } : {}) }); + return finish({ action: req.action, status: "ok", text: readText(reads) }); + } + case "text": { + const reads = await translator.browser().execute({ type: "browser_text" }); + return finish({ action: req.action, status: "ok", text: readText(reads) }); + } + case "find": { + const candidates = await translator.browser().findCandidates(req.query); + if (candidates.length === 0) { + return finish({ action: req.action, status: "not_found", text: `no elements matched ${JSON.stringify(req.query)}` }); + } + return finish({ action: req.action, status: "ok", text: candidates.map(formatCandidate).join("\n") }); + } + case "fill": { + const executor = translator.browser(); + if ("ref" in req) { + await executor.execute({ type: "browser_fill", ref: req.ref, value: refFillValue(req.value) }); + return finish({ action: req.action, status: "ok", text: req.ref }); + } + const candidates = await executor.findCandidates(req.query, undefined, FILLABLE_ROLES); + if (candidates.length === 0) { + return finish({ action: req.action, status: "not_found", text: `no fillable element matched ${JSON.stringify(req.query)}` }); + } + const tied = candidates.filter((c) => c.score === candidates[0]!.score); + if (tied.length > 1) { + const listing = tied.map((c) => `${c.role} ${JSON.stringify(c.name)}`).join(", "); + return finish({ + action: req.action, + status: "not_found", + text: `ambiguous query ${JSON.stringify(req.query)} (${tied.length} matches): ${listing}`, + }); + } + const match = candidates[0]!; + const value = TOGGLE_ROLES.has(match.role) ? parseToggleValue(req.value) : req.value; + await executor.execute({ type: "browser_fill", ref: match.ref, value }); + return finish({ action: req.action, status: "ok", text: `${match.role} ${JSON.stringify(match.name)}` }); + } + case "press": + await translator.executeBatch([{ type: "keypress", keys: req.keys }]); + return finish({ action: req.action, status: "ok" }); + case "click": + if ("ref" in req) { + await translator.browser().execute({ type: "browser_click", ref: req.ref }); + return finish({ action: req.action, status: "ok", text: req.ref }); + } + await translator.executeBatch([{ type: "click", x: req.x, y: req.y }]); + return finish({ action: req.action, status: "ok", coordinates: [req.x, req.y] }); + case "tabs": { + const reads = await translator.browser().execute({ type: "browser_list_tabs" }); + return finish({ action: req.action, status: "ok", text: readText(reads) }); + } + case "screenshot": { + const png = await captureScreenshot(handle.client, handle.browser.session_id); + if (!png) { + return finish({ action: req.action, status: "error", text: "failed to capture screenshot" }); + } + if (req.out === "-") { + // stdout is the PNG bytes; the compact status line would corrupt a pipe. + stdout.write(png); + return finish({ action: req.action, status: "ok", text: "" }); + } + await writeFile(req.out, png); + return finish({ action: req.action, status: "ok", text: req.out }); + } + } + } catch (err) { + const message = (err as Error).message; + // A stale ref is "not found" (exit 1): the caller should re-snapshot, + // same as a failed description match — not an infrastructure error. + const status = /stale|not on the current page/i.test(message) ? "not_found" : "error"; + return finish({ action: req.action, status, text: message }); + } +} + +function formatCandidate(candidate: BrowserFindCandidate): string { + const name = candidate.name ? ` ${JSON.stringify(candidate.name)}` : ""; + return `${candidate.role || "node"}${name} [${candidate.ref}]`; +} + +function readText(reads: BatchReadResult[]): string { + const parts: string[] = []; + for (const read of reads) { + if (read.type === "browser_text") parts.push(read.text); + } + return parts.join("\n"); +} diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts index dc130079..66e351d2 100644 --- a/packages/cli/src/cli-harness.ts +++ b/packages/cli/src/cli-harness.ts @@ -7,16 +7,19 @@ import { type Skill, } from "@onkernel/cua-agent"; import { + type CuaMode, type CuaModelRef, + type CuaNativeToolSpec, parseCuaModelRef, requireCuaEnvApiKey, + resolveCuaRuntimeSpec, } from "@onkernel/cua-ai"; import { parseArgs } from "node:util"; import { stderr, stdout } from "node:process"; import type { CuaBrowserHandle } from "./harness-browser"; import { type ActionRequest, - type ActionType, + type ModelActionType, } from "./action/prompts"; import { runAction, emitCompact } from "./action/harness-runner"; import { buildCuaHarness } from "./harness"; @@ -27,6 +30,8 @@ import { formatRelativeAge, listNamedSessions, type NamedSessionMetadata, + readNamedSession, + recordSessionModel, recordTranscriptPath, shortKernelId, startNamedSession, @@ -177,6 +182,8 @@ export interface HarnessCliFlags { jsonlIncludeDeltas: boolean; jsonlIncludeImages: boolean; playwright: boolean; + mode?: string; + nativeTool?: string; model?: string; thinking?: string; browserProfile?: string; @@ -184,6 +191,7 @@ export interface HarnessCliFlags { maxSteps?: number; out?: string; output?: string; + filter?: string; imageProtocol?: string; namedSession?: string; sessionRef?: string; @@ -191,13 +199,16 @@ export interface HarnessCliFlags { skillPaths: string[]; } -interface ResolvedAuth { +export interface KernelAuth { kernelApiKey: string; kernelBaseUrl?: string; +} + +interface ResolvedAuth extends KernelAuth { modelRef: CuaModelRef; } -function requireKernelApiKey(): { apiKey: string; baseUrl?: string } { +export function requireKernelApiKey(): { apiKey: string; baseUrl?: string } { const apiKey = process.env.KERNEL_API_KEY?.trim(); if (!apiKey) throw new Error("missing Kernel API key (set KERNEL_API_KEY)"); const baseUrl = process.env.KERNEL_BASE_URL?.trim() || undefined; @@ -213,12 +224,12 @@ function resolveAuth(flags: HarnessCliFlags): ResolvedAuth { return { kernelApiKey: apiKey, kernelBaseUrl: baseUrl, modelRef }; } -interface ProvisionedBrowser { +export interface ProvisionedBrowser { handle: CuaBrowserHandle; named?: NamedSessionMetadata; } -async function provisionForFlags(flags: HarnessCliFlags, auth: ResolvedAuth): Promise { +export async function provisionForFlags(flags: HarnessCliFlags, auth: KernelAuth): Promise { if (flags.namedSession) { const { client, browser, meta } = await attachNamedSession({ name: flags.namedSession, @@ -358,10 +369,24 @@ export interface SetupHarnessRuntimeOptions { skipDiskSession?: boolean; } +/** Default -m/--mode/--native-tool from a named session's stored values when not passed explicitly. */ +export function applyNamedSessionDefaults(flags: HarnessCliFlags, meta: NamedSessionMetadata): HarnessCliFlags { + return { + ...flags, + model: flags.model ?? meta.model, + mode: flags.mode ?? meta.mode, + nativeTool: flags.nativeTool ?? meta.native_tool, + }; +} + async function setupHarnessRuntime( flags: HarnessCliFlags, opts: SetupHarnessRuntimeOptions = {}, ): Promise { + if (flags.namedSession) { + const named = await readNamedSession(flags.namedSession); + if (named) flags = applyNamedSessionDefaults(flags, named); + } const auth = resolveAuth(flags); const cwd = process.cwd(); const env = new NodeExecutionEnv({ cwd }); @@ -372,10 +397,40 @@ async function setupHarnessRuntime( disabled: flags.noSkills, }); + // Validate mode/native-tool flags before provisioning so a bad combination + // never leaves an orphaned browser behind. + const mode = parseMode(flags.mode); + const nativeTool = parseNativeTool(flags.nativeTool); + resolveCuaRuntimeSpec(auth.modelRef, { mode, nativeTool }); + const provisioned = await provisionForFlags(flags, auth); + try { + return await finishHarnessRuntime(flags, auth, provisioned, { cwd, skills, contextFiles, mode, nativeTool, skipDisk: opts.skipDiskSession === true }); + } catch (err) { + await provisioned.handle.close().catch(() => {}); + throw err; + } +} + +interface FinishHarnessRuntimeContext { + cwd: string; + skills: Skill[]; + contextFiles: ContextFile[]; + mode: CuaMode | undefined; + nativeTool: CuaNativeToolSpec | undefined; + skipDisk: boolean; +} + +async function finishHarnessRuntime( + flags: HarnessCliFlags, + auth: ResolvedAuth, + provisioned: ProvisionedBrowser, + context: FinishHarnessRuntimeContext, +): Promise { + const { cwd, skills, contextFiles, mode, nativeTool } = context; const repo = createSessionRepo(flags.sessionDir); - const skipDisk = opts.skipDiskSession === true && !hasExplicitSessionFlag(flags); + const skipDisk = context.skipDisk && !hasExplicitSessionFlag(flags); const resolved = skipDisk ? undefined : await resolveSession(repo, cwd, flags, provisioned.named); let inMemorySession: Session | undefined; @@ -396,6 +451,11 @@ async function setupHarnessRuntime( }); if (provisioned.named) { await recordTranscriptPath(provisioned.named.name, resolved.transcriptPath); + await recordSessionModel(provisioned.named.name, { + model: auth.modelRef, + mode: flags.mode, + native_tool: flags.nativeTool, + }); } if (flags.verbose) { stderr.write(`[cua] session=${resolved.transcriptPath}\n`); @@ -414,6 +474,8 @@ async function setupHarnessRuntime( skills, contextFiles, thinkingLevel, + mode, + nativeTool, playwright: flags.playwright, modelBaseUrl: baseUrlOverride, }); @@ -445,6 +507,23 @@ function providerBaseUrlOverride(provider: string): string | undefined { return value && value.length > 0 ? value : undefined; } +export function parseMode(raw: string | undefined): CuaMode | undefined { + if (raw === undefined) return undefined; + const value = raw.trim().toLowerCase(); + if (value === "computer" || value === "browser" || value === "hybrid") return value; + throw new Error(`invalid --mode value "${raw}"; expected one of: computer | browser | hybrid`); +} + +export function parseNativeTool(raw: string | undefined): CuaNativeToolSpec | undefined { + if (raw === undefined) return undefined; + const value = raw.trim().toLowerCase(); + // enable_zoom follows Anthropic's own recommendation for fine-grained + // visual targeting; the executor implements the zoom crop locally. + if (value === "computer_20260701") return { type: "computer_20260701", enable_zoom: true }; + if (value === "browser_20260701") return { type: "browser_20260701" }; + throw new Error(`invalid --native-tool value "${raw}"; expected one of: computer_20260701 | browser_20260701`); +} + function mapThinkingLevel(raw: string | undefined): "off" | "minimal" | "low" | "medium" | "high" | "xhigh" { const v = (raw ?? "low").trim().toLowerCase(); switch (v) { @@ -520,6 +599,7 @@ export async function runInteractiveCommand( resumed: runtime.resolved?.resumed === true, transcriptPath: runtime.resolved?.transcriptPath, skipInitialScreenshot: runtime.resolved?.resumed === true, + namedSession: flags.namedSession, }); } finally { try { @@ -530,27 +610,22 @@ export async function runInteractiveCommand( } } -/** Run a one-shot action subcommand through the new harness wiring. */ +/** Run a one-shot model-mediated action subcommand through the harness wiring. */ export async function runActionCommand( - action: ActionType, + action: ModelActionType, rest: string[], flags: HarnessCliFlags, ): Promise { const runtime = await setupHarnessRuntime(flags, { skipDiskSession: true }); const req: ActionRequest = buildActionRequest(action, rest); if (flags.maxSteps !== undefined) req.maxTurns = flags.maxSteps; - const screenshotOut = flags.out - ? { out: flags.out } - : action === "screenshot" - ? { out: "screenshot.png" } - : undefined; try { const res = await runAction(req, { harness: runtime.harness, browserHandle: runtime.handle, session: runtime.session, skipInitialScreenshot: runtime.resolved?.resumed === true, - }, screenshotOut); + }); return emitCompact(res); } finally { try { @@ -561,22 +636,14 @@ export async function runActionCommand( } } -function buildActionRequest(action: ActionType, rest: string[]): ActionRequest { +function buildActionRequest(action: ModelActionType, rest: string[]): ActionRequest { switch (action) { - case "open": - return { action, text: rest[0] }; case "click": return { action, target: rest.join(" ") }; case "type": return { action, target: rest[0], text: rest[1] }; - case "press": - return { action, keys: rest }; case "observe": return { action, text: rest.join(" ") }; - case "url": - return { action }; - case "screenshot": - return { action }; case "do": return { action, text: rest.join(" ") }; } @@ -601,6 +668,7 @@ export async function runSessionSubcommand(args: string[], flags: HarnessCliFlag browserTimeoutSeconds: flags.browserTimeout, profileSelector: flags.browserProfile, saveProfileChanges: flags.profileSaveChanges, + model: flags.model ? resolveCuaModelRef(flags.model) : undefined, }); stdout.write(`name=${meta.name}\n`); stdout.write(`kernel_session_id=${browser.session_id}\n`); diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 5c20070d..f678ead2 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -1,7 +1,8 @@ #!/usr/bin/env node import { stderr, stdout } from "node:process"; import { parseArgs } from "node:util"; -import { type ActionType } from "./action/prompts"; +import { type ModelActionType } from "./action/prompts"; +import { deterministicActionFor, runDeterministicCommand } from "./cli-executor"; import { runActionCommand, runInteractiveCommand, @@ -17,17 +18,32 @@ const HELP = `cua — Kernel-cloud-browser computer-use agent Usage: cua [options] [prompt...] cua --print "go to news.ycombinator.com and summarize" - cua open + cua open + cua url + cua snapshot [--filter interactive] + cua find "" + cua text + cua fill "" + cua press [key...] + cua click | cua click + cua tabs + cua screenshot [--out file|-] + cua click "" cua type "" "" - cua press [key...] cua observe [""] - cua url - cua screenshot [--out file|-] cua do "" cua models [-p provider] cua session start [name] | stop | list | show +Subcommands above the blank line are model-free: they run directly against +the browser (no LLM, no model API key; only KERNEL_API_KEY). \`click \` +with exactly two integer arguments clicks those viewport coordinates without +a model, and \`click e12\` / \`fill e12 ...\` target an element ref minted by +\`snapshot\` or \`find\`; any other \`click\` argument is a natural-language +description resolved by the model. With \`-s \`, refs span invocations +(re-snapshot on a stale-ref error). Exit codes: 0 ok, 1 not_found, 2 error. + Options: -p, --print Run a single prompt and exit -m, --model Model ref (default: ${DEFAULT_CUA_MODEL_REF}) @@ -47,7 +63,15 @@ Options: --max-steps Max turns for action subcommands (default 3) --playwright Add the playwright_execute tool so the model can run Playwright code against the browser session + --mode Action plane(s) to expose: computer (default) | browser | hybrid + computer: OS-level input only. browser: CDP page tools + (snapshot, find, click-by-ref, navigate, tabs). + hybrid: both, deduplicated (computer_* + browser_* tools). + --native-tool Drive an Anthropic model through its native tool schema: + computer_20260701 (requires --mode computer) or + browser_20260701 (requires --mode browser) --out Output file for screenshot subcommand + --filter Restrict \`cua snapshot\` to interactive elements -o, --output Output format for --print: text (default) | jsonl --jsonl-include-deltas Include assistant_text_delta events (default off) --jsonl-include-images Include base64 screenshots (default off, only sizes) @@ -101,6 +125,8 @@ interface CliFlags { jsonlIncludeDeltas: boolean; jsonlIncludeImages: boolean; playwright: boolean; + mode?: string; + nativeTool?: string; model?: string; thinking?: string; browserProfile?: string; @@ -108,6 +134,7 @@ interface CliFlags { maxSteps?: number; out?: string; output?: string; + filter?: string; imageProtocol?: string; namedSession?: string; sessionRef?: string; @@ -136,6 +163,7 @@ function parseCliArgs(argv: string[]): CliFlags { "browser-timeout": { type: "string" }, "max-steps": { type: "string" }, out: { type: "string" }, + filter: { type: "string" }, "image-protocol": { type: "string" }, "session-name": { type: "string", short: "s" }, continue: { type: "boolean", short: "c", default: false }, @@ -150,6 +178,8 @@ function parseCliArgs(argv: string[]): CliFlags { "jsonl-include-deltas": { type: "boolean", default: false }, "jsonl-include-images": { type: "boolean", default: false }, playwright: { type: "boolean", default: false }, + mode: { type: "string" }, + "native-tool": { type: "string" }, }, allowPositionals: true, strict: true, @@ -171,6 +201,14 @@ function parseCliArgs(argv: string[]): CliFlags { ); } } + const modeRaw = parsed.values.mode as string | undefined; + if (modeRaw !== undefined && !["computer", "browser", "hybrid"].includes(modeRaw.trim().toLowerCase())) { + throw new Error(`invalid --mode value "${modeRaw}"; expected one of: computer | browser | hybrid`); + } + const nativeToolRaw = parsed.values["native-tool"] as string | undefined; + if (nativeToolRaw !== undefined && !["computer_20260701", "browser_20260701"].includes(nativeToolRaw.trim().toLowerCase())) { + throw new Error(`invalid --native-tool value "${nativeToolRaw}"; expected one of: computer_20260701 | browser_20260701`); + } return { help: !!parsed.values.help, @@ -188,6 +226,7 @@ function parseCliArgs(argv: string[]): CliFlags { browserTimeout: Number.isFinite(browserTimeout) ? browserTimeout : undefined, maxSteps: Number.isFinite(maxSteps) ? maxSteps : undefined, out: parsed.values.out as string | undefined, + filter: parsed.values.filter as string | undefined, imageProtocol: parsed.values["image-protocol"] as string | undefined, namedSession: parsed.values["session-name"] as string | undefined, sessionRef: parsed.values.session as string | undefined, @@ -197,6 +236,8 @@ function parseCliArgs(argv: string[]): CliFlags { jsonlIncludeDeltas: !!parsed.values["jsonl-include-deltas"], jsonlIncludeImages: !!parsed.values["jsonl-include-images"], playwright: !!parsed.values.playwright, + mode: parsed.values.mode as string | undefined, + nativeTool: parsed.values["native-tool"] as string | undefined, positionals: parsed.positionals, }; } @@ -213,6 +254,8 @@ function toHarnessFlags(flags: CliFlags): HarnessCliFlags { jsonlIncludeDeltas: flags.jsonlIncludeDeltas, jsonlIncludeImages: flags.jsonlIncludeImages, playwright: flags.playwright, + mode: flags.mode, + nativeTool: flags.nativeTool, model: flags.model, thinking: flags.thinking, browserProfile: flags.browserProfile, @@ -220,6 +263,7 @@ function toHarnessFlags(flags: CliFlags): HarnessCliFlags { maxSteps: flags.maxSteps, out: flags.out, output: flags.output, + filter: flags.filter, imageProtocol: flags.imageProtocol, namedSession: flags.namedSession, sessionRef: flags.sessionRef, @@ -228,7 +272,7 @@ function toHarnessFlags(flags: CliFlags): HarnessCliFlags { }; } -const SUBCOMMANDS = new Set(["open", "click", "type", "press", "observe", "url", "screenshot", "do"]); +const MODEL_SUBCOMMANDS = new Set(["click", "type", "observe", "do"]); export async function main(argv: string[]): Promise { if (argv[0] === "models") { @@ -260,9 +304,21 @@ export async function main(argv: string[]): Promise { } } - if (first && SUBCOMMANDS.has(first)) { + const rest = positionals.slice(1); + + const deterministic = deterministicActionFor(first, rest); + if (deterministic) { + try { + return await runDeterministicCommand(deterministic, rest, toHarnessFlags(flags)); + } catch (err) { + stderr.write(`error: ${(err as Error).message}\n`); + return 2; + } + } + + if (first && MODEL_SUBCOMMANDS.has(first)) { try { - return await runActionCommand(first as ActionType, positionals.slice(1), toHarnessFlags(flags)); + return await runActionCommand(first as ModelActionType, rest, toHarnessFlags(flags)); } catch (err) { stderr.write(`error: ${(err as Error).message}\n`); return 2; diff --git a/packages/cli/src/harness-named-sessions.ts b/packages/cli/src/harness-named-sessions.ts index 9b703d3d..7182c0f2 100644 --- a/packages/cli/src/harness-named-sessions.ts +++ b/packages/cli/src/harness-named-sessions.ts @@ -1,4 +1,4 @@ -import type { KernelBrowser } from "@onkernel/cua-agent"; +import type { BrowserRefState, KernelBrowser } from "@onkernel/cua-agent"; import Kernel from "@onkernel/sdk"; import { mkdir, readdir, readFile, stat, unlink, writeFile } from "node:fs/promises"; import { homedir } from "node:os"; @@ -19,6 +19,10 @@ export interface NamedSessionMetadata { live_url?: string; profile_id?: string; transcript_path?: string; + /** Model ref last used with this session; chained invocations without -m default to it. */ + model?: string; + mode?: string; + native_tool?: string; created_at: number; } @@ -68,21 +72,47 @@ export async function writeNamedSession(meta: NamedSessionMetadata): Promise { const path = sessionFilePath(name); + await unlink(refsFilePath(name)).catch(() => {}); if (!(await fileExists(path))) return false; await unlink(path); return true; } +function refsFilePath(name: string): string { + return join(namedSessionsDir(), `${name}.refs.json`); +} + +/** + * Element refs minted by one invocation (snapshot/find) survive to the next + * via this per-session file, so `cua -s x snapshot` then `cua -s x click e12` + * works across processes. Scoped to the named session's browser; stale state + * is caught by the executor's generation/self-heal machinery. + */ +export async function readNamedSessionRefs(name: string): Promise { + try { + return JSON.parse(await readFile(refsFilePath(name), "utf8")) as BrowserRefState; + } catch { + return undefined; + } +} + +export async function writeNamedSessionRefs(name: string, state: BrowserRefState): Promise { + await mkdir(namedSessionsDir(), { recursive: true }); + await writeFile(refsFilePath(name), JSON.stringify(state) + "\n", { mode: 0o600 }); +} + export async function listNamedSessions(): Promise { const dir = namedSessionsDir(); if (!(await fileExists(dir))) return []; const entries = await readdir(dir); const out: NamedSessionMetadata[] = []; for (const entry of entries) { - if (!entry.endsWith(".json")) continue; + if (!entry.endsWith(".json") || entry.endsWith(".refs.json")) continue; try { const raw = await readFile(join(dir, entry), "utf8"); - out.push(JSON.parse(raw) as NamedSessionMetadata); + const meta = JSON.parse(raw) as NamedSessionMetadata; + if (typeof meta.name !== "string" || typeof meta.kernel_session_id !== "string" || typeof meta.created_at !== "number") continue; + out.push(meta); } catch { // skip unreadable / malformed entries } @@ -99,6 +129,8 @@ export interface StartNamedSessionOptions { /** Profile id or name (created if missing). Same semantics as `--profile`. */ profileSelector?: string; saveProfileChanges?: boolean; + /** Canonical model ref to seed the session with (same semantics as `-m`). */ + model?: string; } export interface StartNamedSessionResult { @@ -138,6 +170,7 @@ export async function startNamedSession(opts: StartNamedSessionOptions): Promise kernel_session_id: browser.session_id, live_url: browser.browser_live_view_url, profile_id: profileId, + model: opts.model, created_at: Date.now(), }; const metadataPath = await writeNamedSession(meta); @@ -232,6 +265,32 @@ export async function recordTranscriptPath(name: string, transcriptPath: string) await writeNamedSession(meta); } +/** Persist the model/mode/native-tool used with a named session so chained invocations reuse them. */ +export async function recordSessionModel( + name: string, + runtime: { model: string; mode?: string; native_tool?: string }, +): Promise { + const meta = await readNamedSession(name); + if (!meta) return; + if (meta.model === runtime.model && meta.mode === runtime.mode && meta.native_tool === runtime.native_tool) return; + meta.model = runtime.model; + meta.mode = runtime.mode; + meta.native_tool = runtime.native_tool; + await writeNamedSession(meta); +} + +/** Patch individual runtime fields (e.g. after a TUI /mode or /model switch) without clobbering the rest. */ +export async function updateNamedSessionRuntime(name: string, patch: { model?: string; mode?: string }): Promise { + const meta = await readNamedSession(name); + if (!meta) return; + const model = patch.model ?? meta.model; + const mode = patch.mode ?? meta.mode; + if (meta.model === model && meta.mode === mode) return; + meta.model = model; + meta.mode = mode; + await writeNamedSession(meta); +} + export function shortKernelId(id: string): string { return id.length > 10 ? `${id.slice(0, 8)}…` : id; } diff --git a/packages/cli/src/harness.ts b/packages/cli/src/harness.ts index 9ee9e4bd..5dfaa970 100644 --- a/packages/cli/src/harness.ts +++ b/packages/cli/src/harness.ts @@ -10,7 +10,9 @@ import { } from "@onkernel/cua-agent"; import { type Api, + type CuaMode, type CuaModelRef, + type CuaNativeToolSpec, type Model, type Models, getCuaModel, @@ -31,6 +33,10 @@ export interface BuildCuaHarnessOptions { /** Context files (AGENTS.md, CLAUDE.md, …) appended to the system prompt. */ contextFiles?: ContextFile[]; thinkingLevel?: ThinkingLevel; + /** Which canonical action plane(s) to expose: "computer" (default), "browser", or "hybrid". */ + mode?: CuaMode; + /** Drive the model through a provider-native tool declaration (validated against `mode`). */ + nativeTool?: CuaNativeToolSpec; /** Expose the playwright_execute tool that runs Playwright code against the browser session. */ playwright?: boolean; /** Override the default coding-tools extraTools (bash/read/edit/write/grep/find/ls). */ @@ -55,22 +61,31 @@ export function buildCuaHarness(opts: BuildCuaHarnessOptions): CuaAgentHarness { const model: CuaModelRef | Model = opts.modelBaseUrl ? { ...getCuaModel(opts.model), baseUrl: opts.modelBaseUrl } : opts.model; - return new CuaAgentHarness({ + // The system-prompt callback re-resolves per turn and must see the live + // mode after /mode switches, so it reads it from the harness (late-bound). + let harness: CuaAgentHarness | undefined; + harness = new CuaAgentHarness({ env: new NodeExecutionEnv({ cwd: opts.cwd }), session: opts.session, model, browser: opts.browser, client: opts.client, extraTools, + mode: opts.mode, + nativeTool: opts.nativeTool, playwright: opts.playwright, resources: { skills }, thinkingLevel: opts.thinkingLevel, systemPrompt: ({ model: activeModel, resources }) => { - const runtime = resolveCuaRuntimeSpec(activeModel); + const runtime = resolveCuaRuntimeSpec(activeModel, { + mode: harness?.getMode() ?? opts.mode, + nativeTool: opts.nativeTool, + }); return composeSystemPrompt(runtime.defaultSystemPrompt, resources.skills ?? [], contextFiles); }, models: opts.models, }); + return harness; } function composeSystemPrompt(base: string, skills: Skill[], contextFiles: ContextFile[]): string { diff --git a/packages/cli/src/print.ts b/packages/cli/src/print.ts index 8d244a9e..40275419 100644 --- a/packages/cli/src/print.ts +++ b/packages/cli/src/print.ts @@ -94,6 +94,9 @@ export async function runPrint(opts: RunPrintOptions): Promise { async function maybeInitialScreenshot(opts: RunPrintOptions): Promise { if (opts.skipInitialScreenshot) return undefined; + // Browser mode's only frame is the viewport; skip the OS-display capture + // rather than mix coordinate frames on the first turn. + if (opts.harness.getMode() === "browser") return undefined; const hasPriorTurn = await sessionHasPriorTurn(opts.session); if (hasPriorTurn) return undefined; const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id); diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts index 67111d39..fffbf3cd 100644 --- a/packages/cli/src/tui/main.ts +++ b/packages/cli/src/tui/main.ts @@ -25,6 +25,7 @@ import { homedir } from "node:os"; import type { ImageContent, Model } from "@onkernel/cua-ai"; import { captureScreenshot, type CuaBrowserHandle } from "../harness-browser"; import { resolveCuaModelRef } from "../harness-models"; +import { updateNamedSessionRuntime } from "../harness-named-sessions"; import type { ContextFile } from "../harness-skills"; import { openTuiDebugLog } from "./debug-log"; import { applyAndSummarizeImageProtocol } from "./diagnostics"; @@ -56,6 +57,8 @@ export interface InteractiveOptions { resumed?: boolean; /** Display path of the on-disk transcript, when one exists. */ transcriptPath?: string; + /** Named session (-s) backing this TUI; /mode and /model switches persist to it. */ + namedSession?: string; /** Enable extra TUI render diagnostics for manual repros. */ debugTui?: boolean; } @@ -303,6 +306,10 @@ export async function runInteractive(opts: InteractiveOptions): Promise await applyModelCommand(opts, footer, status, messages, parsed.argument); return; } + if (parsed?.command === "mode") { + await applyModeCommand(opts, messages, parsed.argument); + return; + } if (parsed?.command === "thinking") { await applyThinkingCommand(opts, footer, messages, parsed.argument); return; @@ -443,6 +450,9 @@ async function maybeInitialScreenshot( ): Promise { if (firstPromptSent) return undefined; if (opts.skipInitialScreenshot) return undefined; + // Browser mode's only frame is the viewport; skip the OS-display capture + // rather than mix coordinate frames on the first turn. + if (opts.harness.getMode() === "browser") return undefined; if (await sessionHasPriorTurn(opts.session)) return undefined; const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id); if (!png) return undefined; @@ -482,8 +492,43 @@ async function applyModelCommand( }); status.update({ model: modelLabel(model) }); messages.addNotice(`model → ${resolved}`); + await persistNamedSessionRuntime(opts, messages, { model: resolved }); + } catch (err) { + messages.addError((err as Error).message); + } +} + +async function applyModeCommand(opts: InteractiveOptions, messages: MessageList, argument: string): Promise { + const value = argument.trim().toLowerCase(); + if (value !== "computer" && value !== "browser" && value !== "hybrid") { + messages.addError("usage: /mode "); + return; + } + try { + await opts.harness.setMode(value); } catch (err) { messages.addError((err as Error).message); + return; + } + messages.addNotice(`mode → ${value}`); + await persistNamedSessionRuntime(opts, messages, { mode: value }); +} + +// Persistence is best-effort: the live switch already happened, so a failed +// metadata write must not masquerade as a failed switch — warn that resume +// will restore the previous value instead. +async function persistNamedSessionRuntime( + opts: InteractiveOptions, + messages: MessageList, + patch: { model?: string; mode?: string }, +): Promise { + if (!opts.namedSession) return; + try { + await updateNamedSessionRuntime(opts.namedSession, patch); + } catch (err) { + messages.addError( + `switched, but failed to persist to session "${opts.namedSession}" (resume will restore the previous value): ${(err as Error).message}`, + ); } } diff --git a/packages/cli/src/tui/slash-commands.ts b/packages/cli/src/tui/slash-commands.ts index 8d4ac475..fe83941f 100644 --- a/packages/cli/src/tui/slash-commands.ts +++ b/packages/cli/src/tui/slash-commands.ts @@ -27,6 +27,13 @@ export function buildAutocompleteProvider( getArgumentCompletions: (prefix: string) => modelCompletions(prefix), }); + commands.push({ + name: "mode", + description: "Switch the action plane(s): computer | browser | hybrid", + argumentHint: "", + getArgumentCompletions: (prefix: string) => modeCompletions(prefix), + }); + commands.push({ name: "thinking", description: "Set the reasoning level for future turns", @@ -58,6 +65,18 @@ function modelCompletions(prefix: string): AutocompleteItem[] { return filtered.map((m) => ({ value: m.ref, label: m.ref, description: m.name })); } +const MODES: ReadonlyArray<{ value: string; description: string }> = [ + { value: "computer", description: "OS-level input only (default)" }, + { value: "browser", description: "CDP page tools: snapshot, find, click-by-ref, navigate, tabs" }, + { value: "hybrid", description: "Both planes: computer_* input + ref-only browser_* tools" }, +]; + +function modeCompletions(prefix: string): AutocompleteItem[] { + const trimmed = prefix.trim().toLowerCase(); + const filtered = trimmed ? MODES.filter((m) => m.value.startsWith(trimmed)) : MODES; + return filtered.map((m) => ({ value: m.value, label: m.value, description: m.description })); +} + const THINKING_LEVELS: ReadonlyArray<{ value: string; description: string }> = [ { value: "off", description: "Disable reasoning" }, { value: "minimal", description: "Minimal reasoning" }, @@ -75,6 +94,7 @@ function thinkingCompletions(prefix: string): AutocompleteItem[] { export type ParsedSlashCommand = | { command: "model"; argument: string } + | { command: "mode"; argument: string } | { command: "thinking"; argument: string } | { command: "compact"; argument: string } | { command: "skill"; name: string; remainder: string }; @@ -91,11 +111,11 @@ export function parseSlashCommand(text: string): ParsedSlashCommand | undefined const [, name, rest] = skillMatch; return { command: "skill", name: name ?? "", remainder: (rest ?? "").trim() }; } - const builtinMatch = trimmed.match(/^\/(model|thinking|compact)\s*(.*)$/); + const builtinMatch = trimmed.match(/^\/(model|mode|thinking|compact)\s*(.*)$/); if (builtinMatch) { const [, name, rest] = builtinMatch; return { - command: name as "model" | "thinking" | "compact", + command: name as "model" | "mode" | "thinking" | "compact", argument: (rest ?? "").trim(), }; } diff --git a/packages/cli/test/action-runner.test.ts b/packages/cli/test/action-runner.test.ts index 4ab513d8..6833e79b 100644 --- a/packages/cli/test/action-runner.test.ts +++ b/packages/cli/test/action-runner.test.ts @@ -48,24 +48,6 @@ describe("action harness-runner", () => { expect(res.result.text).toBe("no match"); }); - it("captures a screenshot via the SDK without invoking the harness", async () => { - fixture = await buildTestHarness({ turns: [] }); - const originalWrite = process.stdout.write.bind(process.stdout); - process.stdout.write = ((_chunk: string | Uint8Array): boolean => true) as typeof process.stdout.write; - try { - const res = await runAction( - { action: "screenshot" }, - { harness: fixture.harness, browserHandle: handleFor(fixture), session: fixture.session }, - { out: "-" }, - ); - expect(res.exitCode).toBe(0); - expect(fixture.provider.callCount()).toBe(0); - expect(fixture.kernel.screenshots).toBe(1); - } finally { - process.stdout.write = originalWrite; - } - }); - it("exits 2 when the provider returns an error", async () => { fixture = await buildTestHarness({ turns: [{ steps: [{ type: "error", message: "boom" }] }], diff --git a/packages/cli/test/cli-executor.test.ts b/packages/cli/test/cli-executor.test.ts new file mode 100644 index 00000000..6cfb791c --- /dev/null +++ b/packages/cli/test/cli-executor.test.ts @@ -0,0 +1,482 @@ +import type { BrowserExecutor, BrowserFindCandidate, BrowserRefState, InternalComputerTranslator as Translator } from "@onkernel/cua-agent"; +import { InternalComputerTranslator } from "@onkernel/cua-agent"; +import type { CuaBrowserAction } from "@onkernel/cua-ai"; +import { mkdtempSync } from "node:fs"; +import { readFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { + deterministicActionFor, + isCoordinatePair, + parseDeterministicArgs, + runDeterministicCommand, + runDeterministicOnHandle, +} from "../src/cli-executor"; +import type { HarnessCliFlags } from "../src/cli-harness"; +import type { CuaBrowserHandle } from "../src/harness-browser"; +import { createFakeKernelEnvironment, type FakeKernelEnvironment } from "./fixtures/fake-kernel"; + +const PROVIDER_ENV_KEYS = [ + "OPENAI_API_KEY", + "ANTHROPIC_API_KEY", + "GOOGLE_API_KEY", + "GEMINI_API_KEY", + "TZAFON_API_KEY", + "YUTORI_API_KEY", +]; + +function baseFlags(overrides: Partial = {}): HarnessCliFlags { + return { + verbose: false, + profileSaveChanges: true, + continueLatest: false, + resumePicker: false, + noSession: false, + noSkills: false, + debugTui: false, + jsonlIncludeDeltas: false, + jsonlIncludeImages: false, + playwright: false, + skillPaths: [], + ...overrides, + }; +} + +interface FakeExecutorState { + actions: CuaBrowserAction[]; + closed: number; + imported: BrowserRefState[]; + exported: number; +} + +interface FakeExecutorScript { + candidates?: BrowserFindCandidate[]; + url?: string; + texts?: Partial>; + failWith?: Error; +} + +const FAKE_REF_STATE: BrowserRefState = { + refCounter: 7, + generations: [["F0", 0]], + refs: [["e7", { backendNodeId: 42, targetId: "F0", frameId: "F0", generation: 0, role: "button", name: "Save", nth: 0, cohort: 1 }]], +}; + +function fakeExecutor(script: FakeExecutorScript = {}): { executor: BrowserExecutor; state: FakeExecutorState } { + const state: FakeExecutorState = { actions: [], closed: 0, imported: [], exported: 0 }; + const executor = { + async execute(action: CuaBrowserAction) { + if (script.failWith) throw script.failWith; + state.actions.push(action); + const text = script.texts?.[action.type]; + return text !== undefined ? [{ type: "browser_text", label: action.type, text }] : []; + }, + async findCandidates(_query: string, _tabId?: string, roles?: ReadonlySet) { + if (script.failWith) throw script.failWith; + const candidates = script.candidates ?? []; + return roles ? candidates.filter((c) => roles.has(c.role)) : candidates; + }, + async currentUrl() { + return script.url ?? ""; + }, + importRefState(refState: BrowserRefState) { + state.imported.push(refState); + }, + exportRefState(): BrowserRefState { + state.exported += 1; + return FAKE_REF_STATE; + }, + close() { + state.closed += 1; + }, + }; + return { executor: executor as unknown as BrowserExecutor, state }; +} + +interface TestSetup { + kernel: FakeKernelEnvironment; + handle: CuaBrowserHandle; + handleCloses: () => number; + createTranslator: (handle: CuaBrowserHandle) => Translator; + state: FakeExecutorState; +} + +function setup(script: FakeExecutorScript = {}): TestSetup { + const kernel = createFakeKernelEnvironment(); + let closes = 0; + const handle: CuaBrowserHandle = { + client: kernel.client, + browser: kernel.browser, + async close() { + closes += 1; + }, + }; + const { executor, state } = fakeExecutor(script); + const createTranslator = (h: CuaBrowserHandle) => + new InternalComputerTranslator({ browser: h.browser, client: h.client, createBrowserExecutor: () => executor }); + return { kernel, handle, handleCloses: () => closes, createTranslator, state }; +} + +let stdoutLines: string[] = []; +let originalWrite: typeof process.stdout.write; +let savedEnv: Record = {}; + +beforeEach(() => { + stdoutLines = []; + originalWrite = process.stdout.write.bind(process.stdout); + process.stdout.write = ((chunk: string | Uint8Array): boolean => { + stdoutLines.push(typeof chunk === "string" ? chunk : Buffer.from(chunk).toString("latin1")); + return true; + }) as typeof process.stdout.write; + savedEnv = {}; + for (const key of [...PROVIDER_ENV_KEYS, "KERNEL_API_KEY"]) { + savedEnv[key] = process.env[key]; + delete process.env[key]; + } +}); + +afterEach(() => { + process.stdout.write = originalWrite; + for (const [key, value] of Object.entries(savedEnv)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } +}); + +describe("deterministicActionFor", () => { + it("routes deterministic subcommands to the executor plane", () => { + expect(deterministicActionFor("url", [])).toBe("url"); + expect(deterministicActionFor("open", ["https://a"])).toBe("open"); + expect(deterministicActionFor("screenshot", [])).toBe("screenshot"); + expect(deterministicActionFor("click", ["10", "20"])).toBe("click"); + expect(deterministicActionFor("click", ["e12"])).toBe("click"); + }); + + it("leaves model-mediated and free-form argv alone", () => { + expect(deterministicActionFor("click", ["3", "dots", "menu"])).toBeUndefined(); + expect(deterministicActionFor("click", ["sign in button"])).toBeUndefined(); + expect(deterministicActionFor("click", ["e12x"])).toBeUndefined(); + expect(deterministicActionFor("click", ["e12", "e13"])).toBeUndefined(); + expect(deterministicActionFor("do", ["open hn"])).toBeUndefined(); + expect(deterministicActionFor("observe", [])).toBeUndefined(); + expect(deterministicActionFor("session", ["list"])).toBeUndefined(); + expect(deterministicActionFor(undefined, [])).toBeUndefined(); + }); +}); + +describe("isCoordinatePair", () => { + it("accepts exactly two integer tokens", () => { + expect(isCoordinatePair(["10", "20"])).toBe(true); + }); + + it("rejects descriptions, partial pairs, and non-integers", () => { + expect(isCoordinatePair(["3", "dots", "menu"])).toBe(false); + expect(isCoordinatePair(["10"])).toBe(false); + expect(isCoordinatePair(["10", "20px"])).toBe(false); + expect(isCoordinatePair([])).toBe(false); + }); +}); + +describe("parseDeterministicArgs", () => { + it("rejects invalid argv before any provisioning", () => { + expect(() => parseDeterministicArgs("open", [], baseFlags())).toThrow("usage: cua open"); + expect(() => parseDeterministicArgs("find", [], baseFlags())).toThrow("usage: cua find"); + expect(() => parseDeterministicArgs("fill", ["query"], baseFlags())).toThrow("usage: cua fill"); + expect(() => parseDeterministicArgs("press", [], baseFlags())).toThrow("usage: cua press"); + expect(() => parseDeterministicArgs("click", ["a", "b"], baseFlags())).toThrow("usage: cua click"); + expect(() => parseDeterministicArgs("snapshot", [], baseFlags({ filter: "everything" }))).toThrow( + "invalid --filter", + ); + }); + + it("rejects extra positionals on url, text, and tabs", () => { + expect(() => parseDeterministicArgs("url", ["extra"], baseFlags())).toThrow("usage: cua url"); + expect(() => parseDeterministicArgs("text", ["extra"], baseFlags())).toThrow("usage: cua text"); + expect(() => parseDeterministicArgs("tabs", ["extra"], baseFlags())).toThrow("usage: cua tabs"); + }); + + it("rejects --filter on subcommands other than snapshot", () => { + expect(() => parseDeterministicArgs("text", [], baseFlags({ filter: "interactive" }))).toThrow( + "--filter only applies to cua snapshot", + ); + }); + + it("accepts the documented forms", () => { + expect(parseDeterministicArgs("open", ["back"], baseFlags())).toEqual({ action: "open", url: "back" }); + expect(parseDeterministicArgs("snapshot", [], baseFlags({ filter: "interactive" }))).toEqual({ + action: "snapshot", + filter: "interactive", + }); + expect(parseDeterministicArgs("fill", ["email", "a@b.c"], baseFlags())).toEqual({ + action: "fill", + query: "email", + value: "a@b.c", + }); + expect(parseDeterministicArgs("click", ["10", "20"], baseFlags())).toEqual({ action: "click", x: 10, y: 20 }); + expect(parseDeterministicArgs("click", ["e12"], baseFlags())).toEqual({ action: "click", ref: "e12" }); + expect(parseDeterministicArgs("fill", ["e12", "a@b.c"], baseFlags())).toEqual({ + action: "fill", + ref: "e12", + value: "a@b.c", + }); + }); + + it("runDeterministicCommand rejects invalid --mode and --native-tool values", async () => { + await expect(runDeterministicCommand("url", [], baseFlags({ mode: "bogus" }))).rejects.toThrow(/invalid --mode/); + await expect(runDeterministicCommand("url", [], baseFlags({ nativeTool: "bogus" }))).rejects.toThrow(/invalid --native-tool/); + }); + + it("runDeterministicCommand surfaces argv errors before touching the Kernel API", async () => { + // KERNEL_API_KEY is unset in this suite: reaching provisioning would + // throw "missing Kernel API key" instead of the usage error. + await expect(runDeterministicCommand("open", [], baseFlags())).rejects.toThrow("usage: cua open"); + }); +}); + +describe("runDeterministicOnHandle", () => { + it("open navigates via CDP and prints ok (no provider keys in env)", async () => { + const t = setup({ texts: { browser_navigate: "Navigated to https://example.test/." } }); + const code = await runDeterministicOnHandle({ action: "open", url: "example.test" }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe("ok\n"); + expect(t.state.actions).toEqual([{ type: "browser_navigate", url: "example.test" }]); + expect(t.state.closed).toBe(1); + expect(t.handleCloses()).toBe(1); + }); + + it("url prints the current URL", async () => { + const t = setup({ url: "https://example.test/page" }); + const code = await runDeterministicOnHandle({ action: "url" }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe("https://example.test/page\n"); + }); + + it("snapshot passes --filter through and prints the tree", async () => { + const t = setup({ texts: { browser_snapshot: 'button "Go" [e1]' } }); + const code = await runDeterministicOnHandle( + { action: "snapshot", filter: "interactive" }, + t.handle, + t.createTranslator, + ); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe('button "Go" [e1]\n'); + expect(t.state.actions).toEqual([{ type: "browser_snapshot", filter: "interactive" }]); + }); + + it("text prints the page text", async () => { + const t = setup({ texts: { browser_text: "hello world" } }); + const code = await runDeterministicOnHandle({ action: "text" }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe("hello world\n"); + }); + + it("tabs prints one line per tab", async () => { + const t = setup({ texts: { browser_list_tabs: 'tab_id AAAA: "One" (https://a)\ntab_id BBBB: "Two" (https://b)' } }); + const code = await runDeterministicOnHandle({ action: "tabs" }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toContain("tab_id AAAA"); + }); + + it("find exits 1 when no candidates match", async () => { + const t = setup({ candidates: [] }); + const code = await runDeterministicOnHandle({ action: "find", query: "missing thing" }, t.handle, t.createTranslator); + expect(code).toBe(1); + expect(stdoutLines.join("")).toBe('not_found no elements matched "missing thing"\n'); + }); + + it("find prints one candidate per line", async () => { + const t = setup({ + candidates: [ + { ref: "e1", role: "button", name: "Search", score: 2 }, + { ref: "e2", role: "link", name: "Search help", score: 1 }, + ], + }); + const code = await runDeterministicOnHandle({ action: "find", query: "search" }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe('button "Search" [e1]\nlink "Search help" [e2]\n'); + }); + + it("fill exits 1 when nothing fillable matches", async () => { + const t = setup({ candidates: [{ ref: "e1", role: "button", name: "Email us", score: 1 }] }); + const code = await runDeterministicOnHandle( + { action: "fill", query: "email", value: "a@b.c" }, + t.handle, + t.createTranslator, + ); + expect(code).toBe(1); + expect(stdoutLines.join("")).toBe('not_found no fillable element matched "email"\n'); + expect(t.state.actions).toEqual([]); + }); + + it("fill exits 1 on a tied top score and lists the matches", async () => { + const t = setup({ + candidates: [ + { ref: "e1", role: "textbox", name: "Email", score: 1 }, + { ref: "e2", role: "textbox", name: "Email confirmation", score: 1 }, + ], + }); + const code = await runDeterministicOnHandle( + { action: "fill", query: "email", value: "a@b.c" }, + t.handle, + t.createTranslator, + ); + expect(code).toBe(1); + expect(stdoutLines.join("")).toBe( + 'not_found ambiguous query "email" (2 matches): textbox "Email", textbox "Email confirmation"\n', + ); + expect(t.state.actions).toEqual([]); + }); + + it("fill fills the unique best fillable match by ref", async () => { + const t = setup({ + candidates: [ + { ref: "e1", role: "button", name: "Email us", score: 3 }, + { ref: "e2", role: "textbox", name: "Email", score: 2 }, + { ref: "e3", role: "textbox", name: "Name", score: 1 }, + ], + }); + const code = await runDeterministicOnHandle( + { action: "fill", query: "email", value: "a@b.c" }, + t.handle, + t.createTranslator, + ); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe('ok filled textbox "Email"\n'); + expect(t.state.actions).toEqual([{ type: "browser_fill", ref: "e2", value: "a@b.c" }]); + }); + + it("fill maps checkbox values to a checked state", async () => { + const t = setup({ candidates: [{ ref: "e1", role: "checkbox", name: "Subscribe", score: 2 }] }); + const code = await runDeterministicOnHandle( + { action: "fill", query: "subscribe", value: "false" }, + t.handle, + t.createTranslator, + ); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe('ok filled checkbox "Subscribe"\n'); + expect(t.state.actions).toEqual([{ type: "browser_fill", ref: "e1", value: false }]); + }); + + it("fill exits 2 on an unrecognized checkbox value", async () => { + const t = setup({ candidates: [{ ref: "e1", role: "checkbox", name: "Subscribe", score: 2 }] }); + const code = await runDeterministicOnHandle( + { action: "fill", query: "subscribe", value: "maybe" }, + t.handle, + t.createTranslator, + ); + expect(code).toBe(2); + expect(stdoutLines.join("")).toContain("error checkbox/radio value must be"); + expect(t.state.actions).toEqual([]); + }); + + it("press dispatches one key chord through the computer batch API", async () => { + const t = setup(); + const code = await runDeterministicOnHandle({ action: "press", keys: ["ctrl", "l"] }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe("ok pressed\n"); + expect(t.kernel.batchCalls).toHaveLength(1); + const body = t.kernel.batchCalls[0]!.body as { actions: Array<{ type: string; press_key?: { keys: string[]; hold_keys?: string[] } }> }; + expect(body.actions).toEqual([{ type: "press_key", press_key: { keys: ["l"], hold_keys: ["Control_L"] } }]); + }); + + it("click dispatches an OS-level click at the coordinates", async () => { + const t = setup(); + const code = await runDeterministicOnHandle({ action: "click", x: 10, y: 20 }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe("ok clicked (10, 20)\n"); + const body = t.kernel.batchCalls[0]!.body as { actions: Array<{ type: string; click_mouse?: { x: number; y: number } }> }; + expect(body.actions[0]!.type).toBe("click_mouse"); + expect(body.actions[0]!.click_mouse).toMatchObject({ x: 10, y: 20 }); + }); + + it("click dispatches a CDP click on the ref", async () => { + const t = setup(); + const code = await runDeterministicOnHandle({ action: "click", ref: "e12" }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe("ok clicked e12\n"); + expect(t.state.actions).toEqual([{ type: "browser_click", ref: "e12" }]); + expect(t.kernel.batchCalls).toHaveLength(0); + }); + + it("click exits 1 when the ref is stale", async () => { + const t = setup({ failWith: new Error("ref e12 is stale or not on the current page. Call snapshot to get fresh refs.") }); + const code = await runDeterministicOnHandle({ action: "click", ref: "e12" }, t.handle, t.createTranslator); + expect(code).toBe(1); + expect(stdoutLines.join("")).toContain("not_found"); + }); + + it("fill fills that element, mapping toggle words to booleans", async () => { + const t = setup(); + expect(await runDeterministicOnHandle({ action: "fill", ref: "e7", value: "a@b.c" }, t.handle, t.createTranslator)).toBe(0); + expect(await runDeterministicOnHandle({ action: "fill", ref: "e8", value: "on" }, t.handle, t.createTranslator)).toBe(0); + expect(t.state.actions).toEqual([ + { type: "browser_fill", ref: "e7", value: "a@b.c" }, + { type: "browser_fill", ref: "e8", value: true }, + ]); + expect(stdoutLines.join("")).toBe("ok filled e7\nok filled e8\n"); + }); + + it("loads persisted ref state before executing and saves it after", async () => { + const t = setup(); + const saved: BrowserRefState[] = []; + const store = { + async load() { + return FAKE_REF_STATE; + }, + async save(state: BrowserRefState) { + saved.push(state); + }, + }; + const code = await runDeterministicOnHandle({ action: "click", ref: "e7" }, t.handle, t.createTranslator, store); + expect(code).toBe(0); + expect(t.state.imported).toEqual([FAKE_REF_STATE]); + expect(saved).toEqual([FAKE_REF_STATE]); + }); + + it("does not touch ref state without a store", async () => { + const t = setup(); + await runDeterministicOnHandle({ action: "click", ref: "e7" }, t.handle, t.createTranslator); + expect(t.state.imported).toEqual([]); + expect(t.state.exported).toBe(0); + }); + + it("screenshot captures via the SDK and writes the file", async () => { + const t = setup(); + const out = join(mkdtempSync(join(tmpdir(), "cua-shot-")), "shot.png"); + const code = await runDeterministicOnHandle({ action: "screenshot", out }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines.join("")).toBe(`${out}\n`); + expect(t.kernel.screenshots).toBe(1); + expect((await readFile(out)).length).toBeGreaterThan(0); + }); + + it("screenshot --out - writes only the PNG bytes to stdout", async () => { + const t = setup(); + const code = await runDeterministicOnHandle({ action: "screenshot", out: "-" }, t.handle, t.createTranslator); + expect(code).toBe(0); + expect(stdoutLines).toHaveLength(1); + expect(stdoutLines[0]!.startsWith("\x89PNG\r\n\x1a\n")).toBe(true); + expect(t.kernel.screenshots).toBe(1); + }); + + it("screenshot exits 2 when capture fails", async () => { + const t = setup(); + const computer = t.kernel.client.browsers.computer as unknown as { captureScreenshot: () => Promise }; + computer.captureScreenshot = async () => { + throw new Error("capture unavailable"); + }; + const code = await runDeterministicOnHandle({ action: "screenshot", out: "-" }, t.handle, t.createTranslator); + expect(code).toBe(2); + expect(stdoutLines.join("")).toBe("error failed to capture screenshot\n"); + }); + + it("exits 2 and still closes executor and handle when the executor throws", async () => { + const t = setup({ failWith: new Error("cdp exploded") }); + const code = await runDeterministicOnHandle({ action: "open", url: "example.test" }, t.handle, t.createTranslator); + expect(code).toBe(2); + expect(stdoutLines.join("")).toBe("error cdp exploded\n"); + expect(t.state.closed).toBe(1); + expect(t.handleCloses()).toBe(1); + }); +}); diff --git a/packages/cli/test/cli-harness-validation.test.ts b/packages/cli/test/cli-harness-validation.test.ts new file mode 100644 index 00000000..c8f20ce3 --- /dev/null +++ b/packages/cli/test/cli-harness-validation.test.ts @@ -0,0 +1,67 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { type HarnessCliFlags, runActionCommand } from "../src/cli-harness"; +import { provisionBrowser } from "../src/harness-browser"; + +vi.mock("../src/harness-browser", () => ({ provisionBrowser: vi.fn() })); + +function flagsWith(overrides: Partial): HarnessCliFlags { + return { + verbose: false, + profileSaveChanges: false, + continueLatest: false, + resumePicker: false, + noSession: true, + noSkills: true, + debugTui: false, + jsonlIncludeDeltas: false, + jsonlIncludeImages: false, + playwright: false, + skillPaths: [], + ...overrides, + }; +} + +describe("mode/native-tool validation before provisioning", () => { + beforeEach(() => { + vi.stubEnv("KERNEL_API_KEY", "test-kernel-key"); + vi.stubEnv("ANTHROPIC_API_KEY", "test-anthropic-key"); + vi.stubEnv("OPENAI_API_KEY", "test-openai-key"); + vi.stubEnv("GOOGLE_API_KEY", "test-google-key"); + }); + + afterEach(() => { + vi.unstubAllEnvs(); + vi.mocked(provisionBrowser).mockClear(); + }); + + it("rejects a native tool whose mode conflicts without provisioning a browser", async () => { + await expect( + runActionCommand("url", [], flagsWith({ + model: "anthropic:claude-opus-4-8", + mode: "hybrid", + nativeTool: "computer_20260701", + })), + ).rejects.toThrow('native tool "computer_20260701" requires mode "computer"; got "hybrid"'); + expect(provisionBrowser).not.toHaveBeenCalled(); + }); + + it("rejects a native tool on a non-anthropic model without provisioning a browser", async () => { + await expect( + runActionCommand("url", [], flagsWith({ + model: "openai:gpt-5.5", + nativeTool: "computer_20260701", + })), + ).rejects.toThrow('native tool "computer_20260701" requires an anthropic model paired with mode "computer"; got provider "openai"'); + expect(provisionBrowser).not.toHaveBeenCalled(); + }); + + it("rejects an unsupported provider/mode pair without provisioning a browser", async () => { + await expect( + runActionCommand("url", [], flagsWith({ + model: "google:gemini-3-flash-preview", + mode: "browser", + })), + ).rejects.toThrow('provider "google" does not support mode "browser" (computer only)'); + expect(provisionBrowser).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/cli/test/harness-named-sessions.test.ts b/packages/cli/test/harness-named-sessions.test.ts new file mode 100644 index 00000000..0446b92b --- /dev/null +++ b/packages/cli/test/harness-named-sessions.test.ts @@ -0,0 +1,119 @@ +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { applyNamedSessionDefaults, type HarnessCliFlags } from "../src/cli-harness"; +import { + listNamedSessions, + type NamedSessionMetadata, + readNamedSession, + recordSessionModel, + updateNamedSessionRuntime, + writeNamedSession, + writeNamedSessionRefs, +} from "../src/harness-named-sessions"; + +const originalXdg = process.env.XDG_DATA_HOME; + +function baseMeta(overrides: Partial = {}): NamedSessionMetadata { + return { name: "foo", kernel_session_id: "ks_123", created_at: Date.now(), ...overrides }; +} + +function baseFlags(overrides: Partial = {}): HarnessCliFlags { + return { + verbose: false, + profileSaveChanges: false, + continueLatest: false, + resumePicker: false, + noSession: false, + noSkills: false, + debugTui: false, + jsonlIncludeDeltas: false, + jsonlIncludeImages: false, + playwright: false, + namedSession: "foo", + skillPaths: [], + ...overrides, + }; +} + +describe("named session model persistence", () => { + beforeEach(() => { + process.env.XDG_DATA_HOME = mkdtempSync(join(tmpdir(), "cua-cli-named-")); + }); + + afterEach(() => { + if (originalXdg === undefined) delete process.env.XDG_DATA_HOME; + else process.env.XDG_DATA_HOME = originalXdg; + }); + + it("records the model/mode/native-tool onto the metadata file", async () => { + await writeNamedSession(baseMeta()); + await recordSessionModel("foo", { model: "anthropic:claude-opus-4-8", mode: "hybrid", native_tool: "computer_20260701" }); + const meta = await readNamedSession("foo"); + expect(meta?.model).toBe("anthropic:claude-opus-4-8"); + expect(meta?.mode).toBe("hybrid"); + expect(meta?.native_tool).toBe("computer_20260701"); + }); + + it("overwrites a previously recorded model on an explicit switch", async () => { + await writeNamedSession(baseMeta({ model: "openai:gpt-5.5" })); + await recordSessionModel("foo", { model: "anthropic:claude-opus-4-8", mode: "hybrid" }); + const meta = await readNamedSession("foo"); + expect(meta?.model).toBe("anthropic:claude-opus-4-8"); + expect(meta?.mode).toBe("hybrid"); + }); + + it("is a no-op for an unknown session", async () => { + await recordSessionModel("missing", { model: "openai:gpt-5.5" }); + expect(await readNamedSession("missing")).toBeUndefined(); + }); + + it("patches individual runtime fields without clobbering the rest", async () => { + await writeNamedSession(baseMeta({ model: "openai:gpt-5.5", mode: "computer", native_tool: "computer_20260701" })); + + await updateNamedSessionRuntime("foo", { mode: "browser" }); + let meta = await readNamedSession("foo"); + expect(meta?.mode).toBe("browser"); + expect(meta?.model).toBe("openai:gpt-5.5"); + expect(meta?.native_tool).toBe("computer_20260701"); + + await updateNamedSessionRuntime("foo", { model: "anthropic:claude-opus-4-8" }); + meta = await readNamedSession("foo"); + expect(meta?.model).toBe("anthropic:claude-opus-4-8"); + expect(meta?.mode).toBe("browser"); + }); + + it("defaults flags from the stored session model when -m is omitted", () => { + const meta = baseMeta({ model: "anthropic:claude-opus-4-8", mode: "hybrid", native_tool: "computer_20260701" }); + const flags = applyNamedSessionDefaults(baseFlags(), meta); + expect(flags.model).toBe("anthropic:claude-opus-4-8"); + expect(flags.mode).toBe("hybrid"); + expect(flags.nativeTool).toBe("computer_20260701"); + }); + + it("keeps explicit flags over stored session values", () => { + const meta = baseMeta({ model: "anthropic:claude-opus-4-8", mode: "hybrid" }); + const flags = applyNamedSessionDefaults(baseFlags({ model: "openai:gpt-5.5", mode: "browser" }), meta); + expect(flags.model).toBe("openai:gpt-5.5"); + expect(flags.mode).toBe("browser"); + }); + + it("excludes refs sidecar files from listNamedSessions", async () => { + await writeNamedSession(baseMeta()); + await writeNamedSessionRefs("foo", { refCounter: 3, generations: [], refs: [] }); + const sessions = await listNamedSessions(); + expect(sessions).toHaveLength(1); + expect(sessions[0]?.name).toBe("foo"); + }); + + it("skips metadata entries missing required fields", async () => { + await writeNamedSession(baseMeta()); + const dir = join(process.env.XDG_DATA_HOME!, "cua", "named-sessions"); + writeFileSync(join(dir, "bogus.json"), JSON.stringify({ unrelated: true })); + writeFileSync(join(dir, "no-age.json"), JSON.stringify({ name: "no-age", kernel_session_id: "k1" })); + const sessions = await listNamedSessions(); + expect(sessions).toHaveLength(1); + expect(sessions[0]?.name).toBe("foo"); + }); +}); diff --git a/skills/cua-cli/SKILL.md b/skills/cua-cli/SKILL.md index 43e5d2ed..c45dea7a 100644 --- a/skills/cua-cli/SKILL.md +++ b/skills/cua-cli/SKILL.md @@ -1,34 +1,58 @@ --- name: cua-cli -description: Drive a Kernel cloud browser from the shell using the `cua` CLI. Use this skill when you need to open URLs, click elements, type into fields, take screenshots, or chain multi-step browser tasks across shell calls. Supports named sessions for stateful workflows. +description: Drive a Kernel cloud browser from the shell using the `cua` CLI. Use this skill when you need to open URLs, click elements, type into fields, inspect pages, fill forms, take screenshots, or chain multi-step browser tasks across shell calls. Supports named sessions for stateful workflows. --- # cua-cli -`cua` is a single-binary CLI that drives a real Chrome session running in Kernel. It's designed for agentic use: each subcommand returns a one-line result on stdout and a deterministic exit code, so you can chain calls together and parse the output. +`cua` is a single-binary CLI that drives a real Chrome session running in Kernel. It's designed for agentic use: each subcommand returns a stable result on stdout and a deterministic exit code (0 ok, 1 not_found, 2 error), so you can chain calls together and parse the output. ## One-shot subcommands Each call below provisions a fresh Kernel browser by default, runs the action, and tears the browser down. Use `-s ` (see "Named sessions" below) to keep state across calls. +### Model-free subcommands + +These run directly against the browser (CDP or OS input) — no LLM involved, no model API key needed, only `KERNEL_API_KEY`. + +| Subcommand | What it does | Stdout | Exit code | +| --- | --- | --- | --- | +| `cua open ` | Navigate via CDP; `back`/`forward` walk history. | `ok` | 0 ok, 2 error | +| `cua url` | Print the active tab's URL. | the URL | 0 ok, 2 error | +| `cua snapshot [--filter interactive]` | Print the page's accessibility tree with element refs like `[e12]`. `--filter interactive` keeps only interactive elements. | the tree (multi-line) | 0 ok, 2 error | +| `cua find ""` | Lexically score elements against the query, best first. | one match per line: `role "name" [eN]` (the quoted name is omitted when the element has none; role falls back to `node`) | 0 ok, 1 not_found, 2 error | +| `cua text` | Print the page's visible text (`innerText`). | the text (multi-line) | 0 ok, 2 error | +| `cua fill ""` | Set a form field's value. With a ref (`e12` from `snapshot`/`find`) it targets that exact element. With a query it finds the unique best-matching form field (textbox, searchbox, combobox, checkbox, radio, listbox, spinbutton); exit 1 with the tied matches listed if the query is ambiguous — tighten it and retry. For checkbox/radio pass `true\|false\|checked\|unchecked\|on\|off` (query form also accepts `1\|0`). `fill` leaves the field focused, so a following `cua press Return` submits the form. | `ok filled ""` (query) or `ok filled e12` (ref) | 0 ok, 1 not_found, 2 error | +| `cua press [...]` | Send one key chord (e.g. `cua press ctrl l`, `cua press Return`). | `ok pressed` | 0 ok, 2 error | +| `cua click ` | OS-level click at viewport coordinates. Exactly two integer arguments. | `ok clicked (x, y)` | 0 ok, 2 error | +| `cua click ` | CDP click on an element ref from `snapshot`/`find`, e.g. `cua click e12`. Any other single `click` argument routes to the model-mediated `click` below. | `ok clicked e12` | 0 ok, 1 not_found (stale ref — re-snapshot), 2 error | +| `cua tabs` | List open tabs. | one line per tab: `tab_id XXXX: "title" (url)` | 0 ok, 2 error | +| `cua screenshot [--out ]` | Save a PNG (default `screenshot.png`). `--out -` writes the bytes to stdout. | the saved path; with `--out -`, stdout is exactly the PNG bytes (safe to pipe) | 0 ok, 2 error | + +**Element refs span invocations within a named session.** Refs printed by `snapshot`/`find` (`[e12]`) are persisted per `-s` session, so `cua -s x snapshot` then `cua -s x click e12` works. Refs self-heal across in-page DOM changes when the element is still unambiguous, but any navigation — including reloading the same URL — invalidates them; the command then exits 1 with a stale-ref message — re-run `snapshot` and use a fresh ref. Without `-s` there is no shared browser, so refs from a previous invocation are meaningless. + +### Model-mediated subcommands + +These resolve a natural-language description with an LLM, so they need the model provider's API key (e.g. `OPENAI_API_KEY` for the default model). + | Subcommand | What it does | Stdout | Exit code | | --- | --- | --- | --- | -| `cua open ` | Navigate to a URL via the address bar. | `ok` | 0 ok, 2 error | | `cua click ""` | Find the element matching the visible, natural-language description and click it. | `ok clicked (x, y)` or `not_found ` | 0 ok, 1 not_found, 2 error | | `cua type "" ""` | Focus the field matching the visible, natural-language description and type text. | `ok typed` or `not_found ` | 0 ok, 1 not_found, 2 error | -| `cua press [...]` | Send a key combo (e.g. `cua press ctrl l`, `cua press Return`). | `ok pressed` | 0 ok, 2 error | -| `cua url` | Read and print the current URL. | the URL | 0 ok, 2 error | | `cua observe ["question"]` | Describe the page; optionally answer a question. | the description | 0 ok, 2 error | -| `cua screenshot --out ` | Save a PNG. `--out -` writes the bytes to stdout. | the path or `(stdout)` | 0 ok, 2 error | | `cua do ""` | Open-ended; let the agent plan and act. Bound by `--max-steps` (default 3). | the assistant's final text | 0 ok, 2 error | Useful flags: -- `-m ` — pick the LLM (default `gpt-5.5`). Other good picks: - `claude-opus-4-7`, `gemini-3-flash-preview`, `n1.5-latest`. +- `-m ` — pick the LLM for model-mediated subcommands (default `gpt-5.5`). + Other good picks: `claude-opus-4-7`, `gemini-3-flash-preview`, `n1.5-latest`. - `cua models` — list supported `-m` values and their providers; filter - with `cua models -p openai|anthropic|gemini|yutori`. + with `cua models -p openai|anthropic|google|yutori|tzafon` (`gemini` is + accepted as an alias for `google`). Model refs print as `provider:model` + (e.g. `google:gemini-3-flash-preview`); `-m` accepts either the full ref or + a bare model id that matches exactly one entry. - `--max-steps ` — bound the agent loop on `cua do` (default 3). +- `--filter interactive` — restrict `cua snapshot` to interactive elements. - `--profile ` — load a Kernel browser profile for cookies / storage. Existing ids or names are reused; a non-id name is created if it does not exist. Use this whenever logged-in state or other persisted browser @@ -45,30 +69,47 @@ named session first: ```bash cua --profile github session start login # creates a Kernel browser, prints `name=login` cua -s login open https://github.com/login -cua -s login type "email field" "$EMAIL" -cua -s login type "password field" "$PASSWORD" -cua -s login click "Sign in" +cua -s login fill "email field" "$EMAIL" # model-free +cua -s login fill "password field" "$PASSWORD" # model-free +cua -s login click "Sign in" # model-mediated cua -s login url # prints the post-login URL cua session stop login # tears down the Kernel browser ``` -Inspect: +Inspecting a page mid-flow, entirely model-free: + +```bash +cua -s login snapshot --filter interactive # what can I interact with? +cua -s login find "sign in button" # score elements against a query +cua -s login click e12 # click a ref from the snapshot/find output +cua -s login fill e7 "$EMAIL" # fill a ref directly +cua -s login text # read the page's visible text +cua -s login tabs # list open tabs +``` + +Inspect sessions: ```bash cua session list # tab-formatted: NAME, KERNEL_ID, AGE, LIVE_URL cua session show login # full JSON metadata ``` +`cua session show ` and `cua session stop ` exit 1 when the named +session does not exist (`no named session ""`); other session failures +exit 2. + Pass `--profile` when starting the named session; later `cua -s login ...` calls attach to that same browser, so they do not need the profile flag. -Liveness: Kernel browsers can time out from inactivity even between your calls. If `cua -s ...` returns `error session "" is no longer alive on Kernel ...`, run `cua session stop && cua --profile github session start ` to provision a fresh one with the same persisted profile. +Liveness: Kernel browsers can time out from inactivity even between your calls. If `cua -s ...` fails with `error: named session "" is no longer alive on Kernel ...` (printed to stderr, exit 2), run `cua session stop && cua --profile github session start ` to provision a fresh one with the same persisted profile. ## Session transcripts -Every `cua --print`, interactive TUI, and `cua -s ` invocation appends to -a JSONL transcript. Treat the on-disk directory name as internal; find the -exact path instead of trying to reconstruct it: +Every `cua --print`, interactive TUI, and model-mediated `cua -s ` +invocation appends to a JSONL transcript. Model-free subcommands do not touch +transcripts — there is no model conversation to record. Treat the on-disk +directory name as internal; find the exact path instead of trying to +reconstruct it: ```bash cua -v --print "..." # stderr includes: [cua] session= @@ -77,10 +118,10 @@ cua session show login | jq -r .transcript_path The default root is `$XDG_DATA_HOME/cua/sessions` or `~/.local/share/cua/sessions`. For named sessions, `transcript_path` appears -after the first `cua -s ...`, `cua -s --print ...`, or TUI attach -records a transcript. +after the first model-mediated `-s` call (`click ""`, `type`, `observe`, +`do`, `--print`, or a TUI attach). -Each line is a JSON object with one of these `role` values: `user`, `assistant`, `toolResult`. There's also a custom `cua-browser` entry written once per session with `kernel_session_id` / `live_url` / `profile_id`. +Each line is a pi `SessionManager` record with a top-level `type` of `session`, `message`, or `custom`. Conversation entries have `type: "message"` with the role nested at `.message.role` (`user`, `assistant`, or `toolResult`). There's also a `type: "custom"` entry with `customType: "cua-browser"` written once per session whose `data` carries `sessionId` / `liveUrl` (and `profileId` when a profile is loaded). Use `cua --print -o jsonl "..."` only when you need live stdout events while a run is happening. That stream is a compact event feed (`tool_call`, @@ -102,7 +143,8 @@ running until you Ctrl+C. ## Don't forget -- Subcommands that need an element or field description (`click`, `type`) match SEMANTICALLY, not by selector. Use natural-language descriptions of what the user would see on screen. +- Prefer the model-free subcommands when they can do the job — they're faster, cheaper, and deterministic. Reach for `click ""` / `type` / `do` only when you need semantic matching or planning. +- Subcommands that take an element or field description (`click ""`, `type`) match SEMANTICALLY, not by selector. Use natural-language descriptions of what the user would see on screen. `fill` matches lexically against accessible role/name — use the words from `snapshot`/`find` output. - Browser viewport defaults to 1920x1080. - Keyboard navigation (`Page_Down`, `Home`, arrow keys via `cua press`) is more reliable than mouse-wheel scrolling. - For multi-step state, you almost always want `-s `. Without it, the second subcommand can't see anything the first one did.