Add action-plane modes (computer/browser/hybrid) and Anthropic native computer/browser tools

docs/architecture.md

@@ -92,6 +92,53 @@ provider conditionals. A new provider difference is a new or extended
 `CuaRuntimeSpec`/`CuaPayloadContext` field plus provider code in
 `@onkernel/cua-ai` — never a branch in `@onkernel/cua-agent`.
 
+## Action planes and modes
+
+The canonical action vocabulary is split into two planes, delineated in code
+under `packages/ai/src/actions/`:
+
+- **Computer plane** (`actions/computer.ts`) — real OS-level input against the browser
+  VM: mouse, keyboard, display capture, executed through Kernel's
+  `browsers.computer` REST API. Coordinates are pixels in the OS screenshot
+  frame.
+- **Browser plane** (`actions/browser.ts`, ids prefixed `browser_`) — CDP-driven page
+  tools: accessibility snapshots with element refs, element-targeted
+  interaction, navigation, tabs, viewport screenshots. Executed by
+  `packages/agent/src/translator/browser.ts` (`BrowserExecutor`) over a raw CDP websocket
+  (`translator/cdp.ts`) to the browser's `cdp_ws_url` — no Playwright.
+  Coordinates, where used, are viewport pixels.
+
+A `CuaMode` selects which plane(s) the model sees:
+
+| mode | tools | coordinate frame |
+| --- | --- | --- |
+| `computer` (default) | computer actions under their canonical ids (`click`, `screenshot`, …) | OS screenshot pixels |
+| `browser` | browser actions with the `browser_` prefix stripped (`snapshot`, `click`, …) plus `wait` | none for refs; viewport pixels where coordinates are allowed |
+| `hybrid` | both planes, one tool per capability: computer actions as `computer_*`, browser reads/element-writes as `browser_*` (ref-only) | OS screenshot pixels — the single live frame |
+
+Hybrid deduplicates capabilities: navigation and tabs live on the browser plane,
+pointer/keyboard input and the (only) screenshot live on the computer plane, and
+hybrid browser tools take element refs only so exactly one coordinate frame exists.
+Element refs are snapshot-scoped (`e12`); a stale ref resolves to an error
+string that tells the model to re-snapshot.
+
+The mode is set at construction (`mode` on `CuaAgent`/`CuaAgentHarness`,
+`--mode` in the CLI) and can be switched at runtime with `setMode()` (the
+TUI's `/mode` command), which refreshes CUA-owned tools and the default
+system prompt.
+
+**Native tools.** `resolveCuaRuntimeSpec(model, { nativeTool })` drives an
+Anthropic model through its provider-defined tool schema instead of the
+canonical function tools: `computer_20260701` pairs with `computer` mode and
+`browser_20260701` with `browser` mode (mismatches throw, mirroring the API's
+own rejection of mixed frames). The spec routes the model to a CUA-owned api
+id; the registered `anthropic` provider dispatches it to pi's builtin
+`anthropic-messages` transport with the tool's `anthropic-beta` header, an
+`onPayload` hook swaps the placeholder tool for the native declaration, and
+`providers/anthropic/native.ts` maps incoming `tool_use` inputs onto the same
+canonical actions the mode uses — so canonical vs native is purely a wire
+format difference over one execution path.
+
 ## Layers
 
 `cua` is a thin TypeScript monorepo on top of the

packages/agent/CHANGELOG.md

@@ -1,5 +1,35 @@
 # Changelog
 
+## 0.5.0 - 2026-07-09
+
+Adds the browser action plane and runtime mode switching. Breaking: the
+`computerUseExtra` option is removed — the `computer_use_extra` navigation
+helper is always registered.
+
+- New `BrowserExecutor`: drives the browser plane over CDP. Accessibility
+  snapshots with element refs (`[e12]`), node states
+  (checked/expanded/disabled/value/…), and cursor:pointer clickable hints
+  for elements with no interactive ARIA role; iframe and OOPIF stitching
+  with per-frame session-aware refs; StaticText dedupe and wrapper
+  collapsing; an unchanged-snapshot short-circuit; lexical `find`, `fill`,
+  CDP navigation and tab management; and a JavaScript dialog guard. Refs invalidate on real
+  navigations (`Page.frameNavigated`), self-heal via (role, name, nth) when
+  the page changes but the element is still unambiguous, and the ref table is
+  bounded (per-target cap, generation sweeps). `exportRefState()` /
+  `importRefState()` persist refs across processes against the same browser.
+- `CuaAgent` and `CuaAgentHarness` accept `mode` (`"computer"` | `"browser"`
+  | `"hybrid"`) and `nativeTool`, and support runtime plane switching via
+  `setMode()` / `getMode()`. Mode switches preserve the requested activation
+  state of surviving tools and keep the translator — CDP connection, tabs,
+  and element refs — alive; the translator is only rebuilt when a model
+  switch changes the provider's coordinate system or screenshot transform.
+  Both switches roll back cleanly on failure.
+- Post-action grounding captures and the navigation helper are mode-aware:
+  browser mode grounds on the viewport and routes navigation through CDP
+  (browser and hybrid modes both route `computer_use_extra` navigation over
+  the browser plane so refs invalidate correctly).
+- Updated `@onkernel/cua-ai` to 0.5.0.
+
 ## 0.4.0 - 2026-07-07
 
 Breaking: follows pi-agent-core 0.80's `Models`-based harness.

packages/agent/README.md

@@ -97,7 +97,6 @@ Both classes mirror pi constructor shapes and behavior, with minimal additions:
 - `client` (Kernel SDK client)
 - CUA model refs (`"provider:model"`) accepted where pi expects a concrete model
 - `extraTools` to add your own pi tools alongside the built-in browser tools
-- `computerUseExtra: true` to let the model use a small navigation helper
 - `playwright: true` to let the model run Playwright/TypeScript against the
   live browser session
 
@@ -122,7 +121,7 @@ or handing off to another service while it also controls the browser.
 
 Not every provider's native computer-use vocabulary includes browser
 navigation — some models can click and type but have no direct way to open a
-URL or go back. `computerUseExtra: true` adds `computer_use_extra`, a
+URL or go back. The classes therefore always add `computer_use_extra`, a
 provider-neutral escape hatch exposing `goto`, `back`, `forward`, and `url`
 so navigation works uniformly regardless of which model is driving.
 

packages/agent/examples/anthropic-native-smoke.ts

@@ -0,0 +1,61 @@
+// Smoke-test the mode/native-tool matrix against a live Kernel browser:
+//
+//   MODEL_REF=anthropic:claude-opus-4-8 CONFIG=native-browser tsx examples/anthropic-native-smoke.ts
+//
+// CONFIG selects the runtime shape:
+//   computer (default)  canonical computer-plane (OS input) tools
+//   browser             canonical browser-plane (CDP page) tools
+//   hybrid              both planes, deduplicated
+//   native-computer     Anthropic computer_20260701 (requires the computer-use beta)
+//   native-browser      Anthropic browser_20260701 (requires the browser-use beta)
+import Kernel from "@onkernel/sdk";
+import { requireCuaEnvApiKeyForModel, type CuaModelRef, type CuaMode, type CuaNativeToolSpec } from "@onkernel/cua-ai";
+import { CuaAgent } from "../src/index";
+import { logAgentEvent, logAssistant } from "./shared/logging";
+
+const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "anthropic:claude-opus-4-8";
+const config = process.env.CONFIG ?? "computer";
+
+const CONFIGS: Record<string, { mode?: CuaMode; nativeTool?: CuaNativeToolSpec }> = {
+	computer: { mode: "computer" },
+	browser: { mode: "browser" },
+	hybrid: { mode: "hybrid" },
+	"native-computer": { nativeTool: { type: "computer_20260701", enable_zoom: true } },
+	"native-browser": { nativeTool: { type: "browser_20260701" } },
+};
+
+const PROMPT = [
+	"Navigate to https://example.com, read the page, and tell me:",
+	"1) the main heading text",
+	"2) the text of the link on the page",
+	"Then follow that link and tell me the title of the page you land on.",
+].join("\n");
+
+async function main(): Promise<void> {
+	const runtime = CONFIGS[config];
+	if (!runtime) throw new Error(`unknown CONFIG "${config}" (expected: ${Object.keys(CONFIGS).join(" | ")})`);
+	const kernelApiKey = process.env.KERNEL_API_KEY;
+	if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required");
+	requireCuaEnvApiKeyForModel(modelRef);
+	const client = new Kernel({ apiKey: kernelApiKey });
+	const browser = await client.browsers.create({ stealth: true });
+
+	try {
+		const agent = new CuaAgent({
+			browser,
+			client,
+			...runtime,
+			initialState: { model: modelRef },
+		});
+		agent.subscribe(logAgentEvent);
+
+		console.log(`running config=${config} model=${modelRef} live_view=${browser.browser_live_view_url}`);
+		await agent.prompt(PROMPT);
+		const assistant = [...agent.state.messages].reverse().find((message) => message.role === "assistant");
+		logAssistant(assistant?.role === "assistant" ? assistant : undefined);
+	} finally {
+		await client.browsers.deleteByID(browser.session_id);
+	}
+}
+
+void main();

packages/agent/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@onkernel/cua-agent",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Kernel browser computer-use Agent and AgentHarness classes built on pi-agent-core",
   "license": "MIT",
   "type": "module",
@@ -42,7 +42,7 @@
   "dependencies": {
     "@earendil-works/pi-agent-core": "0.80.3",
     "@earendil-works/pi-ai": "0.80.3",
-    "@onkernel/cua-ai": "0.4.0",
+    "@onkernel/cua-ai": "0.5.0",
     "@onkernel/sdk": "0.49.0",
     "sharp": "^0.34.5"
   },