Tighten cua-agent design seams: typed translator, declared channels

docs/architecture.md

@@ -45,10 +45,28 @@ someone who wants to read the code, contribute, or fork.
   - executing canonical CUA tool calls against Kernel browsers
   - typed executor coverage and translator integration
 
-In practice this means any new provider quirk should be implemented in
-`@onkernel/cua-ai` and surfaced through provider-neutral runtime specs.
-`@onkernel/cua-agent` should consume that spec without explicit
-provider-specific conditionals.
+The boundary is a single data seam. Every provider difference arrives in
+`@onkernel/cua-agent` as data through `CuaRuntimeSpec` — `toolDefinitions`,
+`toolExecutors`, `defaultSystemPrompt`, `coordinateSystem`, `screenshot`, and
+`onPayload` — resolved per model by `resolveCuaRuntimeSpec()`. In the other
+direction, the agent supplies capabilities back to provider middleware through
+`CuaPayloadContext` (`keepToolNames`, `getScreenshot`): the provider hook
+decides *whether and how* to use a capability (policy), the agent decides
+*how it is performed* against the Kernel browser (mechanism).
+
+The invariant: `packages/agent/src` contains no provider names and no
+provider conditionals. A new provider difference is a new or extended
+`CuaRuntimeSpec`/`CuaPayloadContext` field plus provider code in
+`@onkernel/cua-ai` — never a branch in `@onkernel/cua-agent`. The grep test
+is literal: searching agent `src/` for a provider name should only ever hit
+doc comments.
+
+One deliberate exception to "push it upstream": generic model imprecision.
+Models of every provider are loose about things like key naming
+(`ctrl`/`cmd`/`ArrowLeft`/word-form punctuation), so the agent-side
+translator absorbs that nondeterminism when mapping canonical actions to
+Kernel's X11 key vocabulary. That is corrective plumbing for model output in
+general, not provider policy, and it stays in `@onkernel/cua-agent`.
 
 ## Layers
 

packages/agent/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## 0.3.3 - 2026-06-12
+
+- The action translator now consumes the canonical `CuaAction` union with an
+  exhaustive switch. Malformed action shapes fail loudly instead of silently
+  coercing (previously e.g. a click at 0,0); the documented mouse-button
+  coercion to `"left"` is unchanged.
+- `prepareNextTurn` no longer rebuilds the turn context on every turn: it
+  keeps stock pi behavior until a user hook returns an update or a mid-run
+  model assignment requires a refresh.
+- One translator instance per runtime is shared between the executor tools
+  and the provider screenshot capability.
+- The `CuaAgentHarness` README quickstart showcases session-backed turns and
+  mid-session model switching; `computerUseExtra` is documented with its
+  rationale.
+- Update the `@onkernel/cua-ai` dependency to 0.3.0.
+
 ## 0.3.2 - 2026-06-11
 
 - Update the `@onkernel/cua-ai` dependency to 0.2.2.

packages/agent/README.md

@@ -37,11 +37,17 @@ await agent.prompt("Open news.ycombinator.com and summarize the top story.");
 
 ## Quick Start (`CuaAgentHarness`)
 
+`prompt()` returns the turn's final assistant message, and every turn is
+persisted to the session — later prompts see the full transcript. Runtime
+config like the model can change between turns (or even mid-turn, applying at
+the next provider request):
+
 ```ts
 import { CuaAgentHarness, InMemorySessionRepo, NodeExecutionEnv } from "@onkernel/cua-agent";
+import type { AssistantMessage } from "@onkernel/cua-ai";
 
 const sessionRepo = new InMemorySessionRepo();
-const session = await sessionRepo.create({ id: "example" });
+const session = await sessionRepo.create({ id: "research" });
 
 const harness = new CuaAgentHarness({
   browser,
@@ -51,22 +57,26 @@ const harness = new CuaAgentHarness({
   session,
 });
 
-const response = await harness.prompt("Open example.com and tell me the current URL.");
-const branch = await session.getBranch();
-const lastAssistant = [...branch]
-  .reverse()
-  .flatMap((entry) =>
-    entry.type === "message" && entry.message.role === "assistant" ? [entry.message] : [],
-  )[0];
-const assistant = lastAssistant ?? response;
-const assistantText = assistant.content
-  .flatMap((block) => (block.type === "text" ? [block.text] : []))
-  .join("")
-  .trim();
-console.log("assistant stopReason:", assistant.stopReason);
-console.log("assistant text:", assistantText || "(no text)");
+const textOf = (message: AssistantMessage) =>
+  message.content.flatMap((block) => (block.type === "text" ? [block.text] : [])).join("").trim();
+
+// Turn 1: a session-backed prompt.
+const first = await harness.prompt("Open example.com and describe what you see.");
+console.log(textOf(first));
+
+// Swap providers mid-session; CUA tools and the default prompt refresh to match.
+await harness.setModel("anthropic:claude-opus-4-7");
+
+// Turn 2 continues the same transcript on the new model.
+const second = await harness.prompt("Open the most relevant link from what you found.");
+console.log(textOf(second));
 ```
 
+While a turn is running, `steer()` injects course corrections, `followUp()`
+queues the next instruction, and `subscribe()` streams the underlying agent
+events. `compact()` and session branching are available for long-running
+transcripts — see the pi-agent-core docs for the full harness lifecycle.
+
 Use `CuaAgent` when you want direct pi `Agent` control: raw message state,
 lifecycle events, custom streaming, and explicit prompt/continue/queue control.
 Reach for the harness shape when you want an app layer around the loop:
@@ -108,9 +118,11 @@ computer-use tools. This is useful when the model needs to call
 application-specific code, such as looking up a record, writing a database row,
 or handing off to another service while it also controls the browser.
 
-`computerUseExtra: true` adds the `computer_use_extra` tool. Use it when you
-want one compact helper for common browser navigation/read operations:
-`goto`, `back`, `forward`, and `url`.
+Not every provider's native computer-use vocabulary includes browser
+navigation — some models can click and type but have no direct way to open a
+URL or go back. `computerUseExtra: true` adds `computer_use_extra`, a
+provider-neutral escape hatch exposing `goto`, `back`, `forward`, and `url`
+so navigation works uniformly regardless of which model is driving.
 
 ### Model Switching
 

packages/agent/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@onkernel/cua-agent",
-  "version": "0.3.2",
+  "version": "0.3.3",
   "description": "Kernel browser computer-use Agent and AgentHarness classes built on pi-agent-core",
   "license": "MIT",
   "type": "module",
@@ -42,7 +42,7 @@
   "dependencies": {
     "@earendil-works/pi-agent-core": "0.79.1",
     "@earendil-works/pi-ai": "0.79.1",
-    "@onkernel/cua-ai": "0.2.2",
+    "@onkernel/cua-ai": "0.3.0",
     "@onkernel/sdk": "0.49.0",
     "sharp": "^0.34.5"
   },

packages/agent/src/agent.ts

@@ -13,21 +13,21 @@ import {
 	type Api,
 	CUA_NAVIGATION_TOOL_NAME,
 	type CuaModelRef,
+	type CuaRuntimeSpec,
+	type CuaSimpleStreamOptions,
 	getCuaEnvApiKey,
 	type Model,
 	resolveCuaRuntimeSpec,
 	type SimpleStreamOptions,
 	streamSimple,
 } from "@onkernel/cua-ai";
 import type Kernel from "@onkernel/sdk";
-import { createCuaComputerTools } from "./tools";
+import { buildCuaComputerTools } from "./tools";
 import { InternalComputerTranslator, type KernelBrowser } from "./translator/translator";
 
 /** A CUA model reference string or a concrete pi model object. */
 type CuaRuntimeInput = CuaModelRef | Model<Api>;
 
-type CuaRuntimeSpec = ReturnType<typeof resolveCuaRuntimeSpec>;
-
 /**
  * Agent state exposed by {@link CuaAgent}.
  *
@@ -132,14 +132,13 @@ class CuaRuntimeController {
 
 	tools(): AgentTool[] {
 		return [
-			...createCuaComputerTools({
-				browser: this.options.browser,
-				client: this.options.client,
-				toolExecutors: this.runtimeSpec.toolExecutors,
-				coordinateSystem: this.runtimeSpec.coordinateSystem,
-				screenshot: this.runtimeSpec.screenshot,
-				computerUseExtra: this.options.computerUseExtra,
-			}),
+			...buildCuaComputerTools(
+				{
+					toolExecutors: this.runtimeSpec.toolExecutors,
+					computerUseExtra: this.options.computerUseExtra,
+				},
+				this.translator,
+			),
 			...(this.options.extraTools ?? []),
 		];
 	}
@@ -190,7 +189,9 @@ async function getCuaEnvApiKeyAndHeaders(model: Model<Api>): Promise<{ apiKey: s
 export class CuaAgent extends Agent {
 	private readonly runtime: CuaRuntimeController;
 	private readonly ownsSystemPrompt: boolean;
+	private runtimeDirty = false;
 	private stateProxy?: CuaAgentState;
+	private stateProxyTarget?: AgentState;
 
 	constructor(options: CuaAgentOptions) {
 		const {
@@ -213,11 +214,11 @@ export class CuaAgent extends Agent {
 			onPayload,
 		});
 		const wrappedStreamFn: StreamFn = (model, context, streamOptions) => {
-			const optionsWithCuaRuntime = {
+			const optionsWithCuaRuntime: CuaSimpleStreamOptions = {
 				...streamOptions,
 				onPayload: runtime.onPayload(),
 				keepToolNames: runtime.keepToolNames(),
-			} as SimpleStreamOptions & { keepToolNames?: string[] };
+			};
 			return (streamFn ?? streamSimple)(model, context, optionsWithCuaRuntime);
 		};
 
@@ -236,15 +237,19 @@ export class CuaAgent extends Agent {
 		this.runtime = runtime;
 		this.ownsSystemPrompt = initialState.systemPrompt === undefined;
 		/**
-		 * pi calls `prepareNextTurn` between provider requests. Wrapping it lets CUA
-		 * honor any user-provided turn update while also refreshing provider-specific
-		 * defaults if that update changes the model.
+		 * pi's loop only re-reads model/tools/prompt between provider requests
+		 * through `prepareNextTurn`. The wrapper stays pass-through (returning
+		 * `undefined`, i.e. stock pi behavior) until either the user hook returns
+		 * an update or a mid-run model assignment marks the CUA runtime dirty —
+		 * only then is a turn update built from current state.
 		 */
 		this.prepareNextTurn = async (signal: AbortSignal | undefined) => {
 			const update = await prepareNextTurn?.(signal);
 			if (update?.model) {
 				this.applyRuntime(update.model as CuaRuntimeInput);
 			}
+			if (!update && !this.runtimeDirty) return undefined;
+			this.runtimeDirty = false;
 
 			const state = super.state;
 			const context = update?.context ?? {
@@ -271,14 +276,16 @@ export class CuaAgent extends Agent {
 	 * and payload hooks for the selected provider.
 	 */
 	override get state(): CuaAgentState {
-		if (!this.stateProxy) {
-			this.stateProxy = new Proxy(super.state, {
-				set: (target, prop, value, receiver) => {
+		const target = super.state;
+		if (!this.stateProxy || this.stateProxyTarget !== target) {
+			this.stateProxyTarget = target;
+			this.stateProxy = new Proxy(target, {
+				set: (proxied, prop, value, receiver) => {
 					if (prop === "model") {
 						this.applyRuntime(value as CuaRuntimeInput);
 						return true;
 					}
-					return Reflect.set(target, prop, value, receiver);
+					return Reflect.set(proxied, prop, value, receiver);
 				},
 			}) as CuaAgentState;
 		}
@@ -287,6 +294,7 @@ export class CuaAgent extends Agent {
 
 	private applyRuntime(model: CuaRuntimeInput): void {
 		this.runtime.setModel(model);
+		this.runtimeDirty = true;
 		const state = super.state;
 		state.model = this.runtime.model;
 		state.tools = this.runtime.tools();

packages/agent/src/tools.ts

@@ -43,11 +43,18 @@ type NavigationExecutorSpec = { kind: "navigation"; definition: Tool };
 type ComputerExecutorSpec = CuaToolExecutorSpec | NavigationExecutorSpec;
 
 export function createCuaComputerTools(args: ComputerToolOptions): CuaExecutorTool[] {
-	const translator = new InternalComputerTranslator(args);
+	return buildCuaComputerTools(args, new InternalComputerTranslator(args));
+}
+
+/** Build executor tools against an existing translator (internal; not part of the package surface). */
+export function buildCuaComputerTools(
+	args: Pick<ComputerToolOptions, "toolExecutors" | "computerUseExtra">,
+	translator: InternalComputerTranslator,
+): CuaExecutorTool[] {
 	return withNavigationTool(args).map((executor) => createExecutorTool(executor, translator));
 }
 
-function withNavigationTool(args: ComputerToolOptions): ComputerExecutorSpec[] {
+function withNavigationTool(args: Pick<ComputerToolOptions, "toolExecutors" | "computerUseExtra">): ComputerExecutorSpec[] {
 	const executors: ComputerExecutorSpec[] = [...args.toolExecutors];
 	const existing = new Set(executors.map((executor) => executor.definition.name));
 	if (args.computerUseExtra && !existing.has(CUA_NAVIGATION_TOOL_NAME)) {
@@ -92,7 +99,7 @@ async function executeBatchTool(translator: InternalComputerTranslator, params:
 	const content: ToolContent = [];
 	const readResults: BatchDetails["readResults"] = [];
 	try {
-		const result = await translator.executeBatch(params.actions as unknown as Array<Record<string, unknown>>);
+		const result = await translator.executeBatch(params.actions);
 		for (const read of result.readResults) {
 			if (read.type === "url") {
 				readResults.push({ type: "url", url: read.url });
@@ -124,8 +131,10 @@ async function executeNavigationTool(translator: InternalComputerTranslator, par
 		if (action === "url") {
 			url = await translator.currentUrl();
 			statusText = `Current URL: ${url}`;
+		} else if (action === "goto") {
+			await translator.executeBatch([{ type: "goto", url: params.url ?? "" }]);
 		} else {
-			await translator.executeBatch([{ type: action, url: params.url }]);
+			await translator.executeBatch([{ type: action }]);
 		}
 		const screenshot = await translator.screenshot();
 		return {

packages/agent/src/translator/keys.ts

@@ -1,5 +1,10 @@
 export const KERNEL_MODIFIER_KEYSYMS = ["Control_L", "Alt_L", "Shift_L", "Super_L"] as const;
 
+// Models are imprecise about key naming regardless of provider: the same
+// model may emit W3C KeyboardEvent names ("ArrowLeft"), shorthand ("ctrl",
+// "cmd"), keypad names ("kp_enter"), or word-form punctuation ("plus").
+// This table is the corrective force that absorbs that nondeterminism into
+// Kernel's X11 keysym vocabulary.
 const KEY_ALIASES: Record<string, string> = {
 	alt: "Alt_L",
 	alt_l: "Alt_L",