Tighten cua-agent design seams: typed translator, declared channels

[rgarcia]

Summary

Design-review follow-ups for packages/agent:

• Typed translator (no more silent misclicks): InternalComputerTranslator.executeBatch now takes the canonical CuaAction[] union with an exhaustive switch (never guard), replacing the Record<string, unknown> re-parsing layer. pi already validates tool arguments against the typebox schemas before execute, so the defensive toInt/stringOr fallbacks that could turn malformed input into a click at 0,0 are gone; a new canonical action type is now a compile error here instead of a runtime throw. The one remaining runtime guard is the documented button coercion (wire schemas keep button an open string; out-of-set values coerce to "left" per the CuaMouseButton contract). The duplicated button unions are deleted in favor of cua-ai's.
• Declared keepToolNames channel: cua-ai exports CuaSimpleStreamOptions (SimpleStreamOptions + keepToolNames), and CuaAgent's stream wrapper uses it — the as SimpleStreamOptions & {...} cast is gone, so the agent/adapter agreement is compiler-owned.
• One translator per runtime: CuaRuntimeController shares a single InternalComputerTranslator between the executor tools and the payload getScreenshot capability (previously two instances per runtime). createCuaComputerTools is unchanged for external composition.
• Conditional prepareNextTurn: the wrapper now returns undefined (stock pi behavior, no context rebuild) unless a user hook returns an update or a mid-run model assignment marks the runtime dirty — removing the every-turn context reconstruction and its silent-field-drop risk. The state proxy re-keys off the underlying state object's identity so it can't wrap a stale target.
• keys.ts reframed: comment + test name now state what the alias table is for — absorbing model nondeterminism in key naming (W3C names, shorthand, word-form punctuation) into Kernel's X11 vocabulary — rather than reading as provider-specific knowledge.
• README: the CuaAgentHarness quickstart now showcases the harness (session-backed turns, mid-session setModel provider switch, direct prompt() result handling — it returns the turn's final AssistantMessage) instead of branch-walking gymnastics; computerUseExtra is introduced with its rationale (not every provider's native vocabulary includes navigation; this is the provider-neutral escape hatch). Snippet typechecks against the built packages.
• docs/architecture.md: the ownership-boundary section now encodes the seam explicitly — provider differences arrive as CuaRuntimeSpec data, the agent supplies capabilities back via CuaPayloadContext, packages/agent/src contains no provider names/conditionals (grep-testable), with the deliberate exception for generic model imprecision absorbed by the translator.

Test plan

• New: full fake-stream tool-call turn (model emits toolCall → translator → mocked Kernel batch/screenshot → toolResult with image fed back to the second provider request)
• New: prepareNextTurn pass-through when nothing changed; one-shot turn update (new model + refreshed tools) after agent.state.model assignment, then pass-through again
• npm test -w @onkernel/cua-agent (28 passed, live excluded) and -w @onkernel/cua-ai (88 passed)
• npm run build both workspaces (tsdown) + monorepo tsc -b green
• README harness snippet extracted and typechecked against the built packages
• Live provider matrix (e2e.live.test.ts) — recommend a run before the next release since translator internals changed

🤖 Generated with Claude Code

---

Note

Medium Risk
Translator and turn-prep behavior changed in the core browser execution path; malformed actions now error instead of silent mis-clicks, so live provider matrix runs are advisable before release.

Overview
Tightens @onkernel/cua-agent / @onkernel/cua-ai seams for browser computer-use: provider differences stay data-driven via CuaRuntimeSpec / CuaPayloadContext, documented in docs/architecture.md.

The action translator now takes canonical CuaAction[] with an exhaustive switch instead of re-parsing Record<string, unknown> — malformed shapes fail loudly rather than coercing to bad clicks; documented mouse-button coercion to "left" is unchanged. prepareNextTurn returns stock pi behavior (undefined) unless a user hook updates the turn or a mid-run model change marks the runtime dirty, avoiding per-turn context rebuilds. One InternalComputerTranslator per runtime is shared by executor tools and payload getScreenshot via buildCuaComputerTools.

CuaAgent passes CuaSimpleStreamOptions (new in cua-ai 0.3.0) for keepToolNames without a cast. Navigation goto routes through typed batch actions; key-alias comments/tests frame generic model key imprecision, not provider-specific logic.

Docs/README cover harness session turns, mid-session setModel, and computerUseExtra rationale. Releases: cua-agent 0.3.3, cua-ai 0.3.0.

^{Reviewed by Cursor Bugbot for commit 0d0baf8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[firetiger-agent]

Created a monitoring plan for this PR.

What this PR does: Refactors the internal mechanics of the @onkernel/cua-agent npm package — the typed action translator, how per-turn context is rebuilt, and how a single translator instance is shared. No new user-facing capability; this makes the internals more correct and compiler-enforced.

Intended effect:

• "unknown computer action type" errors: baseline 0; confirmed if it stays at 0 post-publish (a new action type is now a compile error, not a runtime throw for users).
• RunCUANavigationActivity failures: baseline ~730–1,585/hr (per recent managed-auth monitoring); confirmed if no sustained increase above 2× baseline after managed-auth workers pick up the new package version.

Risks:

• Stale turn state after model swap — if runtimeDirty is not set in an edge case, the agent may operate with stale tools after a mid-run agent.state.model = assignment; signal: RunCUANavigationActivity errors with wrong-model context; alert if failures increase >2× baseline sustained for 1+ hour.
• Typed translator throws on unexpected input — the old code silently coerced bad input (e.g. coordinates → 0,0); the new exhaustive switch calls unreachable() for any action type that slips through schema validation; signal: any "unknown computer action type" ERROR log post-publish; alert on first occurrence.
• Live provider matrix untested — the PR test plan explicitly marks the e2e.live.test.ts run as recommended before next release (checkbox open); translator internals changed; alert if this run has not been completed before a production publish.

Status updates will be posted automatically on this PR as monitoring progresses.

View monitor