From a88c3a39094b26bc1171706b341a77e2f4d11b3b Mon Sep 17 00:00:00 2001 From: Rafael Garcia Date: Fri, 24 Jul 2026 10:55:50 -0400 Subject: [PATCH 1/2] Harden Anthropic native beta compatibility --- docs/anthropic-native-tools-investigation.md | 186 ++++++++++++++++++ docs/architecture.md | 32 +-- packages/agent/CHANGELOG.md | 7 + packages/agent/src/agent.ts | 66 +++++++ packages/agent/test/agent.test.ts | 91 ++++++++- packages/ai/CHANGELOG.md | 17 +- packages/ai/src/native-tools.ts | 41 +++- packages/ai/src/providers/anthropic/native.ts | 2 +- packages/ai/src/providers/common.ts | 2 + packages/ai/src/runtime-spec.ts | 1 + .../test/anthropic-native.integration.test.ts | 55 ++++++ packages/ai/test/native-tools.test.ts | 37 +++- packages/cli/src/cli.ts | 8 +- .../cli/test/cli-harness-validation.test.ts | 12 ++ 14 files changed, 525 insertions(+), 32 deletions(-) create mode 100644 docs/anthropic-native-tools-investigation.md create mode 100644 packages/ai/test/anthropic-native.integration.test.ts diff --git a/docs/anthropic-native-tools-investigation.md b/docs/anthropic-native-tools-investigation.md new file mode 100644 index 00000000..39ae30c9 --- /dev/null +++ b/docs/anthropic-native-tools-investigation.md @@ -0,0 +1,186 @@ +# Anthropic native browser/computer tools investigation + +**Investigation date:** 2026-07-24 + +**Scope:** Anthropic's early-access `computer_20260701` and `browser_20260701` client tools in `@onkernel/cua-ai` / `@onkernel/cua-agent`. + +This report intentionally contains no API keys, request IDs, live-view URLs, or verbatim material from Anthropic's confidential early-access documents. Both supplied PDFs were read completely and used as the primary specification; extracted text stayed in a mode-0600 directory under `/tmp` and was not committed. + +## Conclusion + +The July native tools have **not been retired**. With the newly provisioned key, both tools return valid `tool_use` blocks through direct `/v1/messages` requests, through CUA's pi-ai serialization path, and through full Kernel-browser agent runs. + +The reproducible rejection is a **model/tool mismatch exposed by a CUA validation bug**: + +- `claude-opus-4-7` rejects both July tool types with HTTP 400. +- CUA previously accepted any Anthropic computer-use model with either July native tool. Unit tests even used Claude Opus 4.5, which the live API rejects. +- The repository's normal Anthropic examples/tests prominently use Claude Opus 4.7, making it easy for live QA to combine that model with `--native-tool` and reach the server-side 400. + +The other suspected causes were ruled out for the new key and the supported configuration: + +- **Entitlement:** the new key is entitled; correct early-access requests return 200. Entitlement remains an account/workspace prerequisite. +- **Header/version retirement:** the old June computer draft pair is gone, but CUA already sends the live July pair. The July pair returns 200. +- **Endpoint:** the tools work on the first-party Anthropic Messages endpoint, `POST https://api.anthropic.com/v1/messages`. +- **SDK/pi serialization:** CUA's placeholder-tool replacement and beta-header injection work live despite the installed Anthropic SDK not having types for these private tool versions. + +CUA now rejects unsupported model/tool combinations locally, before provisioning a browser, and enforces the native multi-action stop-on-first-failure result contract. + +## Compatibility matrix + +### Tool version and header + +| Tool declaration | `anthropic-beta` | Live result on eligible model | Status | +| --- | --- | --- | --- | +| `computer_20260601` | `computer-use-2026-06-01` | HTTP 400: beta value no longer recognized | Obsolete early draft; do not use | +| `computer_20260601` | `computer-use-2026-07-01` | HTTP 400: tool type unknown | Obsolete tool string | +| `computer_20260701` | omitted | HTTP 400: tool type unknown without beta | Header required | +| `computer_20260701` | `computer-use-2026-07-01` | HTTP 200 + `computer` tool use | Current early-access computer pair | +| `browser_20260701` | omitted | HTTP 400: tool type unknown without beta | Header required | +| `browser_20260701` | `browser-use-2026-07-01` | HTTP 200 + `browser` tool uses | Current early-access browser pair | +| `computer_20251124` | `computer-use-2025-11-24` | HTTP 200 on Claude Opus 4.7 | Current public computer-use fallback; not CUA's July `nativeTool` option | + +The June-to-July computer drift was already discovered in git commit `e6219bb` and merged in `3caf6cc` (#51). The current CUA header and tool strings are correct. + +### Model eligibility observed from the new key + +The model-list endpoint returned the models below. Each was probed with the correct July header/tool pair; a pass means the response contained the expected native `tool_use` block. + +| Model ID | `computer_20260701` | `browser_20260701` | +| --- | --- | --- | +| `claude-sonnet-5` | Pass | Pass | +| `claude-fable-5` | Pass | **Rejected** | +| `claude-opus-4-8` | Pass | Pass | +| `claude-opus-4-7` | **Rejected** | **Rejected** | +| `claude-sonnet-4-6` | **Rejected** | **Rejected** | +| `claude-opus-4-6` | **Rejected** | **Rejected** | +| `claude-opus-4-5-20251101` | **Rejected** | **Rejected** | +| `claude-haiku-4-5-20251001` | **Rejected** | **Rejected** | +| `claude-sonnet-4-5-20250929` | **Rejected** | **Rejected** | +| `claude-opus-4-1-20250805` | **Rejected** | **Rejected** | + +These lists are deliberately fail-closed in `packages/ai/src/native-tools.ts`. Re-run the live integration test before expanding them when Anthropic changes eligibility. + +### Request declaration and flags + +| Dimension | `computer_20260701` | `browser_20260701` | +| --- | --- | --- | +| Required declaration | `type`, name (normally `computer`) | `type`, name (normally `browser`) | +| Optional CUA-exposed fields | `display_number`, `enable_zoom`, `cache_control` | `enable_javascript_exec`, `cache_control` | +| Dimensions | CUA does not send width/height; the screenshot establishes the coordinate frame | No declared dimensions; viewport screenshots establish the frame | +| Zoom | `enable_zoom` defaults false at the API; both true and false passed | `zoom` is part of the browser action set | +| JavaScript | Not applicable | API default is false; CUA intentionally defaults it to true to match canonical browser mode. Explicit false and true both passed live. | +| `strict: true` | HTTP 400 | HTTP 400 | +| Both July tools together | Rejected: desktop and viewport coordinate frames cannot be mixed | Rejected for the same reason | +| Endpoint | First-party Anthropic `/v1/messages` only for this early-access version | First-party Anthropic `/v1/messages` only for this early-access version | +| Account requirement | Matching early-access entitlement | Matching early-access entitlement | + +### Action and result contract + +`computer_20260701` emits one action per `tool_use`: screenshot; left/right/middle/double/triple click; drag; mouse move/down/up; scroll; type; key (with repeat); hold key; wait; cursor position; and optional zoom. Coordinates are screenshot pixels. + +`browser_20260701` emits one action per `tool_use`: navigation/tabs; accessibility-tree and text reads; find/fill/scroll-to; viewport screenshot/zoom; ref- or coordinate-targeted pointer actions; keyboard/scroll/wait; and optional JavaScript execution. Coordinates are viewport pixels and element refs are scoped to their tab/document generation. + +Both tools can emit several `tool_use` blocks in one assistant turn. CUA executes them sequentially and returns one matching `tool_result` per block. Screenshot/zoom results contain images; structured reads and acknowledgements contain text. After the first failed action, CUA now skips every remaining call in that assistant turn and returns the tool-specific required error result rather than executing against stale state. + +## CUA implementation audit + +### Correct before this investigation + +- `packages/ai/src/providers/anthropic/native.ts` replaces pi-ai's permissive placeholder tool with the Anthropic-defined declaration after ordinary tool serialization. +- `packages/ai/src/providers.ts` routes CUA's synthetic native API IDs back through pi-ai's `anthropic-messages` transport and injects the matching beta header. +- Incoming native actions map onto CUA's canonical computer/browser actions, including key repeat, seconds-to-milliseconds duration conversion, browser refs, tabs, zoom, and JavaScript execution. +- Native CUA tools use sequential tool execution, and browser navigation results include tab context. +- Full native computer and browser runs completed against a Kernel browser. + +### Bugs fixed + +1. **Missing model eligibility validation.** `resolveNativeTool()` checked only provider and mode. It now rejects unsupported model families with an error that identifies the early-access requirement and lists the supported families. +2. **Incomplete multi-action failure semantics.** pi-agent-core's sequential mode continued with later calls after an earlier call failed. `CuaRuntimeSpec` now carries the provider-required skip message as data, and both `CuaAgent` and `CuaAgentHarness` block the rest of that assistant turn after the first failure. +3. **Documentation ambiguity.** Architecture and changelog text now distinguish the two headers, list the live-verified models, and state that the tools are allowlisted early access. +4. **No live serialization regression.** `packages/ai/test/anthropic-native.integration.test.ts` now checks both tools through CUA's actual pi-ai transport. + +## Official public documentation comparison + +Fetched on 2026-07-24: + +- [Public computer-use guide](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) +- [Claude API release notes](https://platform.claude.com/docs/en/release-notes/overview) + +The public guide still documents `computer_20251124` / `computer-use-2025-11-24` (and the older January pair). It does not document the July native computer version or the client-side July browser tool. The public release notes likewise do not announce either July pair. The installed `@anthropic-ai/sdk` types contain neither July tool string. + +That discrepancy is expected for allowlisted early access, but it means public docs cannot be used to infer that `computer_20260701` or `browser_20260701` is generally available. The confidential specifications plus live first-party API behavior are the source of truth for these pairs. + +## Sanitized evidence and commands + +Credential-presence check (does not print the value): + +```bash +test -n "${ANTHROPIC_API_KEY:-}" && echo 'ANTHROPIC_API_KEY is set' +``` + +Core direct probe shape (repeat with the browser declaration/header for browser use). The response filter omits IDs and usage: + +```bash +curl -sS https://api.anthropic.com/v1/messages \ + -H 'content-type: application/json' \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H 'anthropic-version: 2023-06-01' \ + -H 'anthropic-beta: computer-use-2026-07-01' \ + -d '{ + "model":"claude-opus-4-8", + "max_tokens":96, + "tools":[{"type":"computer_20260701","name":"computer","enable_zoom":true}], + "messages":[{"role":"user","content":"Use the computer tool to take one screenshot."}] + }' | jq '{stop_reason, blocks: [.content[] | {type, name, action: .input.action}]}' +``` + +Live CUA/pi-ai serialization regression: + +```bash +cd packages/ai +npx vitest --run --config vitest.integration.config.ts \ + test/anthropic-native.integration.test.ts +``` + +Full Kernel-browser smoke tests: + +```bash +MODEL_REF=anthropic:claude-opus-4-8 CONFIG=native-computer \ + NODE_OPTIONS=--conditions=source \ + npx tsx packages/agent/examples/anthropic-native-smoke.ts + +MODEL_REF=anthropic:claude-opus-4-8 CONFIG=native-browser \ + NODE_OPTIONS=--conditions=source \ + npx tsx packages/agent/examples/anthropic-native-smoke.ts +``` + +Repository validation: + +```bash +npm run build +npm run typecheck +npm test --workspace @onkernel/cua-ai +npm test --workspace @onkernel/cua-agent +npm test --workspace @onkernel/cua-cli +cd packages/ai +npx vitest --run --config vitest.integration.config.ts \ + test/anthropic-native.integration.test.ts +``` + +Observed sanitized outcomes: + +- Direct baseline Messages request: 200. +- Direct July computer request on Opus 4.8: 200, `tool_use` action `screenshot`. +- Direct July browser request on Opus 4.8: 200, `tool_use` actions including `navigate`. +- CUA/pi-ai live integration: 2/2 passed. +- Kernel browser native-computer smoke: completed the `example.com` task. +- Kernel browser native-browser smoke: completed the `example.com` task. +- Unit tests added for fail-closed model validation and stop-on-first-failure behavior in both agent classes. + +## Recommended integration path + +1. For the strongest shared configuration, use `anthropic:claude-opus-4-8` with either July native tool and an API key from the entitled organization. +2. Claude Sonnet 5 also supports both tools. Claude Fable 5 supports only the July computer tool in the observed account. +3. Never pair Claude Opus 4.7 (or earlier listed models) with a July native tool. Omit `nativeTool` to use CUA's canonical function-tool path instead. +4. For a generally documented provider-native computer integration, evaluate `computer_20251124` separately. There is no public native browser-tool equivalent to the July early-access browser tool. +5. Keep the live integration test gated on the entitled key. Treat an unrecognized beta header as an entitlement/version incident; treat a “model does not support tool type” response as model eligibility drift. diff --git a/docs/architecture.md b/docs/architecture.md index 5840bd8a..77e6aa82 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -127,17 +127,26 @@ The mode is set at construction (`mode` on `CuaAgent`/`CuaAgentHarness`, TUI's `/mode` command), which refreshes CUA-owned tools and the default system prompt. -**Native tools.** `resolveCuaRuntimeSpec(model, { nativeTool })` drives an -Anthropic model through its provider-defined tool schema instead of the -canonical function tools: `computer_20260701` pairs with `computer` mode and -`browser_20260701` with `browser` mode (mismatches throw, mirroring the API's -own rejection of mixed frames). The spec routes the model to a CUA-owned api -id; the registered `anthropic` provider dispatches it to pi's builtin -`anthropic-messages` transport with the tool's `anthropic-beta` header, an -`onPayload` hook swaps the placeholder tool for the native declaration, and -`providers/anthropic/native.ts` maps incoming `tool_use` inputs onto the same -canonical actions the mode uses — so canonical vs native is purely a wire -format difference over one execution path. +**Native tools.** `resolveCuaRuntimeSpec(model, { nativeTool })` can drive an +Anthropic model through an allowlisted, Anthropic-API-only early-access tool +schema instead of the canonical function tools: `computer_20260701` pairs with +`computer` mode and `browser_20260701` with `browser` mode. Model and mode +mismatches throw locally before a browser is provisioned. The live-verified +model families are `claude-fable-5`, `claude-opus-4-8`, and `claude-sonnet-5` +for `computer_20260701`; only `claude-opus-4-8` and `claude-sonnet-5` support +`browser_20260701`. The API key's organization must also have the matching +beta entitlement. + +The spec routes the model to a CUA-owned api id; the registered `anthropic` +provider dispatches it to pi's builtin `anthropic-messages` transport with the +tool-specific `anthropic-beta` header, an `onPayload` hook swaps the placeholder +tool for the native declaration, and `providers/anthropic/native.ts` maps +incoming `tool_use` inputs onto the same canonical actions the mode uses. The +runtime spec also carries the native tool's stop-on-first-failure result text, +which cua-agent applies without a provider conditional. Canonical vs native is +therefore a wire-format and turn-contract difference over one execution path. +See [the July 2026 investigation](anthropic-native-tools-investigation.md) +for the compatibility matrix and live evidence. ## Layers @@ -353,7 +362,6 @@ flowchart LR | Feature | Status | Notes | | -------------------------------------------------- | -------- | ------------------------------------------------- | -| Anthropic `hold_key` / `zoom` | deferred | Translator returns errors so the model adapts | | `--local` Docker-backed browser | deferred | Remote Kernel cloud only | | pi-tui `SelectList`-based session picker for `-r` | deferred | Plain readline picker today | | Auto-compaction in the harness run loop | deferred | Manual `/compact` from the TUI; `shouldCompact` + `estimateContextTokens` are available from cua-agent re-exports for a future auto-trigger | diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 6229b498..82f4e55c 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## Unreleased + +- Native Anthropic multi-action turns now stop after the first failed tool call. + Every remaining call in that assistant turn receives the provider-required + error result instead of executing against stale browser state. This applies + to both `CuaAgent` and `CuaAgentHarness` via provider-neutral runtime data. + ## 0.7.0 - 2026-07-17 - `CuaAgent` and `CuaAgentHarness` support Moonshot Kimi K3 diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index 7761da99..a3237aeb 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -255,6 +255,10 @@ class CuaRuntimeController { return this.runtimeSpec.mode; } + get stopOnFirstToolFailureMessage(): string | undefined { + return this.runtimeSpec.stopOnFirstToolFailureMessage; + } + setMode(mode: CuaMode): void { if (mode === this.runtimeSpec.mode) return; this.beginSwitch(this.resolveSpec(this.runtimeSpec.model, mode)); @@ -509,9 +513,15 @@ export class CuaAgent extends Agent { }; return retryingStream(model, context, optionsWithCuaRuntime); }; + const guardedToolHooks = stopToolTurnAfterFailure( + runtime.stopOnFirstToolFailureMessage, + agentOptions.beforeToolCall, + agentOptions.afterToolCall, + ); super({ ...agentOptions, + ...guardedToolHooks, getApiKey: agentOptions.getApiKey ?? getCuaEnvApiKey, streamFn: wrappedStreamFn, transformContext: async (messages, signal) => @@ -654,6 +664,8 @@ export class CuaAgentHarness< private requestedActiveToolNames?: string[]; private emptyResponseRecoveryAttempts = 0; private hasPendingActiveQueue = false; + private toolTurnFailed = false; + private removeToolFailureGuard?: () => void; constructor(options: CuaAgentHarnessOptions) { const { @@ -705,6 +717,20 @@ export class CuaAgentHarness< this.runtime = runtime; this.requestedActiveToolNames = activeToolNames; + if (runtime.stopOnFirstToolFailureMessage) { + this.installToolFailureGuard(runtime.stopOnFirstToolFailureMessage); + this.subscribe((event) => { + if (event.type === "message_end" && event.message.role === "assistant") { + this.toolTurnFailed = false; + // Harness hooks are last-result-wins. Reinsert the guard after caller + // hooks at the start of each assistant tool turn so it cannot be + // accidentally overridden once a prior action has failed. + this.installToolFailureGuard(runtime.stopOnFirstToolFailureMessage!); + } else if (event.type === "tool_execution_end" && event.isError) { + this.toolTurnFailed = true; + } + }); + } if (recovery && recovery.maxAttempts > 0) { this.on("before_agent_start", () => { this.emptyResponseRecoveryAttempts = 0; @@ -727,6 +753,13 @@ export class CuaAgentHarness< }); } + private installToolFailureGuard(message: string): void { + this.removeToolFailureGuard?.(); + this.removeToolFailureGuard = this.on("tool_call", () => + this.toolTurnFailed ? { block: true, reason: message } : undefined, + ); + } + private async recoverFromEmptyResponse( recovery: CuaEmptyResponseRecoveryOptions, signal?: AbortSignal, @@ -809,3 +842,36 @@ function composeOnPayload(first: AgentOptions["onPayload"], second: AgentOptions return second(afterFirst ?? payload, modelRef); }; } + +function stopToolTurnAfterFailure( + message: string | undefined, + before: AgentOptions["beforeToolCall"], + after: AgentOptions["afterToolCall"], +): Pick { + if (!message) return { beforeToolCall: before, afterToolCall: after }; + const failedTurns = new WeakSet(); + + return { + beforeToolCall: async (context, signal) => { + if (failedTurns.has(context.assistantMessage)) return { block: true, reason: message }; + try { + const result = await before?.(context, signal); + if (result?.block) failedTurns.add(context.assistantMessage); + return result; + } catch (error) { + failedTurns.add(context.assistantMessage); + throw error; + } + }, + afterToolCall: async (context, signal) => { + try { + const result = await after?.(context, signal); + if (result?.isError ?? context.isError) failedTurns.add(context.assistantMessage); + return result; + } catch (error) { + failedTurns.add(context.assistantMessage); + throw error; + } + }, + }; +} diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index b1d8683c..99c53276 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -88,11 +88,11 @@ function createScriptedStream(texts: Array, contexts?: Array return { streamFn, calls: () => providerCalls }; } -function createModelsFromStream(streamFn: StreamFn) { +function createModelsFromStream(streamFn: StreamFn, provider = "openai") { const models = createCuaModels(); models.setProvider({ - id: "openai", - name: "scripted openai", + id: provider, + name: `scripted ${provider}`, auth: { apiKey: { name: "test key", @@ -259,7 +259,7 @@ describe("CuaAgent", () => { client, nativeTool: { type: "browser_20260701" }, initialState: { - model: "anthropic:claude-opus-4-5", + model: "anthropic:claude-opus-4-8", }, }); expect(agent.getMode()).toBe("browser"); @@ -511,6 +511,47 @@ describe("CuaAgent", () => { expect(fedBack!.content.some((block) => block.type === "image" && block.mimeType === "image/png")).toBe(true); }); + it("stops a native multi-action turn after its first failed action", async () => { + const contexts: Context[] = []; + let providerCalls = 0; + const streamFn: StreamFn = (model, context) => { + contexts.push({ ...context, messages: structuredClone(context.messages) }); + const stream = createAssistantMessageEventStream(); + const message = createAssistantMessage(model); + if (providerCalls++ === 0) { + message.content = [ + { type: "toolCall", id: "tool-1", name: "browser", arguments: { action: "warp" } }, + { type: "toolCall", id: "tool-2", name: "browser", arguments: { action: "navigate", url: "https://example.com" } }, + ]; + message.stopReason = "toolUse"; + } else { + message.content = [{ type: "text", text: "done" }]; + } + stream.push({ type: "start", partial: message }); + stream.push({ type: "done", reason: message.stopReason, message }); + stream.end(message); + return stream; + }; + const agent = new CuaAgent({ + browser, + client, + streamFn, + nativeTool: { type: "browser_20260701" }, + initialState: { model: "anthropic:claude-opus-4-8" }, + }); + + await agent.prompt("run two browser actions"); + + const results = contexts[1]!.messages.filter((message) => message.role === "toolResult"); + expect(results).toHaveLength(2); + expect(results[0]).toMatchObject({ toolCallId: "tool-1", isError: true }); + expect(results[1]).toMatchObject({ + toolCallId: "tool-2", + isError: true, + content: [{ type: "text", text: "Not executed: an earlier action in this turn failed." }], + }); + }); + it("applies screenshot projection after a caller context transform", async () => { const history: AgentMessage[] = []; for (let index = 1; index <= 5; index += 1) { @@ -1160,6 +1201,48 @@ describe("CuaAgentHarness", () => { expect(calls).toBe(2); }); + it("stops a native harness turn after its first failed action", async () => { + const contexts: Context[] = []; + let calls = 0; + const streamFn: StreamFn = (model, context) => { + contexts.push({ ...context, messages: structuredClone(context.messages) }); + const stream = createAssistantMessageEventStream(); + const message = createAssistantMessage(model); + if (calls++ === 0) { + message.content = [ + { type: "toolCall", id: "tool-1", name: "computer", arguments: { action: "warp" } }, + { type: "toolCall", id: "tool-2", name: "computer", arguments: { action: "screenshot" } }, + ]; + message.stopReason = "toolUse"; + } else { + message.content = [{ type: "text", text: "done" }]; + } + stream.push({ type: "start", partial: message }); + stream.push({ type: "done", reason: message.stopReason, message }); + stream.end(message); + return stream; + }; + const harness = new CuaAgentHarness({ + ...(await createHarnessServices()), + browser, + client, + model: "anthropic:claude-opus-4-8", + models: createModelsFromStream(streamFn, "anthropic"), + nativeTool: { type: "computer_20260701" }, + }); + + await harness.prompt("run two computer actions"); + + const results = contexts[1]!.messages.filter((message) => message.role === "toolResult"); + expect(results).toHaveLength(2); + expect(results[0]).toMatchObject({ toolCallId: "tool-1", isError: true }); + expect(results[1]).toMatchObject({ + toolCallId: "tool-2", + isError: true, + content: [{ type: "text", text: "Not executed: an earlier computer action in this turn failed." }], + }); + }); + it.each([[-1], [1.5], [Number.POSITIVE_INFINITY], [Number.NaN]])( "rejects invalid maxAttempts %s", async (maxAttempts) => { diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index e635e1c9..63e3b2ed 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## Unreleased + +- Fail locally when `computer_20260701` or `browser_20260701` is paired with + an Anthropic model that the live early-access API rejects. The verified + computer models are Claude Fable 5, Claude Opus 4.8, and Claude Sonnet 5; + the verified browser models are Claude Opus 4.8 and Claude Sonnet 5. +- Carry each native tool's stop-on-first-failure result contract through + `CuaRuntimeSpec`, allowing cua-agent to skip unsafe remaining actions without + a provider conditional. +- Add opt-in live integration coverage for the beta header, native declaration, + and pi-ai serialization path of both July 2026 tools. + ## 0.7.0 - 2026-07-17 - Added Moonshot Kimi K3 computer-use support: `moonshotai:kimi-k3` @@ -47,8 +59,9 @@ Introduces action planes (modes) and Anthropic native computer-use tools. (`cuaToolNameForAction`), descriptions, schemas, and system prompts. - New `nativeTool` option drives Anthropic models through their native computer-use declarations: `computer_20260701` (computer mode, with - `enable_zoom`) and `browser_20260701` (browser mode) behind - `anthropic-beta: computer-use-2026-07-01`. + `enable_zoom`) behind `anthropic-beta: computer-use-2026-07-01`, and + `browser_20260701` (browser mode) behind + `anthropic-beta: browser-use-2026-07-01`. - JavaScript execution is on by default: `browser_evaluate` is part of the default browser/hybrid action sets, and native `browser_20260701` declarations default `enable_javascript_exec` to true (an explicit value on diff --git a/packages/ai/src/native-tools.ts b/packages/ai/src/native-tools.ts index 99c2dbeb..ce7c4cb6 100644 --- a/packages/ai/src/native-tools.ts +++ b/packages/ai/src/native-tools.ts @@ -54,11 +54,29 @@ interface NativeToolInfo { provider: "anthropic"; betaHeader: string; defaultName: string; + /** Model families verified against the live early-access API. */ + supportedModelFamilies: readonly string[]; + /** Required error result for actions skipped after an earlier action failed. */ + skippedAfterFailureMessage: string; } const NATIVE_TOOL_INFO: Record = { - computer_20260701: { mode: "computer", provider: "anthropic", betaHeader: "computer-use-2026-07-01", defaultName: "computer" }, - browser_20260701: { mode: "browser", provider: "anthropic", betaHeader: "browser-use-2026-07-01", defaultName: "browser" }, + computer_20260701: { + mode: "computer", + provider: "anthropic", + betaHeader: "computer-use-2026-07-01", + defaultName: "computer", + supportedModelFamilies: ["claude-fable-5", "claude-opus-4-8", "claude-sonnet-5"], + skippedAfterFailureMessage: "Not executed: an earlier computer action in this turn failed.", + }, + browser_20260701: { + mode: "browser", + provider: "anthropic", + betaHeader: "browser-use-2026-07-01", + defaultName: "browser", + supportedModelFamilies: ["claude-opus-4-8", "claude-sonnet-5"], + skippedAfterFailureMessage: "Not executed: an earlier action in this turn failed.", + }, }; /** The {@link CuaMode} a native tool requires. */ @@ -85,6 +103,8 @@ export interface ResolvedCuaNativeTool { name: string; /** Required `anthropic-beta` header value. */ betaHeader: string; + /** Error result required for later tool calls after the first failure in a turn. */ + skippedAfterFailureMessage: string; mode: CuaMode; } @@ -101,6 +121,12 @@ export function resolveNativeTool(spec: CuaNativeToolSpec, model: Model, mo if (model.provider !== info.provider) { throw new Error(`native tool "${spec.type}" requires an ${info.provider} model paired with mode "${info.mode}"; got provider "${model.provider}"`); } + if (!info.supportedModelFamilies.some((family) => isModelFamily(model.id, family))) { + throw new Error( + `native tool "${spec.type}" is an allowlisted Anthropic API beta and does not support model "${model.id}"; ` + + `supported model families: ${info.supportedModelFamilies.join(", ")}`, + ); + } if (mode !== info.mode) { throw new Error(`native tool "${spec.type}" requires mode "${info.mode}"; got "${mode}"`); } @@ -110,6 +136,17 @@ export function resolveNativeTool(spec: CuaNativeToolSpec, model: Model, mo declaration: { ...spec, name }, name, betaHeader: info.betaHeader, + skippedAfterFailureMessage: info.skippedAfterFailureMessage, mode, }; } + +function isModelFamily(modelId: string, family: string): boolean { + const id = modelId.toLowerCase(); + if (id === family) return true; + if (!id.startsWith(`${family}-`)) return false; + return id + .slice(family.length + 1) + .split("-") + .every((segment) => /^\d+$/.test(segment)); +} diff --git a/packages/ai/src/providers/anthropic/native.ts b/packages/ai/src/providers/anthropic/native.ts index a1f2b18c..986df8d3 100644 --- a/packages/ai/src/providers/anthropic/native.ts +++ b/packages/ai/src/providers/anthropic/native.ts @@ -31,7 +31,7 @@ export function withAnthropicBetaHeader(options: T | un // The native tool's input schema is Anthropic-defined and validated // server-side; the local placeholder schema stays permissive and the // executor validates during mapping. -const NativeActionSchema = Type.Object({ action: Type.String() }, { additionalProperties: true }); +const NativeActionSchema = Type.Object({ action: Type.Optional(Type.String()) }, { additionalProperties: true }); /** * Build the single execution adapter for a native Anthropic tool: tool calls diff --git a/packages/ai/src/providers/common.ts b/packages/ai/src/providers/common.ts index d55f1567..07c2a456 100644 --- a/packages/ai/src/providers/common.ts +++ b/packages/ai/src/providers/common.ts @@ -373,6 +373,8 @@ export interface CuaRuntimeSpec { mode: CuaMode; /** Present when the model is driven through a provider-native tool declaration. */ nativeTool?: ResolvedCuaNativeTool; + /** When set, stop executing a turn's remaining tool calls after its first failure and return this error text for each. */ + stopOnFirstToolFailureMessage?: string; /** Provider-facing CUA tool definitions used for model requests. */ toolDefinitions: Tool[]; /** Local execution adapters that turn provider tool calls into canonical CUA actions. */ diff --git a/packages/ai/src/runtime-spec.ts b/packages/ai/src/runtime-spec.ts index 2777ac81..cda0a797 100644 --- a/packages/ai/src/runtime-spec.ts +++ b/packages/ai/src/runtime-spec.ts @@ -65,6 +65,7 @@ export function resolveCuaRuntimeSpec(input: CuaRuntimeSpecInput, options: CuaRu provider, mode, nativeTool, + stopOnFirstToolFailureMessage: nativeTool.skippedAfterFailureMessage, toolDefinitions: executors.map((executor) => executor.definition), toolExecutors: executors, defaultSystemPrompt: mod.buildSystemPrompt({ mode }), diff --git a/packages/ai/test/anthropic-native.integration.test.ts b/packages/ai/test/anthropic-native.integration.test.ts new file mode 100644 index 00000000..42b63766 --- /dev/null +++ b/packages/ai/test/anthropic-native.integration.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, it } from "vitest"; +import { createCuaModels, resolveCuaRuntimeSpec, type CuaNativeToolSpec } from "../src/index"; + +const apiKey = process.env.ANTHROPIC_API_KEY; +const liveIt = apiKey ? it : it.skip; + +const cases: Array<{ + name: string; + nativeTool: CuaNativeToolSpec; + prompt: string; + expectedAction: string; +}> = [ + { + name: "computer_20260701", + nativeTool: { type: "computer_20260701", enable_zoom: true }, + prompt: "Use the computer tool to take one screenshot.", + expectedAction: "screenshot", + }, + { + name: "browser_20260701", + nativeTool: { type: "browser_20260701", enable_javascript_exec: true }, + prompt: "Use the browser tool to navigate to example.com.", + expectedAction: "navigate", + }, +]; + +describe("Anthropic early-access native tools", () => { + for (const current of cases) { + liveIt(`${current.name} survives CUA's pi-ai serialization`, async () => { + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { nativeTool: current.nativeTool }); + const response = await createCuaModels().complete( + spec.model, + { + systemPrompt: spec.defaultSystemPrompt, + messages: [{ role: "user", content: current.prompt, timestamp: Date.now() }], + tools: spec.toolDefinitions, + }, + { + apiKey, + maxTokens: 96, + onPayload: spec.onPayload, + }, + ); + + expect(response.stopReason, response.errorMessage).toBe("toolUse"); + expect(response.content).toContainEqual( + expect.objectContaining({ + type: "toolCall", + name: current.name.startsWith("computer") ? "computer" : "browser", + arguments: expect.objectContaining({ action: current.expectedAction }), + }), + ); + }, 60_000); + } +}); diff --git a/packages/ai/test/native-tools.test.ts b/packages/ai/test/native-tools.test.ts index dd118e92..bcc4bb07 100644 --- a/packages/ai/test/native-tools.test.ts +++ b/packages/ai/test/native-tools.test.ts @@ -25,11 +25,11 @@ describe("native tool validation", () => { }); it("rejects a native tool with a conflicting mode", () => { - expect(() => resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "browser", nativeTool: { type: "computer_20260701" } })).toThrow( + expect(() => resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { mode: "browser", nativeTool: { type: "computer_20260701" } })).toThrow( /requires mode "computer"/, ); expect(() => - resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "hybrid", nativeTool: { type: "browser_20260701" } }), + resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { mode: "hybrid", nativeTool: { type: "browser_20260701" } }), ).toThrow(/requires mode "browser"/); }); @@ -38,38 +38,58 @@ describe("native tool validation", () => { /requires an anthropic model paired with mode "computer"/, ); }); + + it("fails locally when the model is ineligible for the early-access tool version", () => { + expect(() => resolveCuaRuntimeSpec("anthropic:claude-opus-4-7", { nativeTool: { type: "computer_20260701" } })).toThrow( + /native tool "computer_20260701" is an allowlisted Anthropic API beta.*claude-opus-4-8/s, + ); + expect(() => resolveCuaRuntimeSpec("anthropic:claude-fable-5", { nativeTool: { type: "browser_20260701" } })).toThrow( + /supported model families: claude-opus-4-8, claude-sonnet-5/, + ); + }); + + it("accepts each live-verified native-tool model family", () => { + for (const model of ["claude-fable-5", "claude-opus-4-8", "claude-sonnet-5"] as const) { + expect(() => resolveCuaRuntimeSpec(`anthropic:${model}`, { nativeTool: { type: "computer_20260701" } })).not.toThrow(); + } + for (const model of ["claude-opus-4-8", "claude-sonnet-5"] as const) { + expect(() => resolveCuaRuntimeSpec(`anthropic:${model}`, { nativeTool: { type: "browser_20260701" } })).not.toThrow(); + } + }); }); describe("native runtime specs", () => { it("routes computer_20260701 to the native api with a single placeholder tool", () => { - const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "computer_20260701", enable_zoom: true } }); + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { nativeTool: { type: "computer_20260701", enable_zoom: true } }); expect(spec.mode).toBe("computer"); expect(spec.model.api).toBe(ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API); expect(spec.nativeTool?.betaHeader).toBe("computer-use-2026-07-01"); + expect(spec.stopOnFirstToolFailureMessage).toBe("Not executed: an earlier computer action in this turn failed."); expect(spec.toolDefinitions.map((tool) => tool.name)).toEqual(["computer"]); }); it("routes browser_20260701 to the native api under the default name", () => { - const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "browser_20260701" } }); + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { nativeTool: { type: "browser_20260701" } }); expect(spec.mode).toBe("browser"); expect(spec.model.api).toBe(ANTHROPIC_NATIVE_BROWSER_MESSAGES_API); + expect(spec.stopOnFirstToolFailureMessage).toBe("Not executed: an earlier action in this turn failed."); expect(spec.toolDefinitions.map((tool) => tool.name)).toEqual(["browser"]); }); it("enables javascript exec on the native browser declaration unless the spec is explicit", () => { - const defaulted = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { + const defaulted = resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { nativeTool: { type: "browser_20260701" }, }); expect(defaulted.nativeTool?.declaration.enable_javascript_exec).toBe(true); - const explicit = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { + const explicit = resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { nativeTool: { type: "browser_20260701", enable_javascript_exec: false }, }); expect(explicit.nativeTool?.declaration.enable_javascript_exec).toBe(false); }); it("swaps the placeholder tool for the native declaration in the payload", async () => { - const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "computer_20260701", enable_zoom: true } }); + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { nativeTool: { type: "computer_20260701", enable_zoom: true } }); const payload = { tools: [ { name: "computer", description: "placeholder", input_schema: {} }, @@ -168,10 +188,11 @@ describe("browser_20260701 action mapping", () => { describe("native tool executors", () => { it("translate native tool calls through the runtime spec executors", () => { - const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "browser_20260701" } }); + const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-8", { nativeTool: { type: "browser_20260701" } }); const executor = spec.toolExecutors[0]!; const actions: CuaAction[] = executor.toActions({ action: "left_click", target: { type: "ref", ref: "e3" } }); expect(actions).toEqual([{ type: "browser_click", ref: "e3" }]); + expect(() => executor.toActions({})).toThrow(/expected an object with an "action" field/); }); it("exports the anthropic namespace surface", () => { diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 41cd1b9d..701e5e34 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -72,9 +72,11 @@ Options: computer: OS-level input only. browser: CDP page tools (snapshot, find, click-by-ref, navigate, tabs). hybrid: both, deduplicated (computer_* + browser_* tools). - --native-tool Drive an Anthropic model through its native tool schema: - computer_20260701 (requires --mode computer) or - browser_20260701 (requires --mode browser) + --native-tool Drive an entitled Anthropic API key through an early-access + native schema. computer_20260701 requires --mode computer + and claude-fable-5, claude-opus-4-8, or claude-sonnet-5. + browser_20260701 requires --mode browser and + claude-opus-4-8 or claude-sonnet-5. --out Output file for screenshot subcommand --filter Restrict \`cua snapshot\` to interactive elements -o, --output Output format for --print: text (default) | jsonl diff --git a/packages/cli/test/cli-harness-validation.test.ts b/packages/cli/test/cli-harness-validation.test.ts index c8f20ce3..18158129 100644 --- a/packages/cli/test/cli-harness-validation.test.ts +++ b/packages/cli/test/cli-harness-validation.test.ts @@ -55,6 +55,18 @@ describe("mode/native-tool validation before provisioning", () => { expect(provisionBrowser).not.toHaveBeenCalled(); }); + it("rejects an ineligible Anthropic model without provisioning a browser", async () => { + await expect( + runActionCommand("url", [], flagsWith({ + model: "anthropic:claude-opus-4-7", + nativeTool: "browser_20260701", + })), + ).rejects.toThrow( + 'native tool "browser_20260701" is an allowlisted Anthropic API beta and does not support model "claude-opus-4-7"', + ); + expect(provisionBrowser).not.toHaveBeenCalled(); + }); + it("rejects an unsupported provider/mode pair without provisioning a browser", async () => { await expect( runActionCommand("url", [], flagsWith({ From df9a1db6fe05314bc125ab4a8bc6e63205d8d68d Mon Sep 17 00:00:00 2001 From: Rafael Garcia Date: Fri, 24 Jul 2026 11:59:16 -0400 Subject: [PATCH 2/2] Remove private beta investigation report --- docs/anthropic-native-tools-investigation.md | 186 ------------------- docs/architecture.md | 2 - 2 files changed, 188 deletions(-) delete mode 100644 docs/anthropic-native-tools-investigation.md diff --git a/docs/anthropic-native-tools-investigation.md b/docs/anthropic-native-tools-investigation.md deleted file mode 100644 index 39ae30c9..00000000 --- a/docs/anthropic-native-tools-investigation.md +++ /dev/null @@ -1,186 +0,0 @@ -# Anthropic native browser/computer tools investigation - -**Investigation date:** 2026-07-24 - -**Scope:** Anthropic's early-access `computer_20260701` and `browser_20260701` client tools in `@onkernel/cua-ai` / `@onkernel/cua-agent`. - -This report intentionally contains no API keys, request IDs, live-view URLs, or verbatim material from Anthropic's confidential early-access documents. Both supplied PDFs were read completely and used as the primary specification; extracted text stayed in a mode-0600 directory under `/tmp` and was not committed. - -## Conclusion - -The July native tools have **not been retired**. With the newly provisioned key, both tools return valid `tool_use` blocks through direct `/v1/messages` requests, through CUA's pi-ai serialization path, and through full Kernel-browser agent runs. - -The reproducible rejection is a **model/tool mismatch exposed by a CUA validation bug**: - -- `claude-opus-4-7` rejects both July tool types with HTTP 400. -- CUA previously accepted any Anthropic computer-use model with either July native tool. Unit tests even used Claude Opus 4.5, which the live API rejects. -- The repository's normal Anthropic examples/tests prominently use Claude Opus 4.7, making it easy for live QA to combine that model with `--native-tool` and reach the server-side 400. - -The other suspected causes were ruled out for the new key and the supported configuration: - -- **Entitlement:** the new key is entitled; correct early-access requests return 200. Entitlement remains an account/workspace prerequisite. -- **Header/version retirement:** the old June computer draft pair is gone, but CUA already sends the live July pair. The July pair returns 200. -- **Endpoint:** the tools work on the first-party Anthropic Messages endpoint, `POST https://api.anthropic.com/v1/messages`. -- **SDK/pi serialization:** CUA's placeholder-tool replacement and beta-header injection work live despite the installed Anthropic SDK not having types for these private tool versions. - -CUA now rejects unsupported model/tool combinations locally, before provisioning a browser, and enforces the native multi-action stop-on-first-failure result contract. - -## Compatibility matrix - -### Tool version and header - -| Tool declaration | `anthropic-beta` | Live result on eligible model | Status | -| --- | --- | --- | --- | -| `computer_20260601` | `computer-use-2026-06-01` | HTTP 400: beta value no longer recognized | Obsolete early draft; do not use | -| `computer_20260601` | `computer-use-2026-07-01` | HTTP 400: tool type unknown | Obsolete tool string | -| `computer_20260701` | omitted | HTTP 400: tool type unknown without beta | Header required | -| `computer_20260701` | `computer-use-2026-07-01` | HTTP 200 + `computer` tool use | Current early-access computer pair | -| `browser_20260701` | omitted | HTTP 400: tool type unknown without beta | Header required | -| `browser_20260701` | `browser-use-2026-07-01` | HTTP 200 + `browser` tool uses | Current early-access browser pair | -| `computer_20251124` | `computer-use-2025-11-24` | HTTP 200 on Claude Opus 4.7 | Current public computer-use fallback; not CUA's July `nativeTool` option | - -The June-to-July computer drift was already discovered in git commit `e6219bb` and merged in `3caf6cc` (#51). The current CUA header and tool strings are correct. - -### Model eligibility observed from the new key - -The model-list endpoint returned the models below. Each was probed with the correct July header/tool pair; a pass means the response contained the expected native `tool_use` block. - -| Model ID | `computer_20260701` | `browser_20260701` | -| --- | --- | --- | -| `claude-sonnet-5` | Pass | Pass | -| `claude-fable-5` | Pass | **Rejected** | -| `claude-opus-4-8` | Pass | Pass | -| `claude-opus-4-7` | **Rejected** | **Rejected** | -| `claude-sonnet-4-6` | **Rejected** | **Rejected** | -| `claude-opus-4-6` | **Rejected** | **Rejected** | -| `claude-opus-4-5-20251101` | **Rejected** | **Rejected** | -| `claude-haiku-4-5-20251001` | **Rejected** | **Rejected** | -| `claude-sonnet-4-5-20250929` | **Rejected** | **Rejected** | -| `claude-opus-4-1-20250805` | **Rejected** | **Rejected** | - -These lists are deliberately fail-closed in `packages/ai/src/native-tools.ts`. Re-run the live integration test before expanding them when Anthropic changes eligibility. - -### Request declaration and flags - -| Dimension | `computer_20260701` | `browser_20260701` | -| --- | --- | --- | -| Required declaration | `type`, name (normally `computer`) | `type`, name (normally `browser`) | -| Optional CUA-exposed fields | `display_number`, `enable_zoom`, `cache_control` | `enable_javascript_exec`, `cache_control` | -| Dimensions | CUA does not send width/height; the screenshot establishes the coordinate frame | No declared dimensions; viewport screenshots establish the frame | -| Zoom | `enable_zoom` defaults false at the API; both true and false passed | `zoom` is part of the browser action set | -| JavaScript | Not applicable | API default is false; CUA intentionally defaults it to true to match canonical browser mode. Explicit false and true both passed live. | -| `strict: true` | HTTP 400 | HTTP 400 | -| Both July tools together | Rejected: desktop and viewport coordinate frames cannot be mixed | Rejected for the same reason | -| Endpoint | First-party Anthropic `/v1/messages` only for this early-access version | First-party Anthropic `/v1/messages` only for this early-access version | -| Account requirement | Matching early-access entitlement | Matching early-access entitlement | - -### Action and result contract - -`computer_20260701` emits one action per `tool_use`: screenshot; left/right/middle/double/triple click; drag; mouse move/down/up; scroll; type; key (with repeat); hold key; wait; cursor position; and optional zoom. Coordinates are screenshot pixels. - -`browser_20260701` emits one action per `tool_use`: navigation/tabs; accessibility-tree and text reads; find/fill/scroll-to; viewport screenshot/zoom; ref- or coordinate-targeted pointer actions; keyboard/scroll/wait; and optional JavaScript execution. Coordinates are viewport pixels and element refs are scoped to their tab/document generation. - -Both tools can emit several `tool_use` blocks in one assistant turn. CUA executes them sequentially and returns one matching `tool_result` per block. Screenshot/zoom results contain images; structured reads and acknowledgements contain text. After the first failed action, CUA now skips every remaining call in that assistant turn and returns the tool-specific required error result rather than executing against stale state. - -## CUA implementation audit - -### Correct before this investigation - -- `packages/ai/src/providers/anthropic/native.ts` replaces pi-ai's permissive placeholder tool with the Anthropic-defined declaration after ordinary tool serialization. -- `packages/ai/src/providers.ts` routes CUA's synthetic native API IDs back through pi-ai's `anthropic-messages` transport and injects the matching beta header. -- Incoming native actions map onto CUA's canonical computer/browser actions, including key repeat, seconds-to-milliseconds duration conversion, browser refs, tabs, zoom, and JavaScript execution. -- Native CUA tools use sequential tool execution, and browser navigation results include tab context. -- Full native computer and browser runs completed against a Kernel browser. - -### Bugs fixed - -1. **Missing model eligibility validation.** `resolveNativeTool()` checked only provider and mode. It now rejects unsupported model families with an error that identifies the early-access requirement and lists the supported families. -2. **Incomplete multi-action failure semantics.** pi-agent-core's sequential mode continued with later calls after an earlier call failed. `CuaRuntimeSpec` now carries the provider-required skip message as data, and both `CuaAgent` and `CuaAgentHarness` block the rest of that assistant turn after the first failure. -3. **Documentation ambiguity.** Architecture and changelog text now distinguish the two headers, list the live-verified models, and state that the tools are allowlisted early access. -4. **No live serialization regression.** `packages/ai/test/anthropic-native.integration.test.ts` now checks both tools through CUA's actual pi-ai transport. - -## Official public documentation comparison - -Fetched on 2026-07-24: - -- [Public computer-use guide](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) -- [Claude API release notes](https://platform.claude.com/docs/en/release-notes/overview) - -The public guide still documents `computer_20251124` / `computer-use-2025-11-24` (and the older January pair). It does not document the July native computer version or the client-side July browser tool. The public release notes likewise do not announce either July pair. The installed `@anthropic-ai/sdk` types contain neither July tool string. - -That discrepancy is expected for allowlisted early access, but it means public docs cannot be used to infer that `computer_20260701` or `browser_20260701` is generally available. The confidential specifications plus live first-party API behavior are the source of truth for these pairs. - -## Sanitized evidence and commands - -Credential-presence check (does not print the value): - -```bash -test -n "${ANTHROPIC_API_KEY:-}" && echo 'ANTHROPIC_API_KEY is set' -``` - -Core direct probe shape (repeat with the browser declaration/header for browser use). The response filter omits IDs and usage: - -```bash -curl -sS https://api.anthropic.com/v1/messages \ - -H 'content-type: application/json' \ - -H "x-api-key: $ANTHROPIC_API_KEY" \ - -H 'anthropic-version: 2023-06-01' \ - -H 'anthropic-beta: computer-use-2026-07-01' \ - -d '{ - "model":"claude-opus-4-8", - "max_tokens":96, - "tools":[{"type":"computer_20260701","name":"computer","enable_zoom":true}], - "messages":[{"role":"user","content":"Use the computer tool to take one screenshot."}] - }' | jq '{stop_reason, blocks: [.content[] | {type, name, action: .input.action}]}' -``` - -Live CUA/pi-ai serialization regression: - -```bash -cd packages/ai -npx vitest --run --config vitest.integration.config.ts \ - test/anthropic-native.integration.test.ts -``` - -Full Kernel-browser smoke tests: - -```bash -MODEL_REF=anthropic:claude-opus-4-8 CONFIG=native-computer \ - NODE_OPTIONS=--conditions=source \ - npx tsx packages/agent/examples/anthropic-native-smoke.ts - -MODEL_REF=anthropic:claude-opus-4-8 CONFIG=native-browser \ - NODE_OPTIONS=--conditions=source \ - npx tsx packages/agent/examples/anthropic-native-smoke.ts -``` - -Repository validation: - -```bash -npm run build -npm run typecheck -npm test --workspace @onkernel/cua-ai -npm test --workspace @onkernel/cua-agent -npm test --workspace @onkernel/cua-cli -cd packages/ai -npx vitest --run --config vitest.integration.config.ts \ - test/anthropic-native.integration.test.ts -``` - -Observed sanitized outcomes: - -- Direct baseline Messages request: 200. -- Direct July computer request on Opus 4.8: 200, `tool_use` action `screenshot`. -- Direct July browser request on Opus 4.8: 200, `tool_use` actions including `navigate`. -- CUA/pi-ai live integration: 2/2 passed. -- Kernel browser native-computer smoke: completed the `example.com` task. -- Kernel browser native-browser smoke: completed the `example.com` task. -- Unit tests added for fail-closed model validation and stop-on-first-failure behavior in both agent classes. - -## Recommended integration path - -1. For the strongest shared configuration, use `anthropic:claude-opus-4-8` with either July native tool and an API key from the entitled organization. -2. Claude Sonnet 5 also supports both tools. Claude Fable 5 supports only the July computer tool in the observed account. -3. Never pair Claude Opus 4.7 (or earlier listed models) with a July native tool. Omit `nativeTool` to use CUA's canonical function-tool path instead. -4. For a generally documented provider-native computer integration, evaluate `computer_20251124` separately. There is no public native browser-tool equivalent to the July early-access browser tool. -5. Keep the live integration test gated on the entitled key. Treat an unrecognized beta header as an entitlement/version incident; treat a “model does not support tool type” response as model eligibility drift. diff --git a/docs/architecture.md b/docs/architecture.md index 77e6aa82..2d3ea280 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -145,8 +145,6 @@ incoming `tool_use` inputs onto the same canonical actions the mode uses. The runtime spec also carries the native tool's stop-on-first-failure result text, which cua-agent applies without a provider conditional. Canonical vs native is therefore a wire-format and turn-contract difference over one execution path. -See [the July 2026 investigation](anthropic-native-tools-investigation.md) -for the compatibility matrix and live evidence. ## Layers