Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,14 +178,15 @@ Catalog compilation composes provider behavior rather than replacing the whole
catalog:

- Ordinary function tools stay ordinary.
- Anthropic native browser/computer declarations replace only their own
placeholders and merge required beta headers with caller headers.
- Anthropic's browser/computer client-toolset declarations replace only their
own placeholders; their member calls and results retain `toolset_name`
through pi's function-tool transcript.
- OpenAI streams through pi's builtin Responses transport and its automatic
prompt caching by default; a Browser Loop-owned adapter handles OpenAI's native
computer tool and tool-search namespace round-trips.
- Anthropic's native browser tool falls back to an equivalent function-tool
declaration when the active credential cannot access `browser_20260701`;
the selected tool identity, name, schema, and executor remain unchanged.
- Anthropic's `computer_toolset_20260801` and `browser_toolset_20260801` may be
selected together; calls remain separated by their computer and viewport
coordinate frames.
- Google's current predefined browser toolset serializes one `computer_use`
declaration plus exact exclusions through the Browser Loop-owned Interactions API
adapter. Excluded calls fail with a named catalog error instead of reaching
Expand Down
4 changes: 4 additions & 0 deletions packages/browser-loop/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Unreleased

- Update Anthropic's computer and browser integrations to use the
`computer_toolset_20260801` and `browser_toolset_20260801` client toolsets.
Member calls/results preserve `toolset_name`, sequential batches stop at the
first failure, and both toolsets may be selected together.
- Rename the product and unpublished package to Browser Loop: the repository is
prepared for `kernel/browser-loop`, the package is `@onkernel/browser-loop`,
and release tags use `browser-loop/v*`. The public `loop` namespace, `Loop*`
Expand Down
35 changes: 23 additions & 12 deletions packages/browser-loop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,8 +318,8 @@ function tools.
```ts
const tools = [
loop.providers.anthropic.tools.computer({
version: "20260701",
enableZoom: true,
version: "20260801",
zoom: true,
}),
loop.tools.browser.snapshot(),
];
Expand All @@ -330,9 +330,9 @@ Available groups:
```ts
loop.providers.openai.tools.computer();

loop.providers.anthropic.source;
loop.providers.anthropic.tools.computer({ version: "20260701" });
loop.providers.anthropic.tools.browser({ version: "20260701" });
loop.providers.anthropic.sources;
loop.providers.anthropic.tools.computer({ version: "20260801" });
loop.providers.anthropic.tools.browser({ version: "20260801" });

loop.providers.google.source;
loop.providers.google.toolsets.browser({ exclude: ["right_click"] });
Expand All @@ -341,6 +341,16 @@ loop.providers.google.toolsets.browser({ exclude: ["right_click"] });
loop.toolsets.browser();
```

Anthropic's browser toolset leaves `javascript_exec` disabled unless it is
enabled explicitly:

```ts
loop.providers.anthropic.tools.browser({
version: "20260801",
javascript: true,
});
```

The Google browser set exposes the current predefined action names and uses
normalized coordinates in `[0, 999]`. Its native `computer_use` declaration
excludes every unselected browser action. If Google emits an excluded name
Expand All @@ -357,11 +367,12 @@ than on the wire. [Models and native surfaces](docs/supported-models.md) lists
which models carry which surface.

Provider-native caller-visible names are fixed by protocol. Version/tool/model
mismatches fail during catalog compilation. If an Anthropic credential cannot
access `browser_20260701`, Browser Loop retries with an equivalent `browser` function
tool and remembers that choice for the credential and process. Every
`loop.providers.*` tool surface exposes its first-party `source` (or versioned
`sources`), and every returned provider spec carries the applicable URL.
mismatches fail during catalog compilation. Anthropic's
`computer_toolset_20260801` and `browser_toolset_20260801` declarations may be
selected together; Browser Loop preserves member `toolset_name` fields and
sequential batch semantics across pi's transcript.
Every `loop.providers.*` tool surface exposes its first-party `source` or
`sources`, and every returned provider spec carries the applicable URL.

## Catalog compilation

Expand Down Expand Up @@ -434,8 +445,8 @@ require different transports fails to compile.
`openai-computer-use` api instead, which a Browser Loop adapter handles; that same
adapter also covers tool-search namespace round-trips regardless of api,
since pi's builtin transport does not replay them.
- **Anthropic**: exact native declarations, beta-header composition, and
adaptive model preparation. No api fork — every Anthropic model streams
- **Anthropic**: client-toolset declarations, member/result transcript
adaptation, and adaptive model preparation. Every Anthropic model streams
through pi's builtin transport.
- **Google**: a model selected without Google's native browser toolset streams
through pi's builtin transport. Selecting
Expand Down
9 changes: 4 additions & 5 deletions packages/browser-loop/docs/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,7 @@ models and need their own entry. Each cites first-party documentation.

| provider | models | surfaces |
| --- | --- | --- |
| `anthropic` | `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` families | computer, browser |
| `anthropic` | `claude-fable-5` family | computer |
| `anthropic` | `claude-fable-5`, `claude-mythos-5`, `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` families | computer, browser |
| `openai` | `gpt-5.6-sol`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.5` | computer |
| `google` | `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-2.5-computer-use-preview-10-2025` | browser |

Expand Down Expand Up @@ -93,9 +92,9 @@ npx tsx packages/browser-loop/scripts/native-action-probe.ts --provider openai -

Update that provider's adapter under `src/pi/providers/` to execute the actions the
probe returns, then add or adjust the `COMPUTER_USE_NATIVE_SURFACES` entry, citing the
provider's documentation. Anthropic's computer tool version and its
`computer-use-*` beta header are chosen by pi-ai per model, so a new dated
version there usually means bumping pi-ai rather than editing this package.
provider's documentation. Anthropic's client toolsets are adapted in
`src/pi/providers/anthropic/toolsets.ts`; a new dated toolset version requires
updating its declarations and member transcript adapter together.

**A model rejects a tool Browser Loop sends.** Add a `LOOP_MODEL_QUIRKS` entry with the
observed error as its `reason`, scoped as narrowly as the evidence supports: a
Expand Down
4 changes: 2 additions & 2 deletions packages/browser-loop/examples/anthropic-native-smoke.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ const CONFIGS: Record<string, readonly LoopAgentTool[]> = {
computer: loop.toolsets.computer(),
browser: loop.toolsets.browser(),
mixed: loop.toolsets.mixed(),
"native-computer": [loop.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true })],
"native-browser": [loop.providers.anthropic.tools.browser({ version: "20260701" })],
"native-computer": [loop.providers.anthropic.tools.computer({ version: "20260801", zoom: true })],
"native-browser": [loop.providers.anthropic.tools.browser({ version: "20260801" })],
};

const PROMPT = [
Expand Down
2 changes: 1 addition & 1 deletion packages/browser-loop/examples/shared/tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ export function toolsForModel(model: LoopModelRef): LoopAgentTool[] {
// Claude 5 can use Anthropic's native browser tool; older models use portable
// Loop tools plus the explicit semantic action-plan surface.
return loop.providers.anthropic.supports.browser(modelId)
? [loop.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]
? [loop.providers.anthropic.tools.browser({ version: "20260801", javascript: true })]
: structuredBrowserTools();
case "google":
// Current Gemini computer-use models expect Google's predefined browser actions.
Expand Down
70 changes: 68 additions & 2 deletions packages/browser-loop/src/core/actions/browser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,20 @@ export const BROWSER_ACTION_TYPES = [
"browser_find",
"browser_click",
"browser_hover",
"browser_mouse_down",
"browser_mouse_up",
"browser_drag",
"browser_fill",
"browser_scroll_to",
"browser_scroll",
"browser_type",
"browser_key",
"browser_hold_key",
"browser_navigate",
"browser_list_tabs",
"browser_new_tab",
"browser_switch_tab",
"browser_close_tab",
"browser_screenshot",
"browser_evaluate",
] as const;
Expand Down Expand Up @@ -138,6 +143,20 @@ export interface BrowserActionHover {
tab_id?: string;
}

export interface BrowserActionMouseDown {
type: "browser_mouse_down";
x: number;
y: number;
tab_id?: string;
}

export interface BrowserActionMouseUp {
type: "browser_mouse_up";
x: number;
y: number;
tab_id?: string;
}

export interface BrowserActionDrag {
type: "browser_drag";
from: { x: number; y: number };
Expand Down Expand Up @@ -180,9 +199,16 @@ export interface BrowserActionKey {
tab_id?: string;
}

export interface BrowserActionHoldKey {
type: "browser_hold_key";
text: string;
duration: number;
tab_id?: string;
}

export interface BrowserActionNavigate {
type: "browser_navigate";
/** A URL, or the sentinels "back" / "forward" for history navigation. */
/** A URL, or the sentinels "back", "forward", or "reload". */
url: string;
tab_id?: string;
}
Expand All @@ -195,6 +221,16 @@ export interface BrowserActionNewTab {
type: "browser_new_tab";
}

export interface BrowserActionSwitchTab {
type: "browser_switch_tab";
tab_id: string;
}

export interface BrowserActionCloseTab {
type: "browser_close_tab";
tab_id: string;
}

export interface BrowserActionScreenshot {
type: "browser_screenshot";
/** Optional crop region, [x0, y0, x1, y1] in viewport pixels. */
Expand All @@ -216,15 +252,20 @@ export type BrowserAction =
| BrowserActionFind
| BrowserActionClick
| BrowserActionHover
| BrowserActionMouseDown
| BrowserActionMouseUp
| BrowserActionDrag
| BrowserActionFill
| BrowserActionScrollTo
| BrowserActionScroll
| BrowserActionTypeText
| BrowserActionKey
| BrowserActionHoldKey
| BrowserActionNavigate
| BrowserActionListTabs
| BrowserActionNewTab
| BrowserActionSwitchTab
| BrowserActionCloseTab
| BrowserActionScreenshot
| BrowserActionEvaluate;

Expand Down Expand Up @@ -403,6 +444,14 @@ export function createBrowserActionSchemaByType(options: BrowserActionSchemaOpti
},
{ additionalProperties: false },
),
browser_mouse_down: Type.Object(
{ type: Type.Literal("browser_mouse_down"), x: Type.Number(), y: Type.Number(), tab_id: TabId() },
{ additionalProperties: false },
),
browser_mouse_up: Type.Object(
{ type: Type.Literal("browser_mouse_up"), x: Type.Number(), y: Type.Number(), tab_id: TabId() },
{ additionalProperties: false },
),
browser_drag: Type.Object(
{
type: Type.Literal("browser_drag"),
Expand Down Expand Up @@ -459,16 +508,33 @@ export function createBrowserActionSchemaByType(options: BrowserActionSchemaOpti
},
{ additionalProperties: false },
),
browser_hold_key: Type.Object(
{
type: Type.Literal("browser_hold_key"),
text: Type.String({ description: "Key or chord to hold." }),
duration: Type.Number({ minimum: 0, maximum: 30 }),
tab_id: TabId(),
},
{ additionalProperties: false },
),
browser_navigate: Type.Object(
{
type: Type.Literal("browser_navigate"),
url: Type.String({ description: "URL to navigate to, or \"back\" / \"forward\" for history navigation." }),
url: Type.String({ description: "URL to navigate to, or \"back\", \"forward\", or \"reload\"." }),
tab_id: TabId(),
},
{ additionalProperties: false },
),
browser_list_tabs: Type.Object({ type: Type.Literal("browser_list_tabs") }, { additionalProperties: false }),
browser_new_tab: Type.Object({ type: Type.Literal("browser_new_tab") }, { additionalProperties: false }),
browser_switch_tab: Type.Object(
{ type: Type.Literal("browser_switch_tab"), tab_id: Type.String() },
{ additionalProperties: false },
),
browser_close_tab: Type.Object(
{ type: Type.Literal("browser_close_tab"), tab_id: Type.String() },
{ additionalProperties: false },
),
browser_screenshot: Type.Object(
{
type: Type.Literal("browser_screenshot"),
Expand Down
32 changes: 23 additions & 9 deletions packages/browser-loop/src/core/anthropic-native.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ function asNativeInput(args: unknown): NativeInput {

const MAX_KEY_REPEAT = 100;

/** Map one `computer_20260701` tool input onto canonical computer-plane actions. */
/** Map one Anthropic computer toolset member onto canonical computer-plane actions. */
export function mapNativeComputerInput(input: NativeInput): ComputerUseAction[] {
switch (input.action) {
case "screenshot":
Expand Down Expand Up @@ -52,19 +52,19 @@ export function mapNativeComputerInput(input: NativeInput): ComputerUseAction[]
}
case "hold_key":
// Canonical keypress duration is milliseconds; the native tool speaks seconds.
return [{ type: "keypress", keys: [text(input)], duration: durationSeconds(input) * 1000 }];
return [{ type: "keypress", keys: [text(input)], duration: durationSeconds(input, 300) * 1000 }];
case "wait":
return [{ type: "wait", ms: durationSeconds(input) * 1000 }];
return [{ type: "wait", ms: durationSeconds(input, 300) * 1000 }];
case "cursor_position":
return [{ type: "cursor_position" }];
case "zoom":
return [{ type: "zoom", region: region(input.region) }];
default:
throw new Error(`unsupported computer_20260701 action "${input.action}"`);
throw new Error(`unsupported computer toolset member "${input.action}"`);
}
}

/** Map one `browser_20260701` tool input onto canonical browser-plane actions. */
/** Map one Anthropic browser toolset member onto canonical browser-plane actions. */
export function mapNativeBrowserInput(input: NativeInput): ComputerUseAction[] {
const tab = tabId(input);
switch (input.action) {
Expand Down Expand Up @@ -100,12 +100,20 @@ export function mapNativeBrowserInput(input: NativeInput): ComputerUseAction[] {
return [{ type: "browser_click", ...pageTarget(input.target), ...modifiers(input.modifiers), ...tab }];
case "right_click":
return [{ type: "browser_click", ...pageTarget(input.target), button: "right", ...modifiers(input.modifiers), ...tab }];
case "middle_click":
return [{ type: "browser_click", ...pageTarget(input.target), button: "middle", ...modifiers(input.modifiers), ...tab }];
case "double_click":
return [{ type: "browser_click", ...pageTarget(input.target), num_clicks: 2, ...modifiers(input.modifiers), ...tab }];
case "triple_click":
return [{ type: "browser_click", ...pageTarget(input.target), num_clicks: 3, ...modifiers(input.modifiers), ...tab }];
case "hover":
return [{ type: "browser_hover", ...pageTarget(input.target), ...tab }];
case "mouse_move":
return [{ type: "browser_hover", ...coordinateTarget(input.target, "target"), ...tab }];
case "left_mouse_down":
return [{ type: "browser_mouse_down", ...coordinateTarget(input.target, "target"), ...tab }];
case "left_mouse_up":
return [{ type: "browser_mouse_up", ...coordinateTarget(input.target, "target"), ...tab }];
case "left_click_drag":
return [{ type: "browser_drag", from: coordinateTarget(input.from, "from"), to: coordinateTarget(input.target, "target"), ...tab }];
case "scroll":
Expand All @@ -124,12 +132,18 @@ export function mapNativeBrowserInput(input: NativeInput): ComputerUseAction[] {
const repeat = clampRepeat(input.repeat);
return Array.from({ length: repeat }, () => ({ type: "browser_key" as const, text: text(input), ...tab }));
}
case "hold_key":
return [{ type: "browser_hold_key", text: text(input), duration: durationSeconds(input, 30), ...tab }];
case "wait":
return [{ type: "wait", ms: durationSeconds(input) * 1000 }];
return [{ type: "wait", ms: durationSeconds(input, 30) * 1000 }];
case "switch_tab":
return [{ type: "browser_switch_tab", tab_id: requireString(input.tab_id, "tab_id") }];
case "close_tab":
return [{ type: "browser_close_tab", tab_id: requireString(input.tab_id, "tab_id") }];
case "javascript_exec":
return [{ type: "browser_evaluate", code: text(input), ...tab }];
default:
throw new Error(`unsupported browser_20260701 action "${input.action}"`);
throw new Error(`unsupported browser toolset member "${input.action}"`);
}
}

Expand Down Expand Up @@ -191,10 +205,10 @@ function requireString(value: unknown, field: string): string {
return value;
}

function durationSeconds(input: NativeInput): number {
function durationSeconds(input: NativeInput, maximum: number): number {
const value = input.duration;
if (typeof value !== "number" || !Number.isFinite(value) || value < 0) throw new Error("invalid duration: expected a non-negative number");
return Math.min(value, 100);
return Math.min(value, maximum);
}

function clampRepeat(value: unknown): number {
Expand Down
Loading
Loading