Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/agent/src/agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ function defaultSystemPrompt(provider: ReturnType<typeof providerForModel>): str
switch (provider) {
case "anthropic":
return anthropic.buildAnthropicSystemPrompt();
case "gemini":
case "google":
return gemini.buildGeminiSystemPrompt();
case "tzafon":
return tzafon.buildTzafonSystemPrompt();
Expand Down
2 changes: 1 addition & 1 deletion packages/agent/src/tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ export function createCuaComputerTools(args: CuaComputerToolsOptions): AgentTool
return createOpenAIComputerTools(args);
case "anthropic":
return createAnthropicComputerTools(args);
case "gemini":
case "google":
return createGeminiComputerTools(args);
case "tzafon":
return createTzafonComputerTools(args);
Expand Down
59 changes: 33 additions & 26 deletions packages/ai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ npm install @onkernel/cua-ai

## Quick Start

See [`examples/quickstart.ts`](./examples/quickstart.ts) for a runnable version
that reads `examples/screenshot.png` and uses `OPENAI_API_KEY`.

```ts
import { readFile } from "node:fs/promises";
import { complete, getCuaModel, openai } from "@onkernel/cua-ai";
Expand Down Expand Up @@ -56,20 +53,38 @@ computer-use model catalog and provider/tool metadata.

### Model Refs

`getCuaModel()` accepts only provider-qualified model refs:
`getCuaModel()` accepts only provider-qualified model refs of the form
`<provider>:<model-id>`:

```ts
getCuaModel("openai:gpt-5.5");
getCuaModel("anthropic:claude-opus-4-7");
getCuaModel("gemini:gemini-2.5-computer-use-preview-10-2025");
getCuaModel("google:gemini-2.5-computer-use-preview-10-2025");
getCuaModel("tzafon:tzafon.northstar-cua-fast");
getCuaModel("yutori:n1.5-latest");
```

`getCuaModel(ref)` returns a pi-ai `Model<Api>` object. You pass that model to
pi-ai functions like `complete(model, context)` or `stream(model, context)`.
`getCuaModel(ref)` returns a pi-ai `Model<Api>` you can pass to `complete()`
or `stream()`.

See [`docs/supported-models.md`](./docs/supported-models.md) for the current
list of CUA-supporting models per provider.

### CuaProvider

`CuaProvider` is the string union of provider IDs this package targets:

```ts
type CuaProvider = "openai" | "anthropic" | "google" | "tzafon" | "yutori";
```

The IDs match pi-ai's `Model.provider` values exactly. `providerForModel(model)`
narrows a pi-ai `Model<Api>` to a `CuaProvider`.

`listCuaModels(provider?)` returns:
### Listing Models

`listCuaModels(provider?)` returns every CUA-supporting model, optionally
filtered to one provider:

```ts
interface CuaModelInfo {
Expand All @@ -86,12 +101,8 @@ Top-level exports:

- `getCuaModel(ref: CuaModelRef): Model<Api>`
- `listCuaModels(provider?: CuaProvider): CuaModelInfo[]`
- `parseCuaModelRef(ref: string): { provider: CuaProvider; model: string }`
- `formatCuaModelRef(provider: CuaProvider, model: string): CuaModelRef`
- `providerForModel(model: Model<Api>): CuaProvider`
- `CUA_PROVIDERS: readonly CuaProvider[]`
- `CuaBatchSchema`, `CuaActionSchema`, `CuaNavigationSchema` TypeBox schemas
- `createCuaActionSchema(actions?)`, `createCuaBatchSchema(actions?)`
- `isCuaProvider(value: string): value is CuaProvider`

Provider namespaces expose `createComputerToolDefinitions({ actions? })` for
building model-facing pi-ai `Tool[]` definitions. Omit `actions` for the
Expand Down Expand Up @@ -130,9 +141,8 @@ Current coordinate contracts:
- `yutori`: normalized coordinates in the 0-1000 range ([source](https://docs.yutori.com/reference/navigator), [SDK helper](https://github.com/yutori-ai/yutori-sdk-python/blob/main/yutori/navigator/coordinates.py))
- `tzafon`: normalized coordinates in the 0-999 range ([source](https://docs.lightcone.ai/guides/coordinates/), [model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast))

`CuaActionSchema` validates one normalized computer action. The action
vocabulary is intentionally provider-neutral and OpenAI-shaped because it maps
cleanly to most browser computer-use APIs:
The action vocabulary is intentionally provider-neutral and OpenAI-shaped
because it maps cleanly to most browser computer-use APIs:

```ts
type CuaAction =
Expand Down Expand Up @@ -171,21 +181,21 @@ type CuaActionGoto = {
};
```

`CuaBatchSchema` validates the input for a batched computer tool:
The provider namespace `createComputerToolDefinitions()` emits a
`batch_computer_actions` tool whose input is:

```ts
type CuaBatchInput = {
actions: CuaAction[];
};
```

Use it for a tool like `batch_computer_actions`, where the model can plan
several writes and reads in one call. Read actions such as `screenshot`, `url`,
and `cursor_position` can be interleaved with writes so your executor can return
fresh state in the same order.
The model can plan several writes and reads in one call. Read actions such as
`screenshot`, `url`, and `cursor_position` can be interleaved with writes so
your executor can return fresh state in the same order.

`CuaNavigationSchema` validates a smaller convenience tool for high-level
navigation:
When `actions` is omitted, the OpenAI namespace also emits a `computer_use_extra`
navigation tool whose input is:

```ts
type CuaNavigationInput = {
Expand All @@ -194,9 +204,6 @@ type CuaNavigationInput = {
};
```

Use it for a simple `computer_use_extra`-style tool when you want navigation
available without exposing the full batch action surface.

Provider namespaces:

- `openai`: `createComputerToolDefinitions`, `COMPUTER_TOOL_COORDINATES`, OpenAI CUA action schemas, and `OPENAI_BATCH_INSTRUCTIONS`
Expand Down
65 changes: 65 additions & 0 deletions packages/ai/docs/supported-models.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Supported CUA Models

`@onkernel/cua-ai` accepts any pi-ai model whose ID is annotated as
CUA-supporting in `CUA_MODEL_ANNOTATIONS` (see
[`src/models.ts`](../src/models.ts)). Annotations are either a `family`
match (root + dated snapshots) or an `exact` ID match. Each annotation
cites the provider's CUA docs.

The list below is the current snapshot. Run
`listCuaModels(provider?)` for the live list — it merges pi-ai's registry
with CUA-only entries that pi-ai does not ship yet.

## `openai`

API: `openai-responses` · coordinates: pixel

Family matches (all dated snapshots accepted):

- `gpt-5.4` ([docs](https://developers.openai.com/api/docs/models/gpt-5.4))
- `gpt-5.5` ([docs](https://developers.openai.com/api/docs/models/gpt-5.5))

## `anthropic`

API: `anthropic-messages` · coordinates: pixel

Family matches (all dated snapshots accepted):

- `claude-3-7-sonnet`
- `claude-opus-4`
- `claude-sonnet-4`
- `claude-haiku-4`

Source: [Anthropic computer use docs](https://docs.anthropic.com/en/docs/build-with-claude/computer-use).

## `google`

API: `google-generative-ai` · coordinates: normalized 0–999

Exact IDs:

- `gemini-3-flash-preview`
- `gemini-2.5-computer-use-preview-10-2025`

Source: [Gemini computer use docs](https://ai.google.dev/gemini-api/docs/computer-use).

## `tzafon`

API: `tzafon-responses` · coordinates: normalized 0–999

Exact IDs:

- `tzafon.northstar-cua-fast` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast))

## `yutori`

API: `yutori-chat-completions` · coordinates: normalized 0–1000

Exact IDs:

- `n1-latest`
- `n1-20260203`
- `n1.5-latest`
- `n1.5-20260428`

Source: [Yutori Navigator reference](https://docs.yutori.com/reference/navigator).
2 changes: 1 addition & 1 deletion packages/ai/examples/quickstart.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ const screenshot = await readFile(screenshotPath);
// const tools = anthropic.createComputerToolDefinitions({ actions: ["click"] });
//
// const apiKey = process.env.GOOGLE_API_KEY;
// const modelRef = "gemini:gemini-2.5-computer-use-preview-10-2025";
// const modelRef = "google:gemini-2.5-computer-use-preview-10-2025";
// const model = getCuaModel(modelRef);
// const tools = gemini.createComputerToolDefinitions({ actions: ["click"] });

Expand Down
41 changes: 39 additions & 2 deletions packages/ai/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,45 @@ import { registerCuaProviders } from "./providers.js";

export * from "@earendil-works/pi-ai";

export * from "./models.js";
export * from "./providers/common.js";
export {
getCuaModel,
isCuaProvider,
listCuaModels,
providerForModel,
} from "./models.js";
export type { CuaModelInfo, CuaModelRef, CuaProvider } from "./models.js";

export {
CUA_ACTION_TYPES,
CUA_BATCH_TOOL_NAME,
CUA_NAVIGATION_TOOL_NAME,
createComputerToolDefinitions,
} from "./providers/common.js";
export type {
ComputerToolCoordinateSystem,
CreateComputerToolDefinitionsOptions,
CuaAction,
CuaActionBack,
CuaActionClick,
CuaActionCursorPosition,
CuaActionDoubleClick,
CuaActionDrag,
CuaActionForward,
CuaActionGoto,
CuaActionKeypress,
CuaActionMouseDown,
CuaActionMouseUp,
CuaActionMove,
CuaActionScreenshot,
CuaActionScroll,
CuaActionType,
CuaActionTypeText,
CuaActionUrl,
CuaActionWait,
CuaBatchInput,
CuaNavigationInput,
} from "./providers/common.js";

export * as anthropic from "./providers/anthropic/index.js";
export * as gemini from "./providers/gemini/index.js";
export * as openai from "./providers/openai/index.js";
Expand Down
52 changes: 13 additions & 39 deletions packages/ai/src/models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import {
getModels,
} from "@earendil-works/pi-ai";

export type CuaProvider = "openai" | "anthropic" | "gemini" | "tzafon" | "yutori";
export type CuaProvider = "openai" | "anthropic" | "google" | "tzafon" | "yutori";
export type CuaModelRef = `${CuaProvider}:${string}`;

export interface CuaModelInfo {
Expand All @@ -15,7 +15,7 @@ export interface CuaModelInfo {
name: string;
}

export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "gemini", "tzafon", "yutori"];
export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "tzafon", "yutori"];

// CUA support annotations.
//
Expand Down Expand Up @@ -52,7 +52,7 @@ export const CUA_MODEL_ANNOTATIONS: Record<CuaProvider, readonly CuaModelAnnotat
{ match: { kind: "family", family: "claude-sonnet-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" },
{ match: { kind: "family", family: "claude-haiku-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" },
],
gemini: [
google: [
{ match: { kind: "exact", id: "gemini-3-flash-preview" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" },
{ match: { kind: "exact", id: "gemini-2.5-computer-use-preview-10-2025" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" },
],
Expand All @@ -78,8 +78,8 @@ const CUA_MODEL_OVERRIDES: Record<CuaProvider, readonly Model<Api>[]> = {
cuaModel("openai", "gpt-5.5-2026-04-23", "GPT-5.5 (2026-04-23)"),
],
anthropic: [],
gemini: [
cuaModel("gemini", "gemini-2.5-computer-use-preview-10-2025", "Gemini 2.5 Computer Use Preview"),
google: [
cuaModel("google", "gemini-2.5-computer-use-preview-10-2025", "Gemini 2.5 Computer Use Preview"),
],
tzafon: [
cuaModel("tzafon", "tzafon.northstar-cua-fast", "Tzafon Northstar CUA Fast"),
Expand All @@ -106,7 +106,6 @@ export function parseCuaModelRef(ref: string): { provider: CuaProvider; model: s
}

export function formatCuaModelRef(provider: CuaProvider, model: string): CuaModelRef {
if (!model.trim()) throw new Error("model id is empty");
return `${provider}:${model}` as CuaModelRef;
}

Expand All @@ -119,7 +118,7 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] {
const ref = formatCuaModelRef(p, model.id);
byRef.set(ref, { ref, provider: p, model: model.id, name: model.name });
}
for (const model of getModels(piProviderFor(p) as never) as Model<Api>[]) {
for (const model of getModels(p as never) as Model<Api>[]) {
if (!supportsCuaProvider(p, model.id)) continue;
const ref = formatCuaModelRef(p, model.id);
if (byRef.has(ref)) continue;
Expand All @@ -140,49 +139,24 @@ export function getCuaModel(ref: CuaModelRef): Model<Api> {
if (!supportsCuaProvider(provider, modelId)) {
throw new Error(`unsupported CUA model "${ref}"`);
}
const fromRegistry = getModel(piProviderFor(provider) as never, modelId as never) as Model<Api> | undefined;
const fromRegistry = getModel(provider as never, modelId as never) as Model<Api> | undefined;
if (fromRegistry) return fromRegistry;
const override = CUA_MODEL_OVERRIDES[provider].find((m) => m.id === modelId);
if (override) return override;
throw new Error(`CUA model "${ref}" is supported but not registered. Add it to pi-ai (models.dev) or CUA_MODEL_OVERRIDES.`);
}

export function providerForModel(model: Model<Api>): CuaProvider {
switch (model.provider) {
case "openai":
return "openai";
case "anthropic":
return "anthropic";
case "google":
return "gemini";
case "tzafon":
return "tzafon";
case "yutori":
return "yutori";
default:
throw new Error(`unsupported CUA model provider "${model.provider}"`);
if (!isCuaProvider(model.provider)) {
throw new Error(`unsupported CUA model provider "${model.provider}"`);
}
return model.provider;
}

export function isCuaProvider(value: string): value is CuaProvider {
return (CUA_PROVIDERS as readonly string[]).includes(value);
}

function piProviderFor(provider: CuaProvider): string {
switch (provider) {
case "openai":
return "openai";
case "anthropic":
return "anthropic";
case "gemini":
return "google";
case "tzafon":
return "tzafon";
case "yutori":
return "yutori";
}
}

function supportsCuaProvider(provider: CuaProvider, modelId: string): boolean {
return findCuaAnnotation(provider, modelId) !== undefined;
}
Expand All @@ -204,8 +178,8 @@ function cuaModel(provider: CuaProvider, id: string, name: string): Model<Api> {
const base = {
id,
name,
provider: piProviderFor(provider),
reasoning: provider === "openai" || provider === "anthropic" || provider === "gemini",
provider,
reasoning: provider === "openai" || provider === "anthropic" || provider === "google",
input: ["text", "image"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
} satisfies Partial<Model<Api>>;
Expand All @@ -215,7 +189,7 @@ function cuaModel(provider: CuaProvider, id: string, name: string): Model<Api> {
return { ...base, api: "openai-responses", baseUrl: "https://api.openai.com/v1", contextWindow: 400_000, maxTokens: 32_768 } as Model<Api>;
case "anthropic":
return { ...base, api: "anthropic-messages", baseUrl: "https://api.anthropic.com", contextWindow: 200_000, maxTokens: 64_000 } as Model<Api>;
case "gemini":
case "google":
return { ...base, api: "google-generative-ai", baseUrl: "https://generativelanguage.googleapis.com/v1beta", contextWindow: 1_048_576, maxTokens: 65_536 } as Model<Api>;
case "tzafon":
return { ...base, api: "tzafon-responses", baseUrl: "https://api.lightcone.ai", contextWindow: 128_000, maxTokens: 4_096 } as Model<Api>;
Expand Down
Loading
Loading