Add provider-aware CUA AgentHarness - #12
Conversation
|
Firetiger deploy monitoring skipped This PR didn't match the auto-monitor filter configured on your GitHub connection:
Reason: PR modifies CUA agent harness in packages/@onkernel/cua-agent, not the kernel API endpoints (packages/api/cmd/api/) or Temporal workflows (packages/api/lib/temporal) specified in the filter. To monitor this PR anyway, reply with |
37a8a22 to
8d9ad6f
Compare
8d9ad6f to
ad18df8
Compare
|
Firetiger deploy monitoring skipped This PR didn't match the auto-monitor filter configured on your GitHub connection:
Reason: PR modifies agent harness code in packages/agent/, not kernel API endpoints (packages/api/cmd/api/) or Temporal workflows (packages/api/lib/temporal) as specified in the filter. To monitor this PR anyway, reply with |
| systemPrompt?: string; | ||
| systemPrompt?: AgentHarnessOptions<TSkill, TPromptTemplate, AgentTool>["systemPrompt"]; | ||
| getApiKey?: (provider: string) => Promise<string | undefined> | string | undefined; | ||
| getApiKeyAndHeaders?: AgentHarnessOptions<TSkill, TPromptTemplate, AgentTool>["getApiKeyAndHeaders"]; |
There was a problem hiding this comment.
why both of these methods (getApiKey and getApiKeyAndHeaders)? seems like pi's AgentHarnessOptions only has getApiKeyAndHeaders so i lean towards simplifying to that. also why does BaseCuaAgentHarnessOptions omit getApiKeyAndHeaders from AgentHarnessOptions only to add it back verbatim from AgentHarnessOptions?
| export type CuaHarnessOptions = Omit<AgentOptions, "initialState"> & { | ||
| type BaseCuaAgentHarnessOptions<TSkill extends Skill, TPromptTemplate extends PromptTemplate> = Omit< | ||
| AgentHarnessOptions<TSkill, TPromptTemplate, AgentTool>, | ||
| "env" | "session" | "model" | "tools" | "systemPrompt" | "getApiKeyAndHeaders" |
There was a problem hiding this comment.
why omit env and session tools systemPrompt and getApiKeyAndHeaders only to add them back in what appears to be basically the same definition that AgentHarnessOptions defines?
| @@ -0,0 +1,55 @@ | |||
| import { mkdir, writeFile } from "node:fs/promises"; | |||
There was a problem hiding this comment.
this file should be typescript not mjs
There was a problem hiding this comment.
it should also have a tsdoc style comment at the top explaining why it exists
| import { InMemorySessionStorage } from "./vendor/pi-agent-core/harness/session/storage/memory"; | ||
|
|
||
| type CuaRuntimeInput = CuaModelRef | Model<Api>; | ||
| type CuaRuntimeSpec = ReturnType<typeof resolveCuaRuntimeSpec>; |
There was a problem hiding this comment.
all main types should have tsdoc-style comments explaining to a newbie what the type's purpose is
|
|
||
| clearFollowUpQueue(): void { | ||
| this.agent.clearFollowUpQueue(); | ||
| override get state(): CuaAgentState { |
There was a problem hiding this comment.
comment explaining the override puprose here
There was a problem hiding this comment.
what's the point of this method? does AgentHarness not expose this already via session or whatever? i think we should remove this and if people want a transcript they can access agent.state.messages
| @@ -1,7 +1,6 @@ | |||
| import Kernel from "@onkernel/sdk"; | |||
There was a problem hiding this comment.
there should be an e2e live test for Agent and AgentHarness model switching after a turn and verifying the next turn works
| @@ -54,17 +54,15 @@ lifecycle events, custom streaming, and explicit prompt/continue/queue control. | |||
| Reach for the harness shape when you want an app layer around the loop: | |||
| session/transcript helpers, resource and prompt entry points, provider/auth | |||
There was a problem hiding this comment.
this example should use something from AgentHarness, e.g. session, to do this. and we should get rid of gettranscript
56040f6 to
e3f2fc1
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed:
setModelreverts active tool selection to constructor value- CuaAgentHarness now updates its tracked requested active tools in an overridden setActiveTools method so setModel preserves runtime tool selections, with a test added to prevent regression.
Or push these changes by commenting:
@cursor push 3b9b2810e6
Preview (3b9b2810e6)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -263,7 +263,7 @@
TPromptTemplate extends PromptTemplate = PromptTemplate,
> extends AgentHarness<TSkill, TPromptTemplate, AgentTool> {
private readonly runtime: CuaRuntimeController;
- private readonly requestedActiveToolNames?: string[];
+ private requestedActiveToolNames?: string[];
constructor(options: CuaAgentHarnessOptions<TSkill, TPromptTemplate>) {
const {
@@ -318,6 +318,11 @@
}
await super.setModel(this.runtime.model);
}
+
+ override async setActiveTools(toolNames: string[]): Promise<void> {
+ this.requestedActiveToolNames = [...toolNames];
+ await super.setActiveTools(toolNames);
+ }
}
function composeOnPayload(first: AgentOptions["onPayload"], second: AgentOptions["onPayload"]): AgentOptions["onPayload"] {
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -217,4 +217,18 @@
expect(harness.agent.state.model.id).toBe(runtime.model.id);
expect(harness.agent.state.tools).toHaveLength(runtime.toolDefinitions.length);
});
+
+ it("preserves active tool selection when setModel refreshes tools", async () => {
+ const harness = new CuaAgentHarness({
+ ...(await createHarnessServices()),
+ browser,
+ client,
+ model: "openai:gpt-5.5",
+ });
+
+ await harness.setActiveTools([]);
+ await harness.setModel("google:gemini-3-pro-preview");
+
+ expect(harness.agent.state.tools).toEqual([]);
+ });
});You can send follow-ups to the cloud agent here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Runtime spec update not rolled back on setModel failure
- CuaAgentHarness.setModel now resolves the target runtime spec up front and only updates the controller runtime after tool and model updates succeed, preventing stale runtime mutations on failure.
- ✅ Fixed: onPayloadFor redundantly re-resolves the CUA runtime spec
- CuaRuntimeController.onPayloadFor now reuses the cached runtime spec when the requested model matches the current runtime model and only re-resolves on mismatch.
Or push these changes by commenting:
@cursor push 2b8d964ed8
Preview (2b8d964ed8)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -129,19 +129,30 @@
this.runtimeSpec = resolveCuaRuntimeSpec(model);
}
- tools(): AgentTool[] {
+ private buildTools(runtimeSpec: CuaRuntimeSpec): AgentTool[] {
return (
this.options.tools ??
createCuaComputerTools({
browser: this.options.browser,
client: this.options.client,
- toolDefinitions: this.runtimeSpec.toolDefinitions,
+ toolDefinitions: runtimeSpec.toolDefinitions,
})
);
}
+ tools(): AgentTool[] {
+ return this.buildTools(this.runtimeSpec);
+ }
+
+ toolsFor(runtimeSpec: CuaRuntimeSpec): AgentTool[] {
+ return this.buildTools(runtimeSpec);
+ }
+
onPayloadFor(model: CuaRuntimeInput): SimpleStreamOptions["onPayload"] {
- const runtimeSpec = resolveCuaRuntimeSpec(model);
+ const runtimeSpec =
+ typeof model === "string" || !isSameModel(model, this.runtimeSpec.model)
+ ? resolveCuaRuntimeSpec(model)
+ : this.runtimeSpec;
return composeOnPayload(runtimeSpec.onPayload, this.options.onPayload);
}
}
@@ -311,12 +322,19 @@
* concrete model selected by `@onkernel/cua-ai`.
*/
override async setModel(model: CuaRuntimeInput): Promise<void> {
- this.runtime.setModel(model);
+ const runtimeSpec = resolveCuaRuntimeSpec(model);
if (this.runtime.ownsTools) {
- const tools = this.runtime.tools();
- await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name));
+ const tools = this.runtime.toolsFor(runtimeSpec);
+ const activeToolNames = this.requestedActiveToolNames ?? tools.map((tool) => tool.name);
+ const toolNameSet = new Set(tools.map((tool) => tool.name));
+ const missingToolNames = activeToolNames.filter((toolName) => !toolNameSet.has(toolName));
+ if (missingToolNames.length > 0) {
+ throw new Error(`Unknown tool(s): ${missingToolNames.join(", ")}`);
+ }
+ await super.setTools(tools, activeToolNames);
}
- await super.setModel(this.runtime.model);
+ await super.setModel(runtimeSpec.model);
+ this.runtime.setModel(runtimeSpec.model);
}
override async setActiveTools(toolNames: string[]): Promise<void> {
@@ -333,3 +351,7 @@
return second(afterFirst ?? payload, modelRef);
};
}
+
+function isSameModel(left: Model<Api>, right: Model<Api>): boolean {
+ return left.api === right.api && left.provider === right.provider && left.id === right.id;
+}
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -231,4 +231,20 @@
expect(harness.agent.state.tools).toEqual([]);
});
+
+ it("keeps runtime spec unchanged if setModel fails validation", async () => {
+ const runtime = resolveCuaRuntimeSpec("openai:gpt-5.5");
+ const harness = new CuaAgentHarness({
+ ...(await createHarnessServices()),
+ browser,
+ client,
+ model: "openai:gpt-5.5",
+ });
+
+ (harness as unknown as { requestedActiveToolNames?: string[] }).requestedActiveToolNames = ["missing-tool"];
+
+ await expect(harness.setModel("google:gemini-3-pro-preview")).rejects.toThrow("Unknown tool(s): missing-tool");
+ expect(harness.agent.state.model.id).toBe(runtime.model.id);
+ expect((harness as unknown as { runtime: { model: { id: string } } }).runtime.model.id).toBe(runtime.model.id);
+ });
});You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 35399e8. Configure here.
| const tools = this.runtime.tools(); | ||
| await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name)); | ||
| } | ||
| await super.setModel(this.runtime.model); |
There was a problem hiding this comment.
Runtime spec update not rolled back on setModel failure
Low Severity
In CuaAgentHarness.setModel, this.runtime.setModel(model) eagerly updates the internal runtimeSpec before super.setTools and super.setModel are called. If super.setTools throws (e.g., validateToolNames fails because requestedActiveToolNames contains names absent from the new provider's tools), the runtimeSpec is left pointing to the new model while the harness model, session, and active tools remain unchanged. Subsequent reads of runtime.model, runtime.systemPrompt, or runtime.tools() would return values for the new model rather than the old one, creating an inconsistent internal state.
Reviewed by Cursor Bugbot for commit 35399e8. Configure here.
| return this.agent.subscribe(listener); | ||
| onPayloadFor(model: CuaRuntimeInput): SimpleStreamOptions["onPayload"] { | ||
| const runtimeSpec = resolveCuaRuntimeSpec(model); | ||
| return composeOnPayload(runtimeSpec.onPayload, this.options.onPayload); |
There was a problem hiding this comment.
onPayloadFor redundantly re-resolves the CUA runtime spec
Low Severity
CuaRuntimeController.onPayloadFor calls resolveCuaRuntimeSpec(model) on every invocation to build a fresh spec from the model parameter, completely ignoring the already-cached this.runtimeSpec. Every provider request triggers this redundant resolution. While the behavior is correct, onPayloadFor could read onPayload from this.runtimeSpec when the model matches, avoiding repeated work.
Reviewed by Cursor Bugbot for commit 35399e8. Configure here.
1 participant
Summary
CuaAgentHarnessextending piAgentHarness.@onkernel/cua-agentcan useAgentHarness,Session, andprepareNextTurnbefore those APIs are available from the published pi package.CuaAgenthandlesstate.modelchanges andCuaAgentHarnesshandlessetModel()changes, refreshing CUA-owned tools/system prompts and payload transforms.setActiveTools()selections acrossCuaAgentHarness.setModel()refreshes.envandsessionrequirements intact, add TSDoc for the public CUA agent types/classes, remove thegetTranscript()helper, and update docs/examples/tests to use session-backedAgentHarnessAPIs.hold_keys, and add shared example logging that prints tool args, result details, URL reads, and assistant text.Notes
packages/agent/CHANGELOG.mdis intentionally unchanged; release workflow should update it.earendil-works/pi@40c05f55391663024a6a05ad33249b616a04e7a1and includes the upstream MIT license in the package contents.Verification
npx tsx packages/agent/scripts/vendor-pi-agent-harness.tsnpm run build --workspace @onkernel/cua-agentnpm test --workspace @onkernel/cua-agentnpm run typechecknpm pack --dry-run --workspace @onkernel/cua-agentnpm run example:harness --workspace @onkernel/cua-agentNote
High Risk
High risk because it replaces the
@earendil-works/pi-agent-coredependency with a large vendored copy (including session/harness/compaction logic) and changes core agent/harness runtime behavior around model switching and tool/payload handling.Overview
Adds
CuaAgentHarnessby vendoring a pinned subset of pi agent core/harness source into@onkernel/cua-agent, removing the@earendil-works/pi-agent-coredependency and exporting pi types/APIs from the vendored entrypoint.Updates
CuaAgentand the new harness to be provider-aware when models change: assigningagent.state.modelor callingharness.setModel()now re-resolves the CUA runtime spec and refreshes CUA-owned defaults (tools, system prompt, payload transforms), while preserving caller-supplied tools/prompts and keeping active tool selections stable across refreshes.Updates README/examples to use session-backed harness APIs (adds
InMemorySessionRepo/NodeExecutionEnvusage), adds shared example logging helpers, includes vendored LICENSE/README in the package output, and fixes Kernel keypress batching by translating modifiers intohold_keys.Reviewed by Cursor Bugbot for commit 35399e8. Bugbot is set up for automated code reviews on this repo. Configure here.