Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions apps/web/src/content/docs/docs/targets/coding-agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -148,14 +148,31 @@ targets:
targets:
- name: pi_target
provider: pi-coding-agent
subprovider: openai-codex
model: gpt-5.5
thinking: medium
grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `subprovider` | No | Pi provider to use, such as `google`, `openai`, `openai-codex`, `azure`, `anthropic`, or `openrouter`. Defaults to Pi's default provider. |
| `model` | No | Model to use. For OpenAI subscription auth through Pi, use `subprovider: openai-codex` with a subscription model such as `gpt-5.5`. |
| `thinking` | No | Pi reasoning level: `off`, `minimal`, `low`, `medium`, `high`, or `xhigh`. Passed to the Pi SDK as `thinkingLevel`. |
| `tools` | No | Comma-separated Pi tool allowlist, such as `read,bash,edit,write`. |
| `api_key` | No | Provider API key. Prefer `${{ ENV_VAR }}` references. Omit for subscription auth handled by Pi. |
| `base_url` | No | Provider base URL or Azure resource URL/name. |
| `cwd` | No | Working directory |
| `timeout_seconds` | No | Per-case timeout |
| `grader_target` | Yes | LLM target for evaluation |

Use `provider: pi-cli` instead when you want AgentV to spawn the `pi` binary directly. It accepts the same Pi fields above plus:

| Field | Required | Description |
|-------|----------|-------------|
| `executable` | No | Pi binary or shim to run. Defaults to `pi`. |
| `args` | No | Extra arguments appended before the prompt. |

## VS Code

```yaml
Expand Down
36 changes: 36 additions & 0 deletions packages/core/test/evaluation/providers/targets.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1243,6 +1243,24 @@ describe('createProvider', () => {
expect(resolved.config.tools).toBe('read,bash,edit,write');
});

it('resolves pi-coding-agent thinking level from target config', () => {
const resolved = resolveTargetDefinition(
{
name: 'pi-openai-codex',
provider: 'pi-coding-agent',
subprovider: 'openai-codex',
model: 'gpt-5.5',
thinking: 'medium',
},
{},
);

expect(resolved.kind).toBe('pi-coding-agent');
if (resolved.kind !== 'pi-coding-agent') throw new Error('expected pi-coding-agent');
expect(resolved.config.model).toBe('gpt-5.5');
expect(resolved.config.thinking).toBe('medium');
});

it('resolves pi-cli with azure subprovider and base_url', () => {
const env = {
AZURE_OPENAI_ENDPOINT: 'https://my-resource.openai.azure.com',
Expand All @@ -1269,4 +1287,22 @@ describe('createProvider', () => {
expect(resolved.config.model).toBe('gpt-4o');
expect(resolved.config.apiKey).toBe('azure-secret');
});

it('resolves pi-cli thinking level from env-backed config', () => {
const resolved = resolveTargetDefinition(
{
name: 'pi-cli-openai-codex',
provider: 'pi-cli',
subprovider: 'openai-codex',
model: 'gpt-5.5',
thinking: '${{ PI_THINKING }}',
},
{ PI_THINKING: 'medium' },
);

expect(resolved.kind).toBe('pi-cli');
if (resolved.kind !== 'pi-cli') throw new Error('expected pi-cli');
expect(resolved.config.model).toBe('gpt-5.5');
expect(resolved.config.thinking).toBe('medium');
});
});
Loading