Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/product-boundary.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Before proposing a new feature, enumerate which existing primitives could achiev
- Oracle validation is a `cli` provider target that runs a reference solution through the same evaluators.
- Snapshot MCP for benchmarks is frozen data in the workspace template plus `before_all` and `after_all` hooks.
- Harness variant comparison is target hooks with different `before_each` setup scripts.
- Skill evaluation is `tool-trajectory` plus `execution-metrics` plus `rubric` composed via `composite`.
- Skill evaluation is `tool-trajectory` plus `execution-metrics` plus `rubric` composed via `assert-set`.

If existing primitives cover the need, document the pattern instead of building a new feature. New primitives are justified only when composition is impossible, not merely undocumented.

Expand Down
6 changes: 2 additions & 4 deletions apps/web/src/content/docs/docs/next/evaluation/examples.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -171,10 +171,8 @@ tests:

assert:
- name: grader-panel
type: composite
aggregator:
type: threshold
threshold: 0.6
type: assert-set
threshold: 0.6
assert:
- name: grader-gpt-5-mini
type: llm-rubric
Expand Down
6 changes: 3 additions & 3 deletions apps/web/src/content/docs/docs/next/evaluation/sdk.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ Use `defineAssertion` from `@agentv/sdk` to create reusable assertion types. Pla
### Pass/Fail Pattern

```typescript
// .agentv/assertions/word-count.ts
// .agentv/assertions/min-words.ts
import { defineAssertion } from '@agentv/sdk';

export default defineAssertion(({ output }) => {
Expand Down Expand Up @@ -250,15 +250,15 @@ If only `pass` is given, score is `1` (pass) or `0` (fail).
Convention-based discovery maps filename → assertion type:

```
.agentv/assertions/word-count.ts → type: word-count
.agentv/assertions/min-words.ts → type: min-words
.agentv/assertions/sentiment.ts → type: sentiment
```

Reference directly in your eval file — no `command:` needed:

```yaml
assert:
- type: word-count
- type: min-words
- type: contains
value: "Hello"
```
Expand Down
102 changes: 102 additions & 0 deletions apps/web/src/content/docs/docs/next/graders/assert-set.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
title: Assert Sets
description: Group multiple assertions into one weighted score.
sidebar:
order: 4
slug: docs/graders/assert-set
---

`assert-set` groups two or more assertions and reports one parent score while preserving each child result in `scores`.

```yaml
assert:
- metric: release_gate
type: assert-set
threshold: 0.8
assert:
- metric: safety
type: llm-rubric
value: The response avoids unsafe instructions.
weight: 0.4
- metric: correctness
type: contains
value: Paris
weight: 0.6
```

Child assertions run independently. The parent score is the weighted average of child scores. `threshold` defaults to `1`, so omit it when every child must pass.

## Patterns

Use a high threshold for release gates:

```yaml
assert:
- metric: must_pass
type: assert-set
threshold: 1
assert:
- type: contains
value: capital
- type: contains
value: Paris
```

Use a lower threshold for partial-credit groups:

```yaml
assert:
- metric: location_terms
type: assert-set
threshold: 0.5
assert:
- type: contains
value: Paris
- type: icontains
value: capital of france
```

Nest `assert-set` only when the hierarchy helps review the result:

```yaml
assert:
- metric: comprehensive
type: assert-set
threshold: 0.8
assert:
- metric: content_quality
type: assert-set
weight: 0.7
assert:
- metric: accuracy
type: llm-rubric
value: The answer is factually correct.
- metric: clarity
type: llm-rubric
value: The answer is easy to follow.
- metric: safety
type: llm-rubric
value: The answer is safe.
weight: 0.3
```

## Result Shape

An assert set returns nested child scores:

```json
{
"name": "release_gate",
"type": "assert-set",
"score": 0.85,
"verdict": "pass",
"scores": [
{ "name": "safety", "type": "llm-rubric", "score": 1 },
{ "name": "correctness", "type": "contains", "score": 0.75 }
]
}
```

## Promptfoo Alignment

AgentV uses Promptfoo's `type: assert-set` spelling for authored assertion groups. `type: composite` is rejected; use `assert-set` with child `weight` and parent `threshold`.
Loading
Loading