docs(eval): align llm-rubric quickstart

.agents/conventions.md

@@ -136,10 +136,10 @@ Before adding a new pointer family, verify that the artifact is large enough or
 
 Grader types use kebab-case everywhere.
 
-- YAML config: `type: llm-rubric`, `type: llm-rubric`, `type: script`, `type: is-json`
-- Internal TypeScript: `EvaluatorKind = 'llm-grader' | 'is-json' | ...`
-- Output `scores[].type`: `"llm-grader"`, `"is-json"`
-- Registry keys: `registry.register('llm-grader', ...)`
+- Authored YAML config: `type: llm-rubric`, `type: script`, `type: is-json`
+- Internal TypeScript still has the shared LLM grader implementation and registry key (`'llm-grader'`), but new authored evals should use `llm-rubric` for semantic LLM grading.
+- Output `scores[].type`: use the authored grader type when available, such as `"llm-rubric"` or `"is-json"`.
+- Registry keys: `registry.register('llm-rubric', ...)` for the authored rubric surface, with `llm-grader` retained as internal/shared implementation plumbing.
 
 Source of truth: `GRADER_KIND_VALUES` in `packages/core/src/evaluation/types.ts`.
 

.agents/product-boundary.md

@@ -40,9 +40,8 @@ AgentV's core should remain minimal. Complex or domain-specific logic belongs in
 Prefer these extension points before adding a built-in:
 
 - `script` graders for custom executable evaluation logic
-- plain assertion strings or `llm-rubric` for structured rubric criteria
-- `llm-rubric` for promptfoo-compatible free-form rubric checks
-- `llm-grader` only when a custom prompt, custom grader target, or preprocessing is needed
+- plain assertion strings for simple semantic rubric checks
+- `llm-rubric` for promptfoo-compatible free-form rubrics, structured rubric criteria, custom prompts, custom grader targets, or preprocessing
 - CLI wrappers that consume AgentV JSON or JSONL output for post-processing such as aggregation, comparison, or reporting
 
 Ask: can this be achieved with existing primitives plus a plugin or wrapper? If yes, it should not be a built-in. That includes niche config overrides for existing graders.

README.md

@@ -35,43 +35,53 @@ agentv init
 
 ```yaml
 targets:
-  - label: copilot-sdk
-    provider: anthropic
-    model: claude-sonnet-4.6
+  - label: local-openai
+    provider: openai
+    api_format: chat
+    base_url: ${{ LOCAL_OPENAI_PROXY_BASE_URL }}
+    api_key: ${{ LOCAL_OPENAI_PROXY_API_KEY }}
+    model: ${{ LOCAL_OPENAI_PROXY_MODEL }}
 ```
 
-**3. Create an eval** in `evals/`:
+**3. Create shared test defaults** in `evals/default-test.yaml`. This is a promptfoo-style partial test config that AgentV applies to each test:
+
+```yaml
+threshold: 0.8
+options:
+  rubric_prompt: |
+    You are an expert grader. Evaluate the candidate answer against each rubric item.
+    Award credit only when the answer directly supports the criterion.
+
+    [[ ## question ## ]]
+    {{ input }}
+
+    [[ ## rubric ## ]]
+    {{ rubrics }}
+
+    [[ ## answer ## ]]
+    {{ output }}
+```
+
+**4. Create an eval** in `evals/my-eval.eval.yaml`:
 ```yaml
 description: Code generation quality
 tags:
   experiment: with-skills
-target: copilot-sdk
+target: local-openai
 evaluate_options:
-  repeat:
-    count: 3
-    strategy: pass_any
-    early_exit: false
-  max_concurrency: 3
-
-default_test:
-  threshold: 0.8
+  max_concurrency: 1
 
-workspace:
-  scope: attempt
-  repos:
-    - path: ./fixture
-      repo: EntityProcess/agentv-contract-fixture
-      commit: 21a34daed7ebcfe36cbed053607622a55e5e94cb
+default_test: file://./default-test.yaml
 
 tests:
   - id: fizzbuzz
-    input: Write FizzBuzz in Python
+    input: Write FizzBuzz in Python. Use lowercase output strings "fizz", "buzz", and "fizzbuzz". Return only one Python code block.
     assert:
       - type: contains
         value: "fizz"
       - Implements correct FizzBuzz logic for multiples of 3, 5, and 15
       - type: script
-        command: ["python3", "./validators/check_syntax.py"]
+        command: ["python3", "../validators/check_syntax.py"]
       - type: llm-rubric
         value:
           - outcome: Solution is simple and idiomatic Python
@@ -83,19 +93,19 @@ tests:
 Plain assertion strings are short-form rubric criteria: AgentV groups them into
 `llm-rubric` and writes each criterion to `grading.json.assertion_results` for the
 Dashboard. Use explicit `type: llm-rubric` when you need weights, required flags, or
-`score_ranges`; use string `value` for promptfoo-compatible free-form rubric
-checks; use `type: llm-grader` only when you need a custom grader prompt,
-grader target, or preprocessing. Executable graders use `type: script`.
+`score_ranges`, or when you need a custom grader prompt, grader target, or
+preprocessing; use string `value` for promptfoo-compatible free-form rubric
+checks. Executable graders use `type: script`.
 
 The target can be an eval-local object when this eval needs target settings of its own:
 
 ```yaml
-description: Code generation quality with Copilot target settings
+description: Code generation quality with eval-local target settings
 tags:
   experiment: with-skills
 target:
-  extends: copilot-sdk
-  model: claude-sonnet-4.6
+  extends: local-openai
+  model: gpt-5.4-mini
 evaluate_options:
   repeat:
     count: 2
@@ -109,7 +119,7 @@ tests:
     input: Write FizzBuzz in Python
 ```
 
-`target: copilot-sdk` resolves the target label from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `copilot-sdk`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The `tags.experiment` label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata.
+`target: local-openai` resolves the target label from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `local-openai`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The `tags.experiment` label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata.
 
 Use `default_test.threshold` for the inherited per-test pass cutoff. `default_test` can also point at a shared file, matching promptfoo's external defaults pattern:
 
@@ -126,22 +136,24 @@ refs:
 
 Then eval files in that project can use `default_test: ref://global-default`.
 
-**4. Run it:**
+The checked-in version of this quickstart lives in [`examples/features/readme-quickstart/`](examples/features/readme-quickstart/).
+
+**5. Run it:**
 ```bash
-agentv eval evals/my-eval.yaml
+agentv eval evals/my-eval.eval.yaml
 ```
 
-**5. Compare two runs** (pass two `index.jsonl` manifests — e.g. before and after a change):
+**6. Compare two runs** (pass two `index.jsonl` manifests — e.g. before and after a change):
 ```bash
-agentv compare .agentv/results/<baseline-run-id>/index.jsonl .agentv/results/<candidate-run-id>/index.jsonl
+agentv results compare .agentv/results/<baseline-run-id>/index.jsonl .agentv/results/<candidate-run-id>/index.jsonl
 ```
 
 ## Results
 
-Each run writes a portable bundle directly under `.agentv/results/<run_id>/`. In this example, `tags.experiment: with-skills` names the condition being measured and `target: copilot-sdk` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv compare`; per-case sidecars include the resolved eval and target configuration used for the run.
+Each run writes a portable bundle directly under `.agentv/results/<run_id>/`. In this example, `tags.experiment: with-skills` names the condition being measured and `target: local-openai` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv results compare`; per-case sidecars include the resolved eval and target configuration used for the run.
 
 ```bash
-agentv eval evals/my-eval.yaml
+agentv eval evals/my-eval.eval.yaml
 cat .agentv/results/<run_id>/index.jsonl
 ```
 
@@ -150,7 +162,7 @@ Run bundle layout:
 ```
 .agentv/results/
 ├── 2026-06-30T08-30-00-000Z/     # <run_id> — one committed run bundle
-│   ├── index.jsonl               # row index for scripts/CI and `agentv compare`
+│   ├── index.jsonl               # row index for scripts/CI and `agentv results compare`
 │   ├── summary.json              # run rollup: metadata, pass rate, counts, cost
 │   └── fizzbuzz--a1b2c3d4/       # <result_dir> for one test/target row
 │       ├── summary.json          # optional per-case rollup across attempts

apps/cli/src/commands/results/index.ts

@@ -1,5 +1,7 @@
 import { subcommands } from 'cmd-ts';
 
+import { compareCommand } from '../compare/index.js';
+import { trendCommand } from '../trend/index.js';
 import { resultsCombineCommand } from './combine.js';
 import { resultsDeleteCommand } from './delete.js';
 import { resultsExportCommand } from './export.js';
@@ -14,12 +16,14 @@ export const resultsCommand = subcommands({
   description: 'Inspect, export, and manage local evaluation results',
   cmds: {
     combine: resultsCombineCommand,
+    compare: compareCommand,
     delete: resultsDeleteCommand,
     export: resultsExportCommand,
     report: resultsReportCommand,
     summary: resultsSummaryCommand,
     failures: resultsFailuresCommand,
     show: resultsShowCommand,
+    trend: trendCommand,
     validate: resultsValidateCommand,
   },
 });

apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx

@@ -83,7 +83,7 @@ agent should..." criteria list. `expected_output` is passive by default: it is
 stored on the case and passed to graders, but it does not choose a grader by
 itself. A grader may treat it as a strict target, semantic reference, structured
 object, or supporting context depending on the grader type. Add explicit
-assertion strings, `llm-grader`, `script`, `field-accuracy`, or another
+assertion strings, `llm-rubric`, `script`, `field-accuracy`, or another
 reference-aware grader when you want the reference data evaluated.
 
 A string expands to a single assistant message:
@@ -114,7 +114,7 @@ tests:
     assert:
       - Provides a detailed explanation
       - name: depth_check
-        type: llm-grader
+        type: llm-rubric
         prompt: ./graders/depth.md
 ```
 
@@ -140,7 +140,7 @@ tests:
     assert:
       - Handles the edge case
       - name: custom_eval
-        type: llm-grader
+        type: llm-rubric
     # Does NOT get latency_check
 ```
 
@@ -233,7 +233,7 @@ tests:
 Use this shape for qualitative requirements. It is less brittle than checking
 for exact substrings in an agent response. When these strings fully define the
 grading contract, do not add a `criteria` field that repeats the same rubric.
-Declare `type: llm-grader` explicitly only when you need a custom prompt, custom
+Declare `type: llm-rubric` explicitly only when you need a custom prompt, custom
 grader target, or a deliberately separate grader panel.
 
 ### Deterministic Assertions
@@ -420,7 +420,7 @@ tests:
 
 When `assert` is defined, only the declared graders run. No implicit grader is
 added because `expected_output` exists. Declared graders such as plain rubric
-strings, `llm-grader`, `script`, or `llm-rubric` receive the case context, including
+strings, explicit `llm-rubric` entries, or `script` graders receive the case context, including
 `expected_output`, as input automatically.
 
 This means a case with `expected_output` and only deterministic assertions evaluates only
@@ -477,7 +477,7 @@ tests:
     input: "Debug this function..."
     assert:
       - Response is helpful and mentions the fix
-      - type: llm-grader        # use explicit form for custom preprocessors
+      - type: llm-rubric        # use explicit form for custom preprocessors
         preprocessors:
           - type: xlsx
             command: ["bun", "run", "scripts/preprocessors/xlsx-to-json.ts"]

apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx

@@ -264,7 +264,7 @@ For semantic or agent-behavior checks, prefer plain assertion strings first;
 AgentV treats them as rubric criteria. Use deterministic assertions or script
 graders when the expected output is exact or requires programmatic inspection.
 If the assertion strings already state the grading contract, omit a duplicate
-`criteria` field on each test. Use explicit `type: llm-grader` entries only
+`criteria` field on each test. Use explicit `type: llm-rubric` entries only
 when you need a custom prompt, a custom grader target, or a deliberately
 separate grader panel.
 
@@ -284,7 +284,7 @@ tests:
 ```
 
 `assert` supports rubric shorthand strings, deterministic assertion types
-(`contains`, `regex`, `is-json`, `equals`), `llm-rubric`, LLM graders, and script
+(`contains`, `regex`, `is-json`, `equals`), `llm-rubric`, and script
 graders. See [Tests](/docs/evaluation/eval-cases/#per-test-assert) for
 per-test assert usage.
 
@@ -599,7 +599,7 @@ suite: math-tests
 target: azure-base
 assert:
   - name: correctness
-    type: llm-grader
+    type: llm-rubric
     prompt: ./graders/correctness.md
 ```
 

apps/web/src/content/docs/docs/next/evaluation/examples.mdx

@@ -85,7 +85,7 @@ tests:
         command: [uv, run, validate_json.py]
         cwd: ./graders
       - name: content_evaluator
-        type: llm-grader
+        type: llm-rubric
         prompt: ./graders/semantic_correctness.md
 
     input: |-
@@ -102,7 +102,7 @@ tests:
 
 ## File Output Preprocessing
 
-Convert a binary file output into text before the `llm-grader` sees it:
+Convert a binary file output into text before the `llm-rubric` sees it:
 
 ```yaml
 description: Grade spreadsheet output via a preprocessor
@@ -176,15 +176,15 @@ assert:
       threshold: 0.6
     assert:
       - name: grader-gpt-5-mini
-        type: llm-grader
+        type: llm-rubric
         target: grader_gpt_5_mini
         prompt: ../prompts/grader-pass-fail-v1.md
       - name: grader-claude-haiku
-        type: llm-grader
+        type: llm-rubric
         target: grader_claude_haiku
         prompt: ../prompts/grader-pass-fail-v1.md
       - name: grader-gemini-flash
-        type: llm-grader
+        type: llm-rubric
         target: grader_gemini_flash
         prompt: ../prompts/grader-pass-fail-v1.md
 ```
@@ -413,5 +413,5 @@ See the [suite-level-input example](https://github.com/EntityProcess/agentv/tree
 
 For complete end-to-end workflows that combine multiple features, see the showcases in [`examples/showcase/`](https://github.com/EntityProcess/agentv/tree/main/examples/showcase):
 
-- **[Multi-Model Benchmark](https://github.com/EntityProcess/agentv/tree/main/examples/showcase/multi-model-benchmark)** — weighted metrics × repeated runs × compare workflow. Run the same eval once per target, then use `agentv compare` or Dashboard analytics to review the completed runs side-by-side.
+- **[Multi-Model Benchmark](https://github.com/EntityProcess/agentv/tree/main/examples/showcase/multi-model-benchmark)** — weighted metrics × repeated runs × compare workflow. Run the same eval once per target, then use `agentv results compare` or Dashboard analytics to review the completed runs side-by-side.
 - **[Export Screening](https://github.com/EntityProcess/agentv/tree/main/examples/showcase/export-screening)** — classification eval with confusion matrix metrics and CI gating.

apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx

@@ -27,7 +27,7 @@ Each `scores[]` entry includes per-grader timing:
   "scores": [
     {
       "name": "format_structure",
-      "type": "llm-grader",
+      "type": "llm-rubric",
       "score": 0.9,
       "verdict": "pass",
       "assertions": [

apps/web/src/content/docs/docs/next/evaluation/sdk.mdx

@@ -59,7 +59,7 @@ from agentv_py.evals import EvalDefinition, JsonlCase, write_eval_yaml, write_js
 def rag_faithfulness():
     return {
         "name": "rag-faithfulness",
-        "type": "llm-grader",
+        "type": "llm-rubric",
         "target": "grader-target",
         "prompt": "Grade whether the answer is supported by the retrieved context.",
     }
@@ -141,7 +141,7 @@ export default defineEval({
         graders.regex(/"message"\s*:/, { name: 'message-key' }),
         graders.json({ name: 'valid-json', required: true }),
         graders.llmRubric(['Greets the user'], { name: 'rubric-review' }),
-        graders.llmGrader({
+        graders.llmRubric(undefined, {
           name: 'llm-review',
           prompt: 'Grade whether the answer is useful.',
           target: 'grader-target',
@@ -153,7 +153,7 @@ export default defineEval({
 });
 ```
 
-The catalog covers `contains`, `equals`/`exact`, `regex`, `is-json`/`json`, `llm-rubric`, `llm-grader`, and `script`. CamelCase SDK options such as `minScore`, `maxSteps`, and rubric `scoreRanges` lower to `min_score`, `max_steps`, and `score_ranges` when AgentV loads or serializes the suite.
+The catalog covers `contains`, `equals`/`exact`, `regex`, `is-json`/`json`, `llm-rubric`, and `script`. CamelCase SDK options such as `minScore`, `maxSteps`, and rubric `scoreRanges` lower to `min_score`, `max_steps`, and `score_ranges` when AgentV loads or serializes the suite.
 
 ## AgentV-Native Helper Factories
 
@@ -163,7 +163,7 @@ If you are coming from Braintrust `scores` or DeepEval metrics, keep the reusabl
 import { defineEval, graders } from '@agentv/sdk';
 
 function ragFaithfulness() {
-  return graders.llmGrader({
+  return graders.llmRubric(undefined, {
     name: 'rag-faithfulness',
     target: 'grader-target',
     prompt: [
@@ -197,7 +197,7 @@ assert:
     type: contains
     value: source
   - name: rag-faithfulness
-    type: llm-grader
+    type: llm-rubric
     target: grader-target
     prompt: |-
       Grade whether the answer is supported by the retrieved context.