diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index d03cef0157..9d31576eba 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -37,6 +37,15 @@ /plugins/dotnet-ai/skills/mcp-csharp-test/ @leslierichardson95 @cathysull /tests/dotnet-ai/mcp-csharp-test/ @leslierichardson95 @cathysull +/plugins/dotnet-ai/skills/configure-agentic-perf-rules/ @leslierichardson95 @cathysull +/tests/dotnet-ai/configure-agentic-perf-rules/ @leslierichardson95 @cathysull + +/plugins/dotnet-ai/skills/scan-agentic-app-perf/ @leslierichardson95 @cathysull +/tests/dotnet-ai/scan-agentic-app-perf/ @leslierichardson95 @cathysull + +/plugins/dotnet-ai/skills/setup-maf-evals/ @leslierichardson95 @cathysull +/tests/dotnet-ai/setup-maf-evals/ @leslierichardson95 @cathysull + # dotnet-upgrade (migrating and upgrading .NET projects) /plugins/dotnet-upgrade/skills/thread-abort-migration/ @dotnet/appmodel @dotnet/skills-upgrade-reviewers /tests/dotnet-upgrade/thread-abort-migration/ @dotnet/appmodel @dotnet/skills-upgrade-reviewers diff --git a/eng/known-domains.txt b/eng/known-domains.txt index c6f32825a4..f77ac51e2c 100644 --- a/eng/known-domains.txt +++ b/eng/known-domains.txt @@ -31,12 +31,14 @@ developer.android.com developer.apple.com # Repos +github.com/dotnet/ai-samples github.com/dotnet/aspnetcore github.com/dotnet/csharplang github.com/dotnet/diagnostics github.com/dotnet/dotnet github.com/dotnet/dotnet-docker github.com/dotnet/efcore +github.com/dotnet/extensions github.com/dotnet/maui github.com/dotnet/msbuild github.com/dotnet/roslyn diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md new file mode 100644 index 0000000000..af0a7f0fb2 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md @@ -0,0 +1,220 @@ +--- +name: configure-agentic-perf-rules +version: 0.3.0 +description: "Installs or updates an always-on agentic-perf rules block in a .NET Microsoft Agent Framework (Microsoft.Agents.AI) app so coding agents volunteer perf and cost concerns by default: agent count, handoff edges, per-agent model choice, message-history strategy, per-turn token cost, and post-change measurement. Written into the project's agent-instructions file (.github/copilot-instructions.md by default) inside a sentinel-delimited, version-aware managed block. WHEN: adding or reviewing agents, handoffs, tools, or models in a Microsoft.Agents.AI project; scaffolding a new MAF app (Aspire, console, ASP.NET Core, or worker); or the user says Copilot misses perf/cost issues or wants up-front agentic guard-rails. NOT-WHEN: non-agentic .NET (use optimizing-dotnet-performance), non-.NET agentic projects, auditing existing code now (use scan-agentic-app-perf), or measuring runtime telemetry (use setup-maf-evals)." +license: MIT +--- + +# Configure Agentic Perf Rules + +This skill writes a managed block of always-on instructions into the target project's +agent-instructions file so coding agents (Copilot, etc.) volunteer agentic-perf concerns +during normal work, instead of waiting to be asked. The block is delimited by sentinel +HTML comments and embeds the skill version so future runs can update it cleanly without +clobbering user-edited threshold values. + +## When to Use + +- A .NET project uses Microsoft Agent Framework (`Microsoft.Agents.AI`) — with or + without Aspire/Foundry — and the user wants default-on perf guidance. +- Scaffolding a new MAF agentic .NET app (Aspire-hosted or plain console / ASP.NET + Core / worker service) and the user wants to start with perf guard-rails in place. +- The user reports that the coding agent is not catching perf issues until prompted. +- The user wants to update an existing managed block to a newer version of the rules. + +## When Not to Use + +- The project is not a .NET agentic app — use `optimizing-dotnet-performance` for general + .NET performance guidance. +- The user wants the agent to actually audit existing code right now — use + `scan-agentic-app-perf` instead. This skill only installs guidance. +- The user wants to measure tokens, latency, or quality scores — use `setup-maf-evals`. +- Generic prompt-engineering or non-perf coding-agent rules (keep those in the user's own + instructions section, outside the managed block). + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Target project root | Yes | Repository root containing the .NET solution | +| Custom thresholds | No | Per-project override values for agent count, handoff edges, token warn levels | + +## Workflow + +> **Outcome:** the project's agent-instructions file contains an up-to-date managed +> rules block. Re-running this skill is safe and idempotent. + +### Step 1: Locate the target instructions file + +Look for `.github/copilot-instructions.md` (preferred) and `AGENTS.md` at the repo root, +and scan whichever exist for an existing managed block (Step 2) so you never create a +duplicate. + +- **Neither exists** — create `.github/copilot-instructions.md` (create the `.github/` + directory if needed). +- **Only one exists** — use it. +- **Both exist** — put the managed block in `.github/copilot-instructions.md` and write a + one-line stub in `AGENTS.md` pointing to it. If the existing block currently lives in + `AGENTS.md`, move it to `.github/copilot-instructions.md` and replace it with the stub. + +### Step 2: Detect any existing managed block + +A managed block is delimited by sentinel HTML comments: + +```markdown + +... + +``` + +Scan the target file. If a managed block is present, parse its version from the +`BEGIN` comment. + +**Sentinel parse rules** (apply in order; fail closed if any rule trips): + +1. The full-line BEGIN regex is `^\s*$`. +2. The full-line END regex is `^\s*$`. +3. The file must contain **exactly one** matching BEGIN and **exactly one** matching END, + with BEGIN appearing before END. +4. If zero, multiple, mismatched, or out-of-order sentinels are detected, **refuse to + edit** and report the malformed state to the user. Do not attempt to repair the file + automatically. +5. Versions are compared numerically as semver triples. The `v` prefix is a + literal part of the BEGIN sentinel (required by rule #1's regex), so the + captured `` group never includes it; the skill's `version:` field + likewise carries no `v`. Compare the two bare triples directly. + +The current skill version is the `version:` field at the top of this SKILL.md. + +| Detected state | Action | +|----------------|--------| +| No block present | Append a new managed block at the end of the file | +| Block present, same version | **No-op — report "already current (vX.Y.Z)" and stop.** A matching version marker is the idempotency source of truth: do not re-validate or rewrite the body. If rule sections look missing, elided, or otherwise customized, note it informationally but leave the block untouched — the user may have intentionally trimmed or edited it. Restore default sections only when the user explicitly asks (e.g. "reinstall" / "restore defaults"). | +| Block present, older version | Replace the block in place, preserving any user-edited threshold values from the existing frontmatter (see Threshold preservation). | +| Block present, newer version than this skill | Refuse to downgrade — report version mismatch and stop | + +**Idempotency is keyed on the version marker, not body equality.** Once the +sentinels are well-formed (Step 2 rules #1–#4) and the captured version equals +this skill's `version:`, the run is a no-op regardless of the body's contents. +The skill never repairs or rewrites a same-version block on its own — doing so +would clobber legitimate user customizations (trimmed sections, hand-tuned +wording). Structural validation only gates the freshly *written* block on +install/update runs (see Validation), never a same-version detection. + +**Threshold preservation algorithm** (used in the older-version path): + +1. Parse the existing managed block's `thresholds:` YAML map into a `prev_user` dict. + If parsing fails, refuse to edit and ask the user to repair the YAML manually. +2. Construct the new defaults map `new_defaults` from `references/threshold-defaults.md`. +3. For each known key in `new_defaults`, override with the value from `prev_user` if + present and the value passes type validation (e.g. integer for `per_turn_input_token_warn`). +4. Drop unknown keys from `prev_user` with a chat warning naming each dropped key. +5. The merged map becomes the new managed block's `thresholds:` content. + +**Path safety:** before any write, resolve the project root and the target path. +Refuse to write to any path outside the project root (after normalization, including +following symlinks). Reject absolute paths and paths containing `..` segments unless +they normalize back inside the project root. + +### Step 3: Render the managed block + +The managed block has three parts in this exact order: + +1. **Sentinel BEGIN comment** with the skill version. +2. **Threshold frontmatter** (a fenced YAML code block) so users can override numeric + defaults without editing prose. The default values are in + `references/threshold-defaults.md`. +3. **Six rule sections**, one per category, in the order listed in the next step. Each + section is short — a "before X, justify Y" lead sentence, the default threshold, and + one or two crisp expectations. Long-form rationale lives in the reference docs and is + not duplicated in the project's instructions file. + +See `references/managed-block-template.md` for the exact rendered output, including the +threshold frontmatter format and section ordering. + +### Step 4: Write the six rule categories + +Each rule is in the form **"Before X, justify Y."** Categories, in order: + +1. **Agent count.** Before adding a new agent to a workflow, justify why the new + responsibility cannot be a tool call on an existing agent. No hard ceiling — + the rule is "every agent earns its keep"; see `references/rule-rationales.md`. +2. **Handoff edges.** Before adding an LLM-routed handoff edge, justify why a + deterministic edge or a conditional `WorkflowBuilder` branch will not work. No + hard ceiling on edges per turn; each LLM-routed hop has to pay for itself. +3. **Model selection.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the + agent's role and pick from the role table inside rule #3 of the managed block. + Routers/validators/formatters/workers → small-fast; planners → reasoning-class. +4. **Message-history strategy.** Before sending the full conversation history to an + agent, state the bound — turn count, token cap, summarization point, or retrieval + strategy. Default warning when unbounded full-history is used in a multi-turn workflow. +5. **Token / cost surfacing.** Before implementing a non-trivial change to an agent's + prompt, tools, or model, estimate per-turn token cost. Default warnings: more than + 8000 input tokens or more than 2000 output tokens projected per turn, or any change + that adds more than 20% to a measured baseline. +6. **Post-change measurement.** After a non-trivial change to a workflow, propose + running `setup-maf-evals` (or an existing `.Evals` project) to confirm the change is + net-positive — or explicitly note why measurement is not warranted. + +Long-form rationale, examples, and counter-examples for each rule live in +`references/rule-rationales.md`. + +### Step 5: Update the cross-tool stub (if applicable) + +If `AGENTS.md` exists alongside `.github/copilot-instructions.md`, ensure `AGENTS.md` +contains (or has appended) a single line: + +```markdown +> Agentic-perf rules for this project live in `.github/copilot-instructions.md` (managed by `configure-agentic-perf-rules`). +``` + +This avoids duplicating the rules across files while keeping cross-tool agents pointed +at the right source. + +## Validation + +After the skill runs, the agent must verify: + +1. **File exists** at the resolved target path. +2. **Sentinel comments present** with matching `BEGIN`/`END` markers and a parseable + version in the `BEGIN` comment. +3. **Threshold frontmatter parses** as valid YAML (no syntax errors introduced). + *(Applies only when the skill wrote or updated the block this run.)* +4. **All six rule sections** are present, in the canonical order. + *(Applies only when the skill wrote or updated the block this run; a detected + same-version no-op leaves a customized or elided block as-is and is still valid.)* +5. **Round-trip:** re-running the skill against the now-updated file is a no-op (reports + "already current"). If a second run produces any modification, the install was not + idempotent and the skill failed. + +If `AGENTS.md` was updated, also confirm the stub line is present exactly once. + +## Common Pitfalls + +- **Managing user-authored content.** Never modify content outside the sentinel block. + All edits stay strictly between `BEGIN` and `END` markers. +- **Re-validating a same-version block.** Idempotency keys on the version marker, not + byte-for-byte body equality. If a re-run finds a block whose version already matches + this skill, report "already current" and stop — never flag elided or user-customized + sections as "needs repair" or silently rewrite them. Restoring default sections is + opt-in (the user explicitly asks to reinstall / restore defaults). +- **Threshold preservation on update.** When updating to a newer version of the rules, + preserve any user-edited values in the threshold frontmatter rather than resetting to + defaults. Diff against the old defaults to detect user edits. +- **Version downgrades.** If the file has a newer skill version than the running skill, + refuse to overwrite. Tell the user to update the skill before re-running. +- **Sentinel collision.** If a different tool has authored a similar-looking managed + block (different `BEGIN` text), do not assume it is ours. Match on the exact sentinel + string `BEGIN: managed by configure-agentic-perf-rules`. +- **Don't re-author rules into the prose.** Keep the SKILL.md body and the project's + instructions file in sync via `references/managed-block-template.md` — do not paste + rule prose directly into multiple places. + +## References + +- `references/managed-block-template.md` — the exact rendered template with sentinels + and frontmatter. +- `references/threshold-defaults.md` — default numeric values and the rationale for each. +- `references/rule-rationales.md` — long-form prose for each of the six rule categories, + with examples and counter-examples. +- Companion skills: `scan-agentic-app-perf`, `setup-maf-evals`. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md new file mode 100644 index 0000000000..b892442cb2 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md @@ -0,0 +1,100 @@ +# Common pitfalls + +Real-world failure modes for `configure-agentic-perf-rules` — observed +during dogfooding (interview-coach v2 first install, behavioral coach +no-op re-run, ELI5Agent fresh install). + +## Sentinel parsing (FAIL CLOSED, always) + +- **Lenient sentinel matching.** The BEGIN and END regexes in step 2 + are intentionally strict (anchored, full-line, exact version + pattern). Do not "fix up" a malformed sentinel by partial-matching + or by accepting trailing whitespace beyond what `\s*$` allows. + Auto-repair of corrupted sentinels is forbidden — the user might + have intentionally renamed the block while migrating, and silent + repair would clobber that. +- **Refusing to edit on multiple BEGIN/END.** If the file contains + more than one BEGIN or END, abort with a chat message that names + every offending line number. Multiple managed blocks usually mean + a merge conflict went unresolved; appending a third would make it + worse. +- **Out-of-order sentinels.** BEGIN must appear before its matching + END. If you find END before BEGIN, treat as malformed and abort — + do not swap them. + +## Threshold preservation + +- **Losing user-edited threshold values on update.** Step 2's + "Threshold preservation algorithm" is the contract: parse the + existing `thresholds:` map into `prev_user`, overlay the new + default map, override each known key from `prev_user`. Skipping + this step and writing the new defaults verbatim is the most + user-visible regression this skill can ship. +- **Silently dropping unknown keys.** If `prev_user` has a key not + in `new_defaults` (e.g. a deprecated threshold), drop it AND emit + a chat warning naming the dropped key. Silent drops break audit + trails — the user needs to know their override no longer applies. +- **Type-validating with `Convert.ToInt32` instead of strict parse.** + `per_turn_input_token_warn: "eight thousand"` should fail validation, not coerce to + some default. Use strict numeric parsing; on failure, keep the + default and warn. + +## Target-file selection + +- **Writing to `AGENTS.md` when `.github/copilot-instructions.md` + exists.** The order in step 2's target-file table is the spec: + `.github/copilot-instructions.md` is the primary destination; + `AGENTS.md` gets a stub pointer. The reverse only happens during + the explicit migration path (existing block in AGENTS.md, none in + copilot-instructions.md). +- **Writing to both files.** "Both have a block" is the abort path — + the user must consolidate manually. Writing the rule prose into + two files would split the source of truth and let the two copies + drift on the next update. +- **Creating `.github/` outside the project root.** If neither file + exists, create `.github/copilot-instructions.md` under the resolved + project root (the directory containing the `.sln`/`.slnx`/`*.AppHost.csproj`). + Never write to a parent directory or a sibling project. + +## Path safety + +- **Following symlinks out of the project root.** Step 2's "Path + safety" rule requires resolving the absolute path AND ensuring it + still starts with the project-root prefix after symlink resolution. + Some CI environments place repos under symlinks; a naive resolve + can end up writing to the symlink target outside the workspace. +- **Accepting `..` in paths.** Reject any path containing `..` + segments before normalization, unless the post-normalization path + is still inside the project root. The simplest safe check: + `Path.GetFullPath(target).StartsWith(Path.GetFullPath(projectRoot))`. + +## Cross-tool stub on AGENTS.md + +- **Re-adding the stub on every run.** The stub is one line: + `> Agentic-perf rules for this project live in .github/copilot-instructions.md (managed by configure-agentic-perf-rules).` + Check whether that exact line is already present before appending; + re-running the skill should not grow the file by one line each time. +- **Replacing user prose in AGENTS.md with the stub.** AGENTS.md + often contains real onboarding prose the user wrote. Append the + stub at the bottom if missing; never overwrite existing content. + +## Version handling + +- **Refusing to downgrade is correct.** If the file has a newer + version than this skill, abort cleanly with a chat message naming + both versions. Do not "merge" or "convert" — that's how data loss + happens. +- **Comparing versions as strings.** "v0.1.10" sorts before "v0.1.2" + lexically. Parse into semver triples and compare numerically. + +## Idempotency + +- **No-op path must actually be a no-op.** When the block is present, + same version, and structurally valid, the skill must not touch the + file at all — not even to rewrite identical content. Tooling and + git status both rely on "no change on second run". Verify by + comparing the SHA256 hash before/after; they should match exactly. +- **Tracking "already installed" silently.** Even on a no-op, the + chat output should say "configure-agentic-perf-rules v0.1.0 block + already current — no changes". Quiet no-ops make the user think + the skill didn't run. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md new file mode 100644 index 0000000000..fe69aa65f2 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md @@ -0,0 +1,103 @@ +# Managed Block Template + +This is the exact content the skill writes into the target instructions file. Variables +in `` are filled in at install time. The outer fence below is **four +backticks** so the inner three-backtick fences in the rendered Markdown are not +interpreted as closing it. + +````markdown + + +## Agentic Performance Rules + +> These rules are managed by the `configure-agentic-perf-rules` skill. Do not edit prose +> inside this managed block — your changes will be overwritten on the next update. +> Numeric defaults can be overridden in the `thresholds` block below; user-edited values +> are preserved across updates. + +```yaml +# thresholds — edit values below to override per-project defaults +thresholds: + per_turn_input_token_warn: 8000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true +``` + +When working in this codebase, apply each rule below by default. Each rule is in the +form **"Before X, justify Y."** When you cannot justify, prefer the safer alternative +(do not add the agent, do not add the edge, etc.) and surface the trade-off to the user. + +### 1. Agent count + +Before adding a new agent to a workflow, justify why the new responsibility cannot be a +tool call on an existing agent. Each additional agent multiplies the routing surface and +inflates per-turn token cost (system prompts + tool descriptions are paid per agent). +**There is no hard ceiling** — the answer "this needs a clearly different system prompt, +toolset, or output style" is a valid justification. If you cannot articulate one, prefer +adding a tool to an existing agent. + +### 2. Handoff edges + +Before adding an LLM-routed handoff edge (e.g. via +`AgentWorkflowBuilder.CreateHandoffBuilderWith`), justify why a deterministic edge or a +conditional `WorkflowBuilder` branch will not work. Every LLM-routed edge is an extra LLM +call before the user gets a response. Deterministic routing is faster and cheaper; reserve +LLM routing for decisions that genuinely require reading user intent. + +### 3. Model selection + +Before defaulting to a frontier model like `gpt-4o`, name the agent's role and pick +from the table below. Routers, validators, formatters, and workers almost never need +a frontier model; defaulting to one is the largest single source of unnecessary spend. + +| Role | Pick | +|-------------------------------------|-----------------------------------------------------------------------------------| +| router / validator / formatter | small-fast model (e.g. `gpt-4o-mini` or current cheap-fast in your Foundry catalog) | +| worker / summarizer / extraction | small-fast model, **or** Foundry `model-router` deployment if prompt length varies | +| planner / decomposer / open reasoning | reasoning-class model (e.g. `o4-mini` or current reasoning model) — state *why* in a code comment | +| creative / nuanced generation | frontier (e.g. `gpt-4o`) — state *why* in a code comment | + +If unsure which role applies, **stop and ask the user** — do not default to `gpt-4o`. +Specific model ids age fast; check your Foundry catalog for the current cheap-fast, +reasoning-class, and frontier ids before pinning. + +### 4. Message-history strategy + +Before sending the full conversation history to an agent, state the bound — turn count, +token cap, summarization point, or retrieval strategy. Unbounded full-history sends in +a multi-turn workflow are flagged when **`unbounded_history_warn`** is true. + +### 5. Token / cost surfacing + +Before implementing a non-trivial change to an agent's prompt, tools, or model, estimate +per-turn token cost. Default warnings: + +- More than **`per_turn_input_token_warn`** input tokens projected per turn +- More than **`per_turn_output_token_warn`** output tokens projected per turn +- Any change that adds more than **`baseline_token_increase_warn_pct`**% to a measured + baseline + +When any of these trips, surface the projection to the user before implementing. + +### 6. Post-change measurement + +After a non-trivial change to a workflow (new agent, new edge, model swap, prompt +rewrite), propose running `setup-maf-evals` (or an existing `.Evals` project) to confirm +the change is net-positive on the metrics that matter — or explicitly note why +measurement is not warranted (e.g. cosmetic refactor with no behavioral change). + + +```` + +## Notes for the skill implementation + +- The outer fence in this file is **four backticks** so the inner three-backtick YAML + fence in the rendered output is preserved verbatim. When transcribing the template, + agents must reproduce the inner three-backtick fences exactly. +- The `` placeholder is filled from the `version:` field in + `SKILL.md`'s frontmatter. Render as `v0.1.0` (lowercase `v`, semver triple). +- The threshold frontmatter is intentionally inside the managed block (not top-of-file + YAML) so it does not interfere with any other YAML frontmatter the project may have. +- Update mode preserves user-edited threshold values per the algorithm in `SKILL.md` + step 2 ("Threshold preservation algorithm"). Do not duplicate that logic here. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md new file mode 100644 index 0000000000..03a465fbc6 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md @@ -0,0 +1,176 @@ +# Rule Rationales + +Long-form rationale, examples, and counter-examples for each of the six rules in the +managed block. The managed block itself is intentionally terse; this file is the +"why" behind each rule, used by the agent when explaining a violation to the user. + +--- + +## 1. Agent count + +**Rule.** Before adding a new agent to a workflow, justify why the new responsibility +cannot be a tool call on an existing agent. **No hard ceiling** — the answer "this needs +a clearly different system prompt, toolset, or output style" is a valid justification. + +**Why it matters.** Each additional agent multiplies the routing surface area (the LLM +has to decide whether *this* turn should go to *that* agent) AND inflates per-turn +input-token cost: every agent's system prompt + tool descriptions are paid on every +turn the agent participates in. Decision quality degrades as choices grow. + +**When a new agent is justified.** The new responsibility involves a meaningfully +different *system prompt*, *toolset*, or *output style* that would muddy the existing +agent's instructions if folded in. Examples: + +- **Yes, new agent:** A "Coder" agent that writes code and a "Reviewer" agent that + critiques code — clearly different roles, different prompts. +- **No, just a tool:** A "Database Reader" agent that looks up rows. This is a tool on + whatever agent needs the data, not its own agent. +- **No, just a tool:** A "Formatter" agent that reformats output. Again, a tool. + +**On thresholds.** Earlier versions of this rule had an `agent_count_max: 3` numeric +ceiling. It was removed because legitimate designs (e.g. specialist-handoff +interview/coaching workflows with 4-5 well-scoped agents) tripped it as often as +genuine bloat did. The justify-the-add gate is the actual mechanism; a number was +illusory precision. + +--- + +## 2. Handoff edges + +**Rule.** Before adding an LLM-routed handoff edge, justify why a deterministic edge or +a conditional `WorkflowBuilder` branch will not work. **No hard ceiling.** + +**Why it matters.** Every LLM-routed edge is an additional LLM call before the user +gets a response. Deterministic routing — "after Coach runs, always return to +Interviewer" expressed as a `WorkflowBuilder` edge — costs zero extra latency and zero +extra tokens. Free-form LLM routing also amplifies decision-quality variance. + +**When LLM routing is justified.** The decision genuinely requires reading the user's +intent — for example, "is this answer detailed enough to grade?" or "which specialist +domain does this question belong to?". When the decision is mechanical ("after coach +runs, always go back to interviewer"), use a deterministic edge. + +**Concrete pattern (good):** +- Interviewer ↔ Coach with one LLM-routed edge from Interviewer ("ready to grade?") + and a deterministic edge back from Coach (always returns to Interviewer for the next + question). + +**Concrete pattern (bad):** +- Five specialists in a fully-connected handoff graph where every transition is + LLM-routed. Symptom: Copilot constantly proposes new edges as the workflow grows. + +**On thresholds.** Earlier versions of this rule had an +`llm_routed_edges_max_per_turn: 2` numeric ceiling. Same reason as rule #1: the gate +is the justification, not the number. A 4-hop deterministic chain with one LLM-routed +intent decision is fine; two LLM-routed edges per turn in a misdesigned graph is not. + +--- + +## 3. Model selection + +**Rule.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the agent's role +and pick from the table below. If unsure which role applies, **stop and ask the user** +— do not silently default to `gpt-4o`. + +**Why it matters.** A frontier model on a router/triage agent costs roughly 10x more +per token than `gpt-4o-mini` and is often *worse* at the routing job (frontier models +are tuned for nuanced generation, not cheap classification). The default-everything-to- +gpt-4o pattern is the largest single source of unnecessary spend in agentic apps. + +**Role → model class:** + +| Role | Pick | Why | +|-------------------------------------|-------------------------------------------------------|--------------------------------------------------------| +| Router / triage / "is this done?" | small-fast (`gpt-4o-mini`, current cheap-fast id) | Classification, not generation; latency dominates | +| Validator / scorer / structured JSON| small-fast + JSON mode + low temp | Deterministic output; cache-friendly | +| Formatter (Markdown / JSON shape) | small-fast, pinned | Output stability matters more than peak quality | +| Worker / summarizer / extraction | small-fast, **or** Foundry `model-router` if prompt length varies | Most calls happen here; latency dominates | +| Planner / decomposer / reasoning | reasoning-class (`o4-mini`, current reasoning id) | Output drives N downstream calls; quality matters most | +| Creative / nuanced generation | frontier (`gpt-4o`, current frontier id) | Genuinely needs frontier capability | + +**When frontier is justified.** The agent's job is genuinely creative or nuanced +generation, or it must follow complex instructions reliably. Routers, validators, +formatters, and most workers almost never fall in this bucket. + +**Specific model ids age fast.** The table above uses `gpt-4o-mini`, `o4-mini`, and +`gpt-4o` as anchor examples. Before pinning, check your Foundry catalog +(https://learn.microsoft.com/azure/foundry/openai/concepts/models) for the current +cheap-fast, reasoning-class, and frontier ids. Foundry's `model-router` deployment is +the recommended pick whenever the prompt length or complexity genuinely varies per +request and you don't need cache stability (typical: worker tier). + +**State the why in code.** When you pick a frontier or reasoning model, leave a +one-line comment naming the role and why the cheaper tier wouldn't work. This makes +the choice auditable and the next person can challenge it. + +--- + +## 4. Message-history strategy + +**Rule.** Before sending the full conversation history to an agent, state the bound — +turn count, token cap, summarization point, or retrieval strategy. + +**Why it matters.** Per-turn token cost grows linearly in history length. A workflow +that sends 50 turns of history into every LLM call is paying for 50 turns of attention +on every single response. This is the most common token-bloat pattern in handoff-style +workflows, where every agent in the chain re-sees everything. + +**Acceptable bounds (in order of preference):** + +1. **Sliding window:** keep only the last N turns. Cheap, simple, bounded. +2. **Summarization checkpoint:** every M turns, replace the oldest portion with a + summary. Preserves long-range context with a fixed-size payload. +3. **Retrieval / per-agent context:** each agent receives only the messages relevant + to its role, not the full transcript. Highest engineering cost; biggest savings. + +**When unbounded full history is justified.** Single-shot interactions (no multi-turn +loop), or workflows where the entire history is short by construction (e.g. always +under 2K tokens). + +--- + +## 5. Token / cost surfacing + +**Rule.** Before implementing a non-trivial change to an agent's prompt, tools, or +model, estimate per-turn token cost. Default warnings: more than 8000 input tokens or +2000 output tokens per turn, or any change that adds more than 20% to a measured +baseline. + +**Why it matters.** The window between "looks fine in dev" and "$5K/month surprise" is +measured in token counts, and most contributors do not look at token counts at dev time +unless something is on fire. Surfacing projected token cost *before* implementation +catches the bloat at the cheapest possible moment. + +**How to estimate.** Token count is roughly characters / 4 for English prose; +structured output is denser. For prompt changes, count the new prompt body. For tool +additions, count the schema + description per tool * expected frequency. For model +swaps, multiply by the new model's per-token price ratio. + +**When to skip.** Cosmetic changes that do not alter prompt or tool schema (e.g. +renaming a class). Changes to non-LLM code paths. + +--- + +## 6. Post-change measurement + +**Rule.** After a non-trivial change to a workflow, propose running `setup-maf-evals` +(or an existing `.Evals` project) to confirm the change is net-positive — or explicitly +note why measurement is not warranted. + +**Why it matters.** "Trial and error" is the default tuning loop in agentic apps and +it produces survivor-bias outcomes — changes that *seemed* better are kept; changes +whose downsides did not surface in the first few hand-tested turns get baked in. A +small eval suite (3-10 scenarios) closes this loop with data. + +**Trigger conditions.** Any of the following count as a "non-trivial change" warranting +a measurement proposal: + +- Adding or removing an agent +- Adding or rewiring a handoff edge +- Swapping a model +- Substantially rewriting an agent's instructions +- Adding or substantially modifying a tool + +**When skipping is fine.** Cosmetic refactors. Test-only changes. Changes that the +existing eval suite already covers — in that case, the proposal is to *run* the suite, +not to author new scenarios. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md new file mode 100644 index 0000000000..d2ce7698d9 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md @@ -0,0 +1,34 @@ +# Threshold Defaults + +Each numeric threshold in the managed block has a default value and a rationale. These +defaults are starting points, not absolutes — projects with different shapes (e.g. very +simple two-agent workflows, or complex tool-heavy pipelines) should adjust. + +| Threshold | Default | Rationale | +|-----------|---------|-----------| +| `per_turn_input_token_warn` | `8000` | Modern reasoning models can take 100K+, but most chat-class models start showing meaningful latency and cost above ~8K input tokens. Projects with retrieval/RAG legitimately exceed this — override locally. | +| `per_turn_output_token_warn` | `2000` | Output tokens are usually 4-10x more expensive than input on a per-token basis. 2000 is a reasonable "are you sure?" threshold; long-form generation tasks should override. | +| `baseline_token_increase_warn_pct` | `20` | A 20% increase per turn meaningfully changes monthly bills at scale. Small tweaks under 20% are noise; over 20% is worth surfacing. | +| `unbounded_history_warn` | `true` | Default-on. Sending full history forever is the single most common token-bloat pattern. Disable only if the workflow has already implemented a windowing/summarization strategy and the warning is now noise. | + +**Note on removed thresholds.** Earlier versions of this skill (≤v0.2.0) shipped +`agent_count_max: 3` and `llm_routed_edges_max_per_turn: 2`. Both were taste-call +ceilings that tripped legitimate designs (5-specialist handoff workflows, multi-hop +deterministic chains with one LLM-routed intent decision) as often as they caught real +bloat. The justify-the-add gates in rules #1 and #2 are the actual mechanism; the +numbers were illusory precision and were removed in v0.3.0. + +## Adjusting thresholds + +Users override defaults inside the managed block's `thresholds:` YAML map. The skill +preserves these overrides on update (it merges new defaults underneath, so user values +win for any key they set). + +Examples of legitimate overrides: + +- A RAG-heavy workflow with retrieval that routinely sends 20K input tokens — set + `per_turn_input_token_warn: 25000`. +- A long-form drafting tool that generates 5K outputs per turn — set + `per_turn_output_token_warn: 6000`. +- A workflow that has implemented summarization-at-N-turns — set + `unbounded_history_warn: false`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md new file mode 100644 index 0000000000..85bb5e903a --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -0,0 +1,337 @@ +--- +name: scan-agentic-app-perf +description: | + Scan a .NET agentic application (MAF; Aspire/Foundry optional) across seven perf/cost/reliability categories — topology, tool inventory, message history, prompt weight, parallelism, OTel coverage, per-agent model assignment. Writes .copilot/perf-reports/scan.md (overwritten each run) with severity-tagged findings (critical/warn/info), file:line citations, evidence, and next actions routing into configure-agentic-perf-rules or setup-maf-evals. WHEN: user asks "why is my agent slow", "scan/audit my agentic app", "find perf issues", "is my topology too complex", or just changed a topology. NOT-WHEN: install always-on rules (use configure-agentic-perf-rules), wire evaluations (use setup-maf-evals); not for non-agentic .NET apps. Read-only. Supported topologies: Aspire AppHost, plain console, ASP.NET Core, worker service (Aspire-specific checks such as the AppHost-only model literal apply only when an AppHost is detected). INVOKES: scripts/Detect-{Topology,OtelCoverage,ModelLiterals}.ps1 (pwsh 7+). +--- + +# scan-agentic-app-perf + +Run a structured audit of a .NET agentic application and produce a single +Markdown report listing the perf, cost, and reliability issues that matter, +each with a concrete next action. + +This skill is **read-only**. It never edits source. The output is a report file +plus a short chat summary of top findings. + +## When to Use + +- The user asks "why is my agent slow", "scan my agentic app", "audit my + agentic app", "find perf issues", or "is my topology too complex". +- The user has just modified an agent topology (added an agent, added a + handoff edge, swapped a model, added tools) and wants a sanity check + before merging. +- A coding agent is about to make a non-trivial change to a `.NET` + agentic app and wants the current perf/cost baseline to compare + against post-change. + +## When Not to Use + +- The user wants to install **always-on** rules so future agent work + surfaces perf concerns by default — use `configure-agentic-perf-rules`. +- The user wants to wire **runtime** measurement (latency, tokens, + cost, quality scores) into the project — use `setup-maf-evals`. +- The project is not a .NET agentic app — use `optimizing-dotnet-performance` + for general .NET performance guidance. +- The user wants the skill to **fix** the findings, not just report + them — this skill is read-only by design; route fixes through + `configure-agentic-perf-rules` (for guard-rails) or normal coding + agent work (for code changes). + +## Supported topologies + +The skill targets any .NET project using Microsoft Agent Framework +(`Microsoft.Agents.AI`), regardless of how it's hosted: + +- **Aspire AppHost** (`*.AppHost.csproj` orchestrating agent services) — the most + common shape; enables AppHost-specific checks such as the "model literal lives + only in the AppHost" check. +- **File-based AppHost** (.NET 10) — a single `apphost.cs` (or similarly named + `.cs`) that carries `#:sdk Aspire.AppHost.Sdk` / `#:package` directives and a + `DistributedApplication.CreateBuilder(...)` call, run via `dotnet run apphost.cs` + with **no** `.csproj`/`.sln`. Detected by parsing source text (see step 1), so + the same checks and detection scripts apply. +- **Plain console / worker service** — `Microsoft.Agents.AI` registered directly + against an OpenAI or Foundry client without an Aspire AppHost. +- **ASP.NET Core minimal API** — agents registered via the same DI patterns. + +AppHost-specific checks (those that depend on `*.AppHost.csproj` being present) +are silently skipped when no AppHost is detected. All other checks (topology, +tool inventory, message history, prompt weight, parallelism, OTel coverage, +per-agent model assignment) apply to every topology. + +## Workflow + +> **Execution discipline — the user asked a question; answer it in chat.** +> - **Your chat response is the primary deliverable.** The user asked you to +> audit their app; they read your answer in the chat, not a file on disk. Your +> final message **must** present the findings (see step 5) — total counts plus +> the top findings with file:line and next action. `scan.md` is a persisted +> copy of that same analysis, not a substitute for it. A run that writes the +> file but ends with an empty or one-line chat message has failed the user. +> - **Analyze first, then write both.** Inspect the sources, reason about the +> findings, then (a) present them in chat and (b) persist the same findings to +> `.copilot/perf-reports/scan.md`. Do not open an empty report file up front +> and treat filling it as the goal — that starves the chat answer. Both the +> chat summary and the file are required; the chat summary is the last thing +> you do. +> - **Anchor on what the user asked.** If the prompt names a concern +> (topology / "too complex", history / "slow on multi-turn", cost / prompt +> weight, observability / OTel), **lead both the chat answer and the report +> with that category** — never bury it under an unrelated one. Still run the +> full seven-category sweep, but the user's stated problem is the headline. +> - **Stay lean; don't over-tool.** On a small app (a handful of source files) +> **read each source file once, skip the detection scripts, and open no +> reference docs.** The PowerShell detectors and per-category reference docs are +> accelerators for large codebases, not a mandatory gauntlet; running them on a +> tiny fixture just burns your turn budget and adds nothing a direct read +> doesn't already give you. Concretely, for an app of ~5 files or fewer, aim to +> finish inventory + all seven categories in **well under ~10 tool calls** — +> read the files, reason, answer. Do not re-open a file you already read, never +> pre-read all seven reference docs, and open one only to confirm a specific +> candidate finding you can't resolve from the source. + +### 1. Inventory the app + +Detect and record: + +- AppHost project (`*.AppHost.csproj`), **or** a file-based AppHost — a bare + `.cs` (commonly `apphost.cs`) with `#:sdk Aspire.AppHost.Sdk` / + `#:package Aspire.Hosting.*` directives and a `DistributedApplication.CreateBuilder` + call and no `.csproj`/`.sln` — and agent service projects +- Agent registrations (`AddAgent`, `ChatClientAgent`, `IChatClient` builders) +- Agent count, handoff edges, tool count per agent +- OTel wiring (`AddOpenTelemetry`, Aspire dashboard reference) + +**Automated detection (large apps).** For the topology, OTel-coverage, and +model-assignment categories on a **larger codebase**, the deterministic detection +scripts save you from reading every source file by hand. **On a small app (a +handful of files), skip them and read the sources directly** — invoking them on a +tiny fixture costs more than it saves. When you do use them, invoke each as +`& "/scripts/.ps1" -Path -Json` and parse the +JSON: + +- `Detect-Topology.ps1` — agent nodes, handoff edges, orphaned agents, per-file + fan-out. +- `Detect-OtelCoverage.ps1` — presence of Aspire service defaults, the OTel SDK, + an OTLP exporter, tracing/metrics builders, and `gen_ai.*` token telemetry. +- `Detect-ModelLiterals.ps1` — model-id literals and Foundry enum refs, split by + AppHost vs service file, with a distinct-id count. + +The scripts parse source text, so they also cover a file-based AppHost. They do +**detection only** — they never assign severity. Feed their output into the check +classes in step 2, then apply the evidence gate and severity rules yourself. If a +script errors or emits nothing, fall back to reading the reference doc and +scanning by hand (graceful degradation). See `references/detection-scripts.md` +for the exact contract. + +Only abort if the project has **no agent/AI surface at all** — no Aspire +AppHost, no `Microsoft.Agents.AI` / `ChatClientAgent`, and no `IChatClient` or +OpenAI/Foundry client anywhere. An Aspire AppHost that orchestrates services, or +any file that wires an `IChatClient` or agent, is in scope — proceed and run the +checks (the OTel-coverage, topology, and model-assignment categories all apply to +an Aspire app even before you open a service file). **Do not refuse to analyze a +project just because it does not reference `Microsoft.Agents.AI` specifically** — +plain `IChatClient` / Azure OpenAI / Foundry apps are supported topologies. Only +a genuinely non-AI .NET project (no AI clients, no agents, no Aspire) is out of +scope. + +### 2. Run the seven check classes + +Evaluate the seven categories below and collect the confirmed findings; you will +present them in chat (step 5) and persist them to the report (step 4). **Do not +pre-read all seven reference docs** and **do not open an empty report file first** +— reason about the findings before you write anything. Open a category's +reference doc only to confirm a specific candidate finding. + +| # | Category | Reference | +|---|-------------------|--------------------------------------------| +| 1 | Topology | `references/topology-checks.md` | +| 2 | Tool inventory | `references/tool-inventory-checks.md` | +| 3 | Message history | `references/message-history-checks.md` | +| 4 | Prompt weight | `references/prompt-weight-checks.md` | +| 5 | Parallelism | `references/parallelism-checks.md` | +| 6 | OTel coverage | `references/otel-coverage-checks.md` | +| 7 | Model assignment | `references/model-assignment-checks.md` | + +For each check, record any findings with the schema in step 3. + +Categories 1 (topology), 6 (OTel coverage), and 7 (model assignment) can be +seeded by the detection scripts **if you ran them** (large apps) — read the +parsed JSON, then confirm each candidate against its reference doc. **On a small +app you skipped the scripts**, so evaluate these three the same way as the rest: +by reading the source directly. All remaining categories are evaluated by +inspecting the cited sources directly; consult a `references/` doc only when a +candidate finding needs confirmation. + +### 3. Finding schema + +Every finding is a dict with these fields: + +```yaml +severity: critical | warn | info +check: one of the slugs listed in references/check-glossary.md + (e.g. model.same-default, history.full-share, otel.missing-sdk) +title: short imperative phrase +file: path relative to repo root +line: 1-based line number, or null +evidence: 1-3 line code snippet or measurement (must be present in the cited file) +why: one paragraph explaining the impact +next: concrete action the developer can take next +ref: optional cross-skill route (e.g. "skill:configure-agentic-perf-rules") +``` + +The `check` field is a dotted slug: `.`. The +prefix before the dot encodes the category — there is no separate +`category` field. Category card (also rendered at the top of the +report's `## Findings` section): + +| Category | Coverage | Reference | +|------------|-------------------------|----------------------------------------| +| `topology` | agent graph shape | `references/topology-checks.md` | +| `tools` | per-agent tool list | `references/tool-inventory-checks.md` | +| `history` | chat history strategy | `references/message-history-checks.md` | +| `prompt` | system prompt size/reuse| `references/prompt-weight-checks.md` | +| `parallel` | concurrent invocations | `references/parallelism-checks.md` | +| `otel` | instrumentation | `references/otel-coverage-checks.md` | +| `model` | per-agent model id | `references/model-assignment-checks.md`| + +See `references/check-glossary.md` for the full slug→description table. + +**Evidence gate** — before adding a finding to the report, re-open the +cited file and confirm the `evidence` snippet is present at or near +the cited line. Drop any finding that fails this check. Findings whose +evidence is "absence of X" must list the exact files and patterns +that were searched in the `evidence` field instead of a snippet. + +Severity rules: + +- **critical** — likely to break a user-visible flow, blow the token budget, + or cause a cost spike. Always surfaced in chat. +- **warn** — measurable perf or cost regression, but app still works. +- **info** — observation worth knowing, no action required. + +### 4. Aggregate and write the report + +Once you have the confirmed findings, sort them by severity (critical → warn → +info), then by `check` slug (stable lexical order so `history.*` < `model.*` < +`otel.*` < `parallel.*` < `prompt.*` < `tools.*` < `topology.*`) and write them +to the report with the `create` tool. This is a persisted copy of the analysis +you present in chat (step 5) — not the primary deliverable, but always written. + +**Always write the report** — even for a read-only audit and even when +zero criticals are found (an empty-findings report is still worth persisting; +writing it is not a source edit). Write it to +`.copilot/perf-reports/scan.md` **relative to the scanned app's root** — +the directory that contains the solution / `*.AppHost.csproj`, or the +path the user pointed you at (e.g. auditing `./fixture` writes +`fixture/.copilot/perf-reports/scan.md`). Overwrite it on every run. +Git provides whatever history you want; the skill does +not maintain timestamped copies or `latest-*` mirrors. (Exception: when +a solution has multiple `*.AppHost.csproj` projects, write one report +per host as `scan-.md` — see `references/common-pitfalls.md`.) + +The category legend (one-liners describing what each prefix covers) is +inlined into the report itself — see `references/report-template.md`. +There is no separate glossary file. The slug encodes the check (e.g. +`history.full-share`, `topology.cycle`); readers don't need a lookup +table. + +Create `.copilot/perf-reports/` if it does not exist. + +This skill never touches `.gitignore`. If the user wants the report +ignored, recommend in chat that they add `.copilot/perf-reports/` to +their `.gitignore` themselves; do not edit it from this skill. + +### 5. Surface top findings in chat (required — this is your answer) + +This step is the primary deliverable. Your final chat message **must** contain +the analysis, even after you have written `scan.md`. Print: + +1. Total counts (critical / warn / info). +2. **The finding(s) in the user's stated concern category first** (e.g. topology + when they said "too complex"), then the first up to 3 critical findings — + each with title + `check` slug + file:line + next action. +3. The full report path. +4. If any findings have a `ref:` field, list the suggested follow-up skills. + +Do not paste the entire report into chat, but never end with an empty or +one-line message — a saved file is not a substitute for answering the user. + +### 6. Offer to route into follow-up skills + +After surfacing the top findings, **ask the user once** whether they +want to act on the routed follow-ups. The skill itself never edits +source — this step only routes into a sibling skill that owns its own +diff-and-confirm flow. + +1. Aggregate the unique `ref:` values across all findings. +2. If the set is non-empty, print one prompt of the form: + > Want me to follow up on any of these? + > - **A.** Install/update perf rules via `configure-agentic-perf-rules` to enforce role-aware model selection on future code. + > - **B.** Run `setup-maf-evals` to capture token/quality numbers. + > - **C.** Run `configure-agentic-perf-rules` to install always-on rules. + > - **D.** No — just leave the report. + Render only the lettered options that correspond to refs actually + present in the findings (skip letters whose target skill is not + referenced). +3. Wait for the user's response. If they pick a letter, hand off to the + named skill with the audit report path as context. If they pick + "no" or anything else, stop. +4. Do **not** infer intent from the original audit-request prompt + ("audit and fix it" is *intent* — the follow-up skill must still + present its own diff and obtain its own confirmation before any + write). This skill's job ends at the routing offer. + +### 7. Stop + +This skill never edits source. If the user declined the offer in +step 6, or if there were no `ref:` fields in any finding, the skill +ends here. + +## Validation + +After running: + +- A file exists at `.copilot/perf-reports/scan.md` (overwritten if it + was there before). +- The report has a `## Findings` section, even if empty (containing + `_No findings._`). +- The summary counts in the report match the chat output. + +## Common pitfalls + +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never edit `.gitignore`, + source files, config files, or anything else. If a check tempts you + to fix the issue inline, stop and add it as a finding instead. +- **Hallucinating findings.** Every finding must cite a real file and + (where applicable) a real line. Before adding a finding to the + report, re-open the cited file and verify the snippet exists at the + cited line. If you cannot point to evidence, drop the finding. +- **Burying critical findings.** Always lift the top 3 critical + findings into chat. Do not say "see the report" without surfacing + the worst issues. +- **Confusing this with rules install.** If the user wants the rules + themselves embedded into their instructions file, run + `configure-agentic-perf-rules` instead. +- **Running on non-agentic apps.** If no agent registrations are found, + abort cleanly. Do not invent an audit for a plain web API. +- **Recommending a model downgrade without an eval gate.** Any MA* + finding that says "downgrade Agent X from gpt-4o to gpt-4o-mini" + must be paired in the `next:` field with "validate via + `setup-maf-evals` quality mode before shipping". Apparent free wins + on cost frequently regress quality on edge cases — the eval gate + protects against that. + +## References + +- `references/topology-checks.md` — agent count, handoff edges, cycles. +- `references/tool-inventory-checks.md` — tools per agent, redundancy, dead tools. +- `references/message-history-checks.md` — full-history sharing, summarization. +- `references/prompt-weight-checks.md` — system-prompt size, per-agent token cost. +- `references/parallelism-checks.md` — sequential calls that could fan out. +- `references/otel-coverage-checks.md` — Aspire dashboard, token/cost telemetry. +- `references/model-assignment-checks.md` — single-model defaulting, role mismatch. +- `references/detection-scripts.md` — contract for the `scripts/Detect-*.ps1` detectors (topology, OTel, model). +- `references/check-glossary.md` — dev-facing catalog of all check slugs (NOT copied to user repos; for skill maintainers). +- `references/report-template.md` — exact Markdown layout for `scan.md`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md new file mode 100644 index 0000000000..3faa29a706 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md @@ -0,0 +1,41 @@ +# Check glossary (dev-facing) + +Canonical catalog of every `check` slug the skill can emit. This file +is for **skill maintainers** — adding or renaming a check requires a +row here. It is **not copied** into user repositories; the slugs are +self-describing in the report itself. + +If you change this table, also update the matching `*-checks.md` +reference file with the per-check detection logic. + +## Catalog + +| Check | Sev | What it catches | +|------------------------------------|----------|-----------------| +| `topology.cycle` | critical | Directed cycle in the agent handoff graph | +| `topology.deep-single-leaf` | warn | 3+ hops to reach a single terminal agent | +| `tools.duplicate` | warn | Same tool description on >1 agent | +| `tools.dead` | info | Tool registered but never referenced | +| `tools.description-too-long` | warn | `[Description]` attribute > 200 chars | +| `history.full-share` | critical | Entire chat history passed to a downstream agent | +| `history.unbounded` | warn | No `MaxMessages` / reducer / summarizer wired | +| `history.through-deterministic` | warn | Full history given to a deterministic agent | +| `prompt.oversized` | warn | System prompt > 2K tokens (critical at > 4K) | +| `prompt.duplicate-preamble` | warn | Same > 100-token block shared across agents | +| `parallel.independent-handoffs` | warn | Sequential `await`s on agents that don't share data | +| `parallel.hidden-tool-fanout` | info | Tool internally serializes 3+ API calls | +| `otel.missing-sdk` | critical | No `AddOpenTelemetry` / Aspire dashboard wiring | +| `otel.no-aspire-dashboard` | warn | Aspire dashboard not declared | +| `otel.no-token-cost` | warn | No `gen_ai.usage.*` tags surfaced | +| `otel.no-per-agent-source` | info | All agents share one `ActivitySource` | +| `model.same-default` | warn | All agents use the same model id | +| `model.reasoning-on-deterministic` | warn | Frontier model on a formatter / validator | +| `model.cheap-on-planner` | warn | Small model on the planner, larger on workers | +| `model.hardcoded` | info | Model id literal in agent service `.cs` | + +## Cross-skill routes + +| `ref:` value | Skill to run next | +|--------------------------------------|------------------------------------------------------------| +| `skill:configure-agentic-perf-rules` | Install/update the always-on perf rules (rule #3 covers role-aware model selection) | +| `skill:setup-maf-evals` | Wire eval reports + telemetry | diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md new file mode 100644 index 0000000000..9f9bd25cc6 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md @@ -0,0 +1,95 @@ +# Common pitfalls + +Real-world failure modes for `scan-agentic-app-perf` — observed during +dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). + +## Output discipline + +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never touch `.gitignore`, + source files, config files, AppHost, or `*.csproj`. If a check tempts + you to fix the issue inline, stop and add it as a finding instead — + the user opts into the fix through the step-6 routing prompt. +- **Surfacing more than 3 critical findings in chat.** Step 5 caps + chat output at 3 critical findings + counts + report path. Anything + beyond that goes in the report file only. Pasting the entire report + into chat defeats the "single file you can re-open" UX. +- **Generating extra report files.** One file: `scan.md`. Do not + write timestamped copies, `latest-*` mirrors, glossary files, or + any other artifact. Git is the history mechanism if the user wants + one. + +## Evidence integrity + +- **Hallucinating findings.** Every finding must cite a real file + and (where applicable) a real line. Before adding a finding, re-open + the cited file and verify the snippet exists at the cited line — that + is the "evidence gate" in step 3. If you can't point to evidence, + drop the finding. A 5-finding report with verified citations beats + a 15-finding report with two hallucinations every time. +- **Citing line ranges without re-reading.** Files drift between + the inventory pass (step 1) and the per-category checks (step 2). + Re-read the cited region right before writing each finding, not + before the entire batch. +- **Absence-of-X findings without searched-pattern evidence.** When + a finding fires because something is *missing* (e.g. `otel.no-token-cost` "no token + surfacing"), the `evidence` field MUST list the exact files and + patterns that were searched — not a snippet. This is what lets the + user reproduce the negative result. + +## False positives we keep seeing + +- **`model.hardcoded` firing on AppHost code.** This check is about + model ids living in *agent service* files (`*.Agent/Program.cs` + etc.) where swapping requires a code change. Model ids declared in + the AppHost via `foundry.AddDeployment("chat", FoundryModel.OpenAI.Gpt4oMini)` + are the **canonical Aspire-native place** — that's not a defect. + When you encounter this pattern, either suppress `model.hardcoded` + entirely or downgrade to `info` with a "no action required" `next:`. +- **`model.same-default` on single-agent apps.** The check explicitly + assumes ≥2 agents (different roles, different needs). Do not fire on + 1-agent apps; the check is trivially "satisfied" with one model. +- **`otel.no-token-cost` when `Microsoft.Extensions.AI` activity + source is registered.** MEAI emits `gen_ai.usage.*` activity tags + automatically; if `AddSource("Microsoft.Extensions.AI")` is wired in + OTel and an OTLP exporter is configured, this check should NOT + fire. The check is for codebases that strip the source or wrap MEAI + behind custom infrastructure that loses the tags. + +## Per-check sharpening + +- **`prompt.oversized`.** Use a rough token estimator (chars/4) or + `cl100k_base` if available. Never claim an exact token count without + naming the encoder you used. +- **`tools.duplicate`.** Compare tool *descriptions* (the + `[Description("...")]` attribute), not method names. Two tools with + the same method name but different descriptions are usually fine; + two tools with different names and a copy-pasted description are + usually a refactor candidate. + +## Routing offer (step 6) + +- **Inferring intent from the original prompt.** Even if the user + said "audit and fix it", this skill's job ends at the routing + prompt. The follow-up skill is responsible for its own + diff-and-confirm flow. Never call into `configure-agentic-perf-rules` + (apply mode) without the user explicitly + picking that letter at the prompt. +- **Listing routes for skills not actually referenced.** Step 6 says + to render only letters whose target skill is named in some + finding's `ref:` field. If no finding routed to `setup-maf-evals`, + do not offer it. Empty routing offer = skip the prompt entirely. + +## Inventory edge cases + +- **Multiple AppHost projects.** If a solution has more than one + `*.AppHost.csproj`, scan each one and emit one report per host as + `.copilot/perf-reports/scan-.md` (e.g. `scan-WebApp.md`, + `scan-Worker.md`). Do NOT merge — the topology, model set, and + OTel wiring belong to each host independently. Each per-host file + is still overwritten on each scan; no timestamp suffixes. +- **Agent registered via DI lambda without `AddAIAgent`.** Patterns + like `services.AddSingleton(...)` or custom factories also + count as agents. The detection in step 1 must include any path + that ends up registering an `IAgent` / `AIAgent` in the host's + service provider. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/detection-scripts.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/detection-scripts.md new file mode 100644 index 0000000000..9a77a17107 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/detection-scripts.md @@ -0,0 +1,84 @@ +# Detection scripts + +The `scripts/` directory ships three deterministic detectors that automate the +mechanical extraction behind the `topology.*`, `otel.*`, and `model.*` checks. +They exist to cut token cost and run-to-run variance: the skill spends its +budget on judgment (severity, the `why`/`next` narrative, role-vs-model calls) +instead of re-reading every source file. + +## Contract + +- **Detection only.** A script never assigns severity, never writes a report, + and never edits the scanned project. It emits facts + best-effort `flags`. +- **Invocation.** `& "/scripts/.ps1" -Path -Json`. + Omit `-Json` for a short human summary. The skill loader supplies + ``. Requires PowerShell 7+ (`pwsh`). +- **Source-text parsing.** No build, no Roslyn — regex over `.cs` (and + `appsettings*.json` for OTel). This is why they also work on a file-based + AppHost (a bare `.cs` with no `.csproj`/`.sln`). +- **Always exit 0.** On success `ok = true`. The skill must still guard against + a non-zero exit, empty output, or malformed JSON and **fall back to reading + the reference doc and scanning by hand** (graceful degradation). +- **Evidence gate still applies.** Every `file`/`line` a script emits must be + re-opened and confirmed before it becomes a report finding (see SKILL.md + step 3). A `flag` set by a script is a *candidate*, not a verdict. + +## `Detect-Topology.ps1` + +Feeds `references/topology-checks.md` (`topology.cycle`, `topology.deep-single-leaf`). + +JSON shape: + +``` +{ ok, scanned, + metrics: { agentCount, edgeCount, orphanCount, maxFanout }, + nodes: [ { name, kind, file, line } ], # kind: project | agent + edges: [ { source, target, file, line } ], + orphans:[ "" ], + notes } +``` + +Edge `source` is inferred from the defining file/project (best-effort); confirm +direction against the cited file before asserting a cycle or a deep chain. The +script does not decide severity — a warn-level orphan/fan-out signal only +becomes critical if you confirm a cycle. + +## `Detect-OtelCoverage.ps1` + +Feeds `references/otel-coverage-checks.md`. + +JSON shape: + +``` +{ ok, scanned, + present: { serviceDefaults, addOpenTelemetry, otlpExporter, withTracing, + withMetrics, genAiTokens, aspireDashboard, sensitiveData }, # booleans + evidence: { : { file, line, text } | null }, + flags: { otel.missing-sdk, otel.no-aspire-dashboard, otel.no-token-cost }, # booleans + notes } +``` + +`otel.missing-sdk` fires only when **neither** `AddServiceDefaults` **nor** +`AddOpenTelemetry` is present. `gen_ai.*` tags are emitted automatically by +`Microsoft.Extensions.AI`, so confirm exporter wiring before asserting +`otel.no-token-cost`. + +## `Detect-ModelLiterals.ps1` + +Feeds `references/model-assignment-checks.md`. + +JSON shape: + +``` +{ ok, scanned, + distinctIds: [ "" ], + hits: [ { id, form, appHost, file, line, text } ], # form: literal | enum | deployment-call + flags: { model.same-default, model.hardcoded }, # booleans (candidates) + notes } +``` + +An AppHost `AddDeployment(...)` is the canonical model-id location and does +**not** trip `model.hardcoded`; only a literal in a non-AppHost service file +does. The role-driven checks (`model.reasoning-on-deterministic`, +`model.cheap-on-planner`) are **not** decided by the script — they require +reading each agent's prompt/tools, which the skill does directly. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md new file mode 100644 index 0000000000..db98a2b686 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md @@ -0,0 +1,39 @@ +# Message-history checks + +Detect strategies that pass too much history to too many agents. + +## Checks + +### `history.full-share` (critical) + +**Detect:** code that passes the entire `IList` (or +`History`) from the entry agent to a downstream agent without +filtering, slicing, or summarizing. + +**Why:** every additional agent that sees the full history pays the +input token cost. With 4 agents and a 6K-token history, you spend 24K +input tokens per turn doing nothing. + +**Next:** "Pass only the last user message and a one-paragraph summary +to ``. Use `IChatHistoryReducer` or a manual slice." + +### `history.unbounded` (warn) + +**Detect:** no usage of `MaxMessages`, `IChatHistoryReducer`, +summarization tool, or sliding-window code anywhere in agent setup. + +**Why:** unbounded history = monotonically growing per-turn cost. + +**Next:** "Wire a `ChatHistoryReducer` with `MaxMessages = 20` or +summarize-and-replace at the agent level." + +### `history.through-deterministic` (warn) + +**Detect:** an agent whose role is purely deterministic (formatter, +validator, tool router) is given full chat history. + +**Why:** deterministic steps do not need conversational context. Their +prompt cost should be near-constant. + +**Next:** "Pass only the immediate input artifact to ``; drop +the chat history." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md new file mode 100644 index 0000000000..dbcdfe5577 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md @@ -0,0 +1,66 @@ +# Model-assignment checks + +Detect single-model defaulting and role-model mismatch. + +## Checks + +### `model.same-default` (warn) + +**Detect:** every agent is constructed with the same model id (e.g. +`gpt-4o-mini`). Look at the AppHost connection string or the +`IChatClient` builder per agent service. Requires ≥2 agents — skip on +single-agent apps. + +**Why:** different agent roles have different latency and quality +needs. A single default usually overspends on cheap roles and +underspends on hard roles. + +**Next:** "See rule #3 in `.github/copilot-instructions.md` (managed by +`configure-agentic-perf-rules`) — most apps want a small-fast model +for routers/validators/workers and a reasoning-class model only for +planners. If both agents are workers on the same cheap model, this is +expected and can be downgraded to info." + +**Ref:** `skill:configure-agentic-perf-rules` + +### `model.reasoning-on-deterministic` (warn) + +**Detect:** an agent whose prompt and tool list indicate a +deterministic role (formatter, validator, classifier with ≤3 outputs) +is using a frontier reasoning model. + +**Why:** the marginal quality is near-zero; you are paying for unused +capability and per-call latency. + +**Next:** "Downgrade `` to a small-fast model per rule #3 in +`.github/copilot-instructions.md`. Validate via `setup-maf-evals` +quality mode." + +**Ref:** `skill:configure-agentic-perf-rules` + +### `model.cheap-on-planner` (warn) + +**Detect:** the agent that decides the plan or decomposes the task is +on a small model while leaf workers are on a large one. + +**Why:** plan-quality drives every downstream call. A bad plan from a +cheap planner makes the expensive workers run more turns. + +**Next:** "Promote the planner to a reasoning-class model per rule #3 +in `.github/copilot-instructions.md`; consider demoting one or more +workers." + +**Ref:** `skill:configure-agentic-perf-rules` + +### `model.hardcoded` (info) + +**Detect:** model id literal (e.g. `"gpt-4o-mini"`) appears inside an +agent service `.cs` file rather than `appsettings.json` or AppHost +parameters. AppHost code that declares model ids via +`foundry.AddDeployment(...)` is the canonical Aspire pattern and does +NOT trigger this check. + +**Why:** swapping models for an A/B becomes a code change. + +**Next:** "Move model ids into `appsettings.json` and bind them via +`IOptions<...>`, or declare them in the AppHost as Aspire deployments." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md new file mode 100644 index 0000000000..d25687e6e8 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md @@ -0,0 +1,54 @@ +# OTel coverage checks + +Detect missing instrumentation that makes perf invisible at dev time. + +## Checks + +### `otel.missing-sdk` (critical) + +**Detect:** the AppHost or service projects do not call +`builder.Services.AddOpenTelemetry()` and do not import +`OpenTelemetry.Exporter.OpenTelemetryProtocol` or +`Aspire.Hosting.Dashboard`. + +**Why:** without OTel, you cannot see per-call latency, token counts, +or spans. Every other check in this audit becomes a guess. + +**Next:** "Add `builder.AddServiceDefaults()` (Aspire) or wire OTel +manually with HTTP + Activity sources for `Microsoft.Extensions.AI`." + +### `otel.no-aspire-dashboard` (warn) + +**Detect:** the AppHost does not declare the dashboard, or the +`appsettings.json` lacks a `Dashboard:OtlpEndpointUrl`. + +**Why:** the dashboard is the cheapest way to see per-agent token use +during local dev. + +**Next:** "Run with `dotnet run --project ` and ensure the +dashboard URL is logged. If not, install `Aspire.Hosting.Dashboard`." + +### `otel.no-token-cost` (warn) + +**Detect:** no log, no meter, no tag for `gen_ai.usage.input_tokens` / +`gen_ai.usage.output_tokens` anywhere in the codebase. + +**Why:** without token telemetry the team has no early-warning signal +for prompt bloat. Cost spikes are discovered in the bill, not the +dashboard. + +**Next:** "Microsoft.Extensions.AI emits `gen_ai.*` activity tags +automatically. Confirm the OTel exporter forwards them, or run +`setup-maf-evals` to capture them in eval reports." + +**Ref:** `skill:setup-maf-evals` + +### `otel.no-per-agent-source` (info) + +**Detect:** all agents share a single activity source name; no way to +filter the dashboard by agent. + +**Why:** with 3+ agents, traces become unreadable without filtering. + +**Next:** "Give each agent its own `ActivitySource` named after the +agent role." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md new file mode 100644 index 0000000000..e27f78e3e9 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md @@ -0,0 +1,37 @@ +# Parallelism checks + +Detect sequential agent invocations that could run concurrently. + +## Checks + +### `parallel.independent-handoffs` (warn) + +**Detect:** two consecutive `await downstreamA.RunAsync(...)` and +`await downstreamB.RunAsync(...)` calls in the same method where B's +input does not depend on A's output. + +**Why:** the second call could start as soon as the inputs are known. +Each sequential LLM hop adds full per-call latency. + +**Next:** "Run `` and `` with `Task.WhenAll`. Rejoin in the +parent agent for the consolidation step." + +### `parallel.hidden-tool-fanout` (info) + +**Detect:** a tool method that internally loops and calls 3+ external +APIs sequentially. + +**Why:** tools hide their own latency from the agent. A single slow +tool that is internally serial is the hardest kind of latency to find +from the outside. + +**Next:** "Parallelize the inner calls in ``; document the +expected bound in the tool description so the agent can plan around +it." + +## What used to live here + +`parallel.sequential-awaits` (was `P1`, a generic `foreach (var x in +xs) await ...` pattern) was removed — that's a general .NET concurrency +anti-pattern, not specific to agents. For that class of finding run +`optimizing-dotnet-performance` instead. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md new file mode 100644 index 0000000000..7e76736b0c --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md @@ -0,0 +1,40 @@ +# Prompt-weight checks + +Detect oversized system prompts and per-agent prompt cost. + +## Checks + +### `prompt.oversized` (warn at >2K tokens / critical at >4K) + +**Detect:** count tokens (or chars / 4 as approximation) in each +agent's `Instructions` or system prompt string. + +**Why:** every turn pays this cost. A 4K-token system prompt at +$0.005/1K input tokens × 1000 turns/day = $20/day per agent on prompt +overhead alone. + +**Next:** "Move static rules into a tool the agent can call when +needed, or split the prompt into a short policy section and a separate +few-shot example doc retrieved on demand." + +### `prompt.duplicate-preamble` (warn) + +**Detect:** two or more agents share the same > 100-token block at the +start or end of their system prompts. + +**Why:** the same tokens are billed N times per turn (once per agent). + +**Next:** "Lift the shared block into a single deterministic +preprocessor or attach it as a tool result rather than a system prompt +repeat." + +## Token estimation + +If a real tokenizer is not available, approximate as `chars / 4`. Mark +findings using approximation as `(estimated)` in the evidence. + +## What used to live here + +`prompt.too-many-fewshots` (was `PW2`, fired at >3 few-shot examples) +was a taste-based threshold. Few-shot count alone is not a reliable +signal; `prompt.oversized` already captures the cost dimension. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md new file mode 100644 index 0000000000..818ceabdc7 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md @@ -0,0 +1,63 @@ +# Report template + +The exact Markdown layout written to `.copilot/perf-reports/scan.md`. +This file is **overwritten on every run**. Git provides history; the +skill does not maintain timestamped copies or `latest-*` mirrors. + +```markdown +# Agentic perf scan — {{ project_name }} + +Run: {{ utc_timestamp }} +Project: {{ relative_project_path }} + +## Inventory + +- AppHost: `{{ apphost_path }}` +- Agents: {{ agent_count }} ({{ agent_list }}) +- Handoff edges: {{ edge_count }} +- Tools (total): {{ tool_count }} +- Distinct models: {{ model_set }} +- OTel wired: {{ true | false }} + +## Summary + +- critical: {{ count }} +- warn: {{ count }} +- info: {{ count }} + +## Findings + +> **Slug format:** `.` — categories are +> `topology` (graph shape) · `tools` (per-agent tool list) · +> `history` (chat history strategy) · `prompt` (system prompt +> size/reuse) · `parallel` (concurrent invocations) · `otel` +> (instrumentation) · `model` (per-agent model id). Severities: +> `critical` (likely to break a flow or blow the budget) · +> `warn` (measurable cost/perf regression) · `info` (observation). + +### [critical] [`{{ check }}`] {{ title }} +- **File:** `{{ file }}:{{ line }}` +- **Evidence:** + ```csharp + {{ snippet }} + ``` +- **Why:** {{ paragraph }} +- **Next:** {{ action }} +- **Cross-ref:** {{ skill: ... | omit if none }} + +(... repeat per finding, ordered: critical → warn → info, then by `check` slug ...) + +## Next steps + +- If you want to fix the model assignments above, see rule #3 in `.github/copilot-instructions.md` (managed by `configure-agentic-perf-rules`). +- If you want to capture token/quality numbers before vs after, run + `setup-maf-evals`. +- If you do not yet have always-on rules to prevent regressions, run + `configure-agentic-perf-rules`. +``` + +## Empty-report contract + +If there are zero findings, the `## Findings` section still appears +(after the legend blockquote) with the literal text `_No findings._`. +The `## Summary` section shows zeros. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md new file mode 100644 index 0000000000..180c4abeee --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md @@ -0,0 +1,46 @@ +# Tool inventory checks + +Detect bloat and redundancy in the per-agent tool list. + +## Checks + +### `tools.duplicate` (warn) + +**Detect:** two or more tools across different agents with the same +description or near-identical signatures. + +**Why:** duplication forces the router LLM to disambiguate every turn +and inflates aggregate prompt size. + +**Next:** "Consolidate `` and `` into a single shared +tool exposed by both agents." + +### `tools.dead` (info) + +**Detect:** tools registered but never invoked in any code path or +call trace. Static check: tool name does not appear in any agent's +system prompt or instructions. + +**Why:** every registered tool costs prompt tokens whether it gets +called or not. + +**Next:** "Remove `` from ``'s tool list." + +### `tools.description-too-long` (warn) + +**Detect:** a tool's `[Description]` attribute or `description:` field +is longer than 200 characters. Measure after string concatenation — +multi-line `+` concatenations count as one description. + +**Why:** long descriptions multiply across agents that import the +tool. Most tools can be described in one sentence. + +**Next:** "Trim ``'s description from `` chars to ≤ 200; move +the detailed contract into XML docs on the parameters." + +## What used to live here + +`tools.too-many-per-agent` (was `TI1`) was a taste-based threshold +(>8 tools per agent) that fired on legitimate designs. The real +question — "is this tool earning its prompt-token weight" — is better +served by `tools.dead` and `tools.duplicate`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md new file mode 100644 index 0000000000..7f07e4e5fb --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md @@ -0,0 +1,40 @@ +# Topology checks + +Detect structural issues in the agent graph that drive latency or runaway loops. + +## Checks + +### `topology.cycle` (critical) + +**Detect:** any directed cycle in the static handoff graph. + +**Why:** cycles risk infinite loops if the loop-break condition is +LLM-judged. Even with a turn cap, a cycle burns budget on retries. + +**Next:** "Break the cycle `` → `` → `` by making ``'s exit +condition deterministic." + +### `topology.deep-single-leaf` (warn) + +**Detect:** graph that always ends at one agent but routes through 3+ +agents to reach it. + +**Why:** the intermediate hops are usually classification or routing +that could be one tool call. + +**Next:** "Move the routing logic into a tool on the entry agent and +call `` directly." + +## Out of scope here + +- Tool counts → see `tool-inventory-checks.md`. +- Per-agent model selection → see `model-assignment-checks.md`. + +## What used to live here + +`topology.agent-count` (was `T1`) and `topology.handoff-fanout` (was +`T2`) were removed in v0.2 — both were taste-based thresholds (>3 +agents, >2 handoff edges) that fired on legitimate designs as often +as on real bloat. If you want broad architectural feedback, run +`configure-agentic-perf-rules` so rule #1 (single-agent default) and +rule #2 (handoff justification) can guide the design at scaffold time. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-ModelLiterals.ps1 b/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-ModelLiterals.ps1 new file mode 100644 index 0000000000..9a8f232642 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-ModelLiterals.ps1 @@ -0,0 +1,112 @@ +<# +.SYNOPSIS + Deterministic model-assignment detector for scan-agentic-app-perf. + +.DESCRIPTION + Parses the C# sources of a .NET agentic app and reports where model ids are + declared: string literals (e.g. `"gpt-4o-mini"`), Aspire/Foundry enum refs + (e.g. `FoundryModel.OpenAI.Gpt4oMini`), and deployment/registration calls + (`AddDeployment(...)`, `deploymentName:`, `modelId:`). It separates AppHost + declarations (the canonical place for model ids) from agent-service `.cs` + files (where a hard-coded literal trips `model.hardcoded`), and counts the + distinct model ids to inform `model.same-default`. Automates the mechanical + extraction behind the `model.*` checks (references/model-assignment-checks.md). + + This is DETECTION only. It never assigns severity and never writes to the + scanned project. The skill re-opens each cited file (evidence gate) and fills + in severity/why/next per the finding schema, including the role-vs-model + judgment (planner/worker) that text matching cannot make. If parsing yields + nothing, the skill falls back to the reference-doc guidance (graceful + degradation). + + Works on any topology, including a file-based AppHost (a bare `.cs` with no + project/solution), because it parses source text directly. + +.PARAMETER Path + App root to scan. Defaults to the current directory. + +.PARAMETER Json + Emit machine-readable JSON (default is a short human summary). + +.EXAMPLE + ./Detect-ModelLiterals.ps1 -Path ./fixture -Json +#> +[CmdletBinding()] +param( + [string]$Path = ".", + [switch]$Json +) + +$ErrorActionPreference = "Stop" + +function Get-SourceFiles { + param([string]$Root) + if (Test-Path -LiteralPath $Root -PathType Leaf) { return @((Get-Item -LiteralPath $Root)) } + Get-ChildItem -LiteralPath $Root -Recurse -File -Filter *.cs -ErrorAction SilentlyContinue | + Where-Object { $_.FullName -notmatch '[\\/](obj|bin)[\\/]' } +} + +# Well-known model-id shapes as string literals (kept broad but anchored). +$literalRx = '"(?(?:gpt-|o1|o3|o4|text-embedding-|claude-|gemini-|phi-|llama|mistral|deepseek)[A-Za-z0-9._:-]*)"' +# Aspire/Foundry strongly-typed model refs. +$enumRx = '(?FoundryModel\.[A-Za-z0-9_.]+)' +# Deployment / registration call sites where a model id is bound. +$deployRx = '(?:AddDeployment|WithDeployment)\s*\(|deploymentName\s*:|modelId\s*:' + +$hits = New-Object System.Collections.Generic.List[object] +$files = @(Get-SourceFiles -Root $Path) + +foreach ($f in $files) { + $rel = $f.FullName + try { $rel = (Resolve-Path -LiteralPath $f.FullName -Relative -ErrorAction Stop) } catch {} + $isAppHost = ($f.FullName -match '\.AppHost' ) -or ($f.Directory.Name -match 'AppHost') + $lines = Get-Content -LiteralPath $f.FullName -ErrorAction SilentlyContinue + for ($i = 0; $i -lt $lines.Count; $i++) { + $line = $lines[$i] + foreach ($m in [regex]::Matches($line, $literalRx)) { + $hits.Add([pscustomobject]@{ id = $m.Groups['id'].Value; form = 'literal'; appHost = $isAppHost; file = $rel; line = ($i + 1); text = $line.Trim() }) + } + foreach ($m in [regex]::Matches($line, $enumRx)) { + $hits.Add([pscustomobject]@{ id = $m.Groups['id'].Value; form = 'enum'; appHost = $isAppHost; file = $rel; line = ($i + 1); text = $line.Trim() }) + } + if ([regex]::IsMatch($line, $deployRx)) { + $hits.Add([pscustomobject]@{ id = $null; form = 'deployment-call'; appHost = $isAppHost; file = $rel; line = ($i + 1); text = $line.Trim() }) + } + } +} + +$hits = [object[]]$hits.ToArray() +$distinctIds = @($hits | Where-Object { $_.id } | Select-Object -ExpandProperty id -Unique) +# Literals living in a non-AppHost source file are what model.hardcoded flags. +$hardcodedInService = @($hits | Where-Object { $_.form -eq 'literal' -and -not $_.appHost }) + +$flags = [ordered]@{ + # >=2 model ids but only one distinct value hints at model.same-default; + # the skill confirms agent count and roles before asserting. + 'model.same-default' = ($distinctIds.Count -eq 1 -and @($hits | Where-Object { $_.id }).Count -ge 2) + 'model.hardcoded' = ($hardcodedInService.Count -gt 0) +} + +$result = [ordered]@{ + ok = $true + scanned = @($files).Count + distinctIds = $distinctIds + hits = $hits + flags = $flags + notes = "Role-vs-model judgment (planner/worker, reasoning-on-deterministic, cheap-on-planner) is NOT decided here; confirm each agent's role from its prompt/tools per the finding schema. AppHost AddDeployment(...) is the canonical model-id location and does not itself trip model.hardcoded." +} + +if ($Json) { + [pscustomobject]$result | ConvertTo-Json -Depth 6 +} else { + "=== Model assignments ===" + " Files scanned : $($result.scanned)" + " Distinct model ids: $($distinctIds.Count) [$($distinctIds -join ', ')]" + foreach ($h in $hits) { + $id = if ($h.id) { $h.id } else { "(deployment call)" } + $scope = if ($h.appHost) { "apphost" } else { "service" } + " $($h.form)/$scope : $id ($($h.file):$($h.line))" + } + "" + foreach ($k in $flags.Keys) { if ($flags[$k]) { " flag: $k" } } +} diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-OtelCoverage.ps1 b/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-OtelCoverage.ps1 new file mode 100644 index 0000000000..8258d66348 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-OtelCoverage.ps1 @@ -0,0 +1,112 @@ +<# +.SYNOPSIS + Deterministic OpenTelemetry-coverage detector for scan-agentic-app-perf. + +.DESCRIPTION + Parses the C# / config sources of a .NET agentic app and reports which + observability primitives are present: Aspire service defaults, the + OpenTelemetry SDK, an OTLP exporter, tracing/metrics builders, `gen_ai.*` + token telemetry, and an Aspire dashboard. Automates the mechanical + presence-detection behind the `otel.*` checks (references/otel-coverage-checks.md) + so the calling skill spends tokens on judgment (severity, the `why`/`next` + narrative) rather than on grepping every file. + + This is DETECTION only. It never assigns severity and never writes to the + scanned project. The skill re-opens each cited file (evidence gate) and fills + in severity/why/next per the finding schema. If parsing yields nothing, the + skill falls back to the reference-doc guidance (graceful degradation). + + Works on any topology, including a file-based AppHost (a bare `.cs` with no + project/solution), because it parses source text directly. + +.PARAMETER Path + App root to scan. Defaults to the current directory. + +.PARAMETER Json + Emit machine-readable JSON (default is a short human summary). + +.EXAMPLE + ./Detect-OtelCoverage.ps1 -Path ./fixture -Json +#> +[CmdletBinding()] +param( + [string]$Path = ".", + [switch]$Json +) + +$ErrorActionPreference = "Stop" + +function Get-ScanFiles { + param([string]$Root) + if (Test-Path -LiteralPath $Root -PathType Leaf) { return @((Get-Item -LiteralPath $Root)) } + Get-ChildItem -LiteralPath $Root -Recurse -File -ErrorAction SilentlyContinue | + Where-Object { + $_.FullName -notmatch '[\\/](obj|bin)[\\/]' -and + ($_.Extension -eq '.cs' -or $_.Name -match '^appsettings.*\.json$') + } +} + +# signal-name -> regex. First hit records file:line as evidence. +$signals = [ordered]@{ + serviceDefaults = 'AddServiceDefaults\s*\(' + addOpenTelemetry = 'AddOpenTelemetry\s*\(' + otlpExporter = '(?:Add|Use)OtlpExporter\s*\(' + withTracing = 'WithTracing\s*\(' + withMetrics = 'WithMetrics\s*\(' + genAiTokens = 'gen_ai\.usage\.(?:input|output)_tokens' + aspireDashboard = 'Aspire\.Hosting\.Dashboard|Dashboard:OtlpEndpointUrl' + sensitiveData = 'EnableSensitiveData' +} + +$files = @(Get-ScanFiles -Root $Path) +$evidence = [ordered]@{} +foreach ($k in $signals.Keys) { $evidence[$k] = $null } + +foreach ($f in $files) { + $rel = $f.FullName + try { $rel = (Resolve-Path -LiteralPath $f.FullName -Relative -ErrorAction Stop) } catch {} + $lines = Get-Content -LiteralPath $f.FullName -ErrorAction SilentlyContinue + for ($i = 0; $i -lt $lines.Count; $i++) { + $line = $lines[$i] + foreach ($k in $signals.Keys) { + if ($null -eq $evidence[$k] -and [regex]::IsMatch($line, $signals[$k])) { + $evidence[$k] = [pscustomobject]@{ file = $rel; line = ($i + 1); text = $line.Trim() } + } + } + } +} + +$present = [ordered]@{} +foreach ($k in $signals.Keys) { $present[$k] = ($null -ne $evidence[$k]) } + +# Roll up the reference-doc checks. otel.missing-sdk fires only when NEITHER +# Aspire service defaults NOR a raw OTel SDK registration is present. +$hasAnySdk = $present.serviceDefaults -or $present.addOpenTelemetry +$flags = [ordered]@{ + 'otel.missing-sdk' = (-not $hasAnySdk) + 'otel.no-aspire-dashboard' = (-not $present.aspireDashboard) + 'otel.no-token-cost' = (-not $present.genAiTokens) +} + +$result = [ordered]@{ + ok = $true + scanned = @($files).Count + present = $present + evidence = $evidence + flags = $flags + notes = "Presence detection is best-effort (text match). `gen_ai.*` tags are emitted automatically by Microsoft.Extensions.AI even without an explicit literal, so confirm the exporter wiring before asserting otel.no-token-cost." +} + +if ($Json) { + [pscustomobject]$result | ConvertTo-Json -Depth 6 +} else { + "=== OTel coverage ===" + " Files scanned : $($result.scanned)" + foreach ($k in $signals.Keys) { + $mark = if ($present[$k]) { "yes" } else { "no " } + $loc = if ($evidence[$k]) { " ($($evidence[$k].file):$($evidence[$k].line))" } else { "" } + " {0,-16} : {1}{2}" -f $k, $mark, $loc + } + "" + foreach ($k in $flags.Keys) { if ($flags[$k]) { " flag: $k" } } +} diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-Topology.ps1 b/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-Topology.ps1 new file mode 100644 index 0000000000..28305d29a8 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/scripts/Detect-Topology.ps1 @@ -0,0 +1,135 @@ +<# +.SYNOPSIS + Deterministic topology detector for scan-agentic-app-perf. + +.DESCRIPTION + Parses the C# sources of a .NET agentic app and emits a best-effort agent + handoff graph: agent nodes, handoff edges, orphaned agents, and per-file + fan-out. Automates the mechanical extraction behind the `topology.*` checks + (references/topology-checks.md) so the graph is consistent run-to-run and the + calling skill spends tokens on judgment (cycle / deep-single-leaf severity, + the `why`/`next` narrative) rather than on reading every source file. + + This is DETECTION only. It never assigns severity and never writes to the + scanned project. The skill re-opens each cited file (evidence gate) and fills + in severity/why/next per the finding schema. If parsing yields nothing, the + skill falls back to the reference-doc guidance (graceful degradation). + + Works on any topology, including a file-based AppHost (a bare `.cs` with no + project/solution), because it parses source text directly. + +.PARAMETER Path + App root to scan (directory containing the solution / AppHost / agent + sources, or a single file-based AppHost `.cs`). Defaults to the current dir. + +.PARAMETER Json + Emit machine-readable JSON (default is a short human summary). + +.EXAMPLE + ./Detect-Topology.ps1 -Path ./fixture -Json +#> +[CmdletBinding()] +param( + [string]$Path = ".", + [switch]$Json +) + +$ErrorActionPreference = "Stop" + +function Get-SourceFiles { + param([string]$Root) + if (Test-Path -LiteralPath $Root -PathType Leaf) { return @((Get-Item -LiteralPath $Root)) } + Get-ChildItem -LiteralPath $Root -Recurse -File -Filter *.cs -ErrorAction SilentlyContinue | + Where-Object { $_.FullName -notmatch '[\\/](obj|bin)[\\/]' } +} + +# Node-declaration patterns. Each captures the agent's name/alias where present. +$nodePatterns = @( + # Aspire AppHost project registration: AddProject("alias") + @{ kind = 'project'; rx = 'AddProject<[^>]+>\s*\(\s*"(?[^"]+)"' }, + # MAF host agent registration: AddAIAgent("name", ...) + @{ kind = 'agent'; rx = 'AddAIAgent\s*\(\s*"(?[^"]+)"' }, + # AsAIAgent(name: "name", ...) + @{ kind = 'agent'; rx = 'As(?:AI)?Agent\s*\(\s*name\s*:\s*"(?[^"]+)"' }, + # CreateAIAgent(name: "name") + @{ kind = 'agent'; rx = 'CreateAIAgent\s*\(\s*(?:name\s*:\s*)?"(?[^"]+)"' }, + # new ChatClientAgent(... ) — name not always present; recorded per-file + @{ kind = 'agent'; rx = 'new\s+ChatClientAgent\s*\(' } +) + +# Handoff-edge patterns: AddHandoff("target") / WithHandoff("target") +$edgePatterns = @( + 'Add(?:Handoff|Edge)\s*\(\s*"(?[^"]+)"', + 'WithHandoff\s*\(\s*"(?[^"]+)"' +) + +$files = @(Get-SourceFiles -Root $Path) +$nodes = New-Object System.Collections.Generic.List[object] +$edges = New-Object System.Collections.Generic.List[object] +$fanoutByFile = @{} + +foreach ($f in $files) { + $rel = $f.FullName + try { $rel = (Resolve-Path -LiteralPath $f.FullName -Relative -ErrorAction Stop) } catch {} + $lines = Get-Content -LiteralPath $f.FullName -ErrorAction SilentlyContinue + for ($i = 0; $i -lt $lines.Count; $i++) { + $line = $lines[$i] + foreach ($np in $nodePatterns) { + foreach ($m in [regex]::Matches($line, $np.rx)) { + $name = if ($m.Groups['name'].Success) { $m.Groups['name'].Value } else { [System.IO.Path]::GetFileNameWithoutExtension($f.Name) } + $nodes.Add([pscustomobject]@{ name = $name; kind = $np.kind; file = $rel; line = ($i + 1) }) + } + } + foreach ($ep in $edgePatterns) { + foreach ($m in [regex]::Matches($line, $ep)) { + $src = [System.IO.Path]::GetFileNameWithoutExtension($f.Directory.Name) # best-effort: source project/dir + if ([string]::IsNullOrWhiteSpace($src)) { $src = [System.IO.Path]::GetFileNameWithoutExtension($f.Name) } + $edges.Add([pscustomobject]@{ source = $src; target = $m.Groups['target'].Value; file = $rel; line = ($i + 1) }) + if (-not $fanoutByFile.ContainsKey($rel)) { $fanoutByFile[$rel] = 0 } + $fanoutByFile[$rel]++ + } + } + } +} + +# Materialize the generic Lists as arrays. PowerShell's array-subexpression +# @(...) .Count throws "Argument types do not match" on a List[object] of +# PSCustomObjects, so cast once and treat everything as arrays downstream. +$nodes = [object[]]$nodes.ToArray() +$edges = [object[]]$edges.ToArray() + +$nodeNames = @($nodes | Select-Object -ExpandProperty name -Unique) +$edgeTargets = @($edges | Select-Object -ExpandProperty target -Unique) +$edgeSources = @($edges | Select-Object -ExpandProperty source -Unique) +$touched = @($edgeTargets + $edgeSources | Select-Object -Unique) +$orphans = @($nodeNames | Where-Object { $touched -notcontains $_ }) +$maxFanout = 0 +if ($fanoutByFile.Values.Count -gt 0) { $maxFanout = ($fanoutByFile.Values | Measure-Object -Maximum).Maximum } + +$metrics = [ordered]@{ + agentCount = @($nodeNames).Count + edgeCount = @($edges).Count + orphanCount = @($orphans).Count + maxFanout = [int]$maxFanout +} +$result = [ordered]@{ + ok = $true + scanned = @($files).Count + metrics = $metrics + nodes = @($nodes) + edges = @($edges) + orphans = @($orphans) + notes = "Edge source is inferred from the defining file/project (best-effort); confirm direction against the cited file before asserting a cycle or deep chain." +} + +if ($Json) { + [pscustomobject]$result | ConvertTo-Json -Depth 6 +} else { + "=== Topology ===" + " Files scanned : $($result.scanned)" + " Agents : $($metrics.agentCount) [$($nodeNames -join ', ')]" + " Handoff edges : $($metrics.edgeCount)" + " Orphan agents : $($metrics.orphanCount) [$($orphans -join ', ')]" + " Max fan-out : $($metrics.maxFanout) (handoffs in one file)" + foreach ($e in $edges) { " edge: $($e.source) -> $($e.target) ($($e.file):$($e.line))" } +} diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md new file mode 100644 index 0000000000..2c9c33e732 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -0,0 +1,446 @@ +--- +name: setup-maf-evals +description: | + Scaffold a .NET agentic-app (MAF; Aspire/Foundry optional) evaluation harness — `.Evals.Tests` MSTest project wired to the GA Microsoft.Extensions.AI.Evaluation (MEAI) reporting pipeline. WHEN: "set up evals", "add evaluation harness/coverage", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators"; or troubleshooting — "why are my Quality columns erroring/empty", "reasoning model breaks evals / max_tokens vs max_completion_tokens", "which evaluators fail my stylistic/summarizer agent". Categories: **NLP** (BLEU/GLEU/F1, no key), **Quality** (LLM-as-judge), **Safety** (content-harm via Foundry). Auto-installs `aieval`, detects `IChatClient`, generates a factory (`EVAL_USE_REAL_AGENT=1`), emits an HTML report; optional PR workflow. Topologies: Aspire/console/ASP.NET/worker. NOT-WHEN: one-shot audit (scan-agentic-app-perf), install rules (configure-agentic-perf-rules), reasoning-model questions unrelated to evals, or running an existing suite (dotnet test).--- + +# setup-maf-evals + +Scaffold an `.Evals.Tests` MSTest project that measures latency, +token usage, cost, quality, and safety on every change to a .NET +agentic app — and produces the proper Microsoft.Extensions.AI.Evaluation +HTML report by default. + +## When to Use + +- The user asks to "set up evals", "add an evaluation harness", "wire + up MEAI evaluation", or "measure my agent's quality". +- The user wants to **validate quality** after a model change ("does + swapping gpt-4o → gpt-4o-mini regress responses?") and needs a + reproducible baseline. +- The user wants to **compare** model assignments side-by-side + ("compare gpt-4o vs gpt-4o-mini across my agents"). +- The user wants to **add safety evaluators** (Hate/Violence/SelfHarm/ + Sexual via Azure AI Foundry) to an existing agent. +- The user wants a recurring eval run wired into CI (the optional + GitHub Actions workflow). +- The user is **troubleshooting an eval setup** — "why are my Quality + columns erroring or empty?", "my reasoning model rejects `max_tokens` + in evals", "which evaluators will systematically fail my stylistic / + summarizer agent?". The skill owns these answers; see + `references/common-pitfalls.md`. + +## When Not to Use + +- The user wants a **one-shot audit** of existing code without scaffolding + a new test project — use `scan-agentic-app-perf` instead. +- The user wants **always-on perf guard-rails** in the project's + agent-instructions file — use `configure-agentic-perf-rules`. +- The project is not a .NET agentic app, or the user is not using + `Microsoft.Extensions.AI` / `Microsoft.Agents.AI`. +- The user explicitly does not want an MSTest dependency and is not + willing to use the opt-in `--shape console` runner. +- The user asks a **general model-behavior question unrelated to + evaluation** (e.g. how reasoning models work in production), or just + wants to **run an existing eval suite** (`dotnet test`) — no + scaffolding or eval-specific guidance is needed. + +## Supported topologies + +The skill targets any .NET project using Microsoft Agent Framework +(`Microsoft.Agents.AI`), regardless of how it's hosted: + +- **Aspire AppHost** (`*.AppHost.csproj` orchestrating agent services) — the + most common shape; the generated `AgentChatClientFactory` mirrors the + AppHost's connection-string-driven `IChatClient` registration. +- **File-based AppHost** (.NET 10) — a single `apphost.cs` (with + `#:sdk Aspire.AppHost.Sdk` / `#:package` directives, run via + `dotnet run apphost.cs`, no `.csproj`/`.sln`). Detection reads the `.cs` + source directly, and the Evals project references the agent service + project(s) it can find; if the host itself is file-based with no referenced + service project, the factory falls back to a direct client registration. +- **Plain console / worker service** — `Microsoft.Agents.AI` registered + directly against an OpenAI or Foundry client without an Aspire AppHost. + The factory mirrors the app's direct registration (e.g. `AddOpenAIClient` + or `AddAzureOpenAIChatClient` at the host level). +- **ASP.NET Core minimal API** — agents registered via the same DI patterns. + +`IChatClient` detection (see `references/ichatclient-detection.md`) works +the same way across all topologies; only the AppHost-vs-Program.cs search +locations differ. + +## Workflow + +### 1. Discover the target app + +Detect: + +- Solution file (`*.sln` / `*.slnx`) +- AppHost project name (`*.AppHost.csproj`), **or** a file-based AppHost — a bare + `.cs` (commonly `apphost.cs`) with `#:sdk Aspire.AppHost.Sdk` / `#:package` + directives and a `DistributedApplication.CreateBuilder` call and no + `.csproj`/`.sln` +- Agent service projects +- Existing test / eval projects (avoid clobbering) +- **`IChatClient` registration** (scan AppHost + agent projects for + `AddChatClient` / `AddAzureOpenAIChatClient` / `AddOllamaChatClient` + / `AddOpenAIChatClient` and any explicit `services.AddSingleton` or + Foundry deployment alias references). See `references/ichatclient-detection.md`. + +If no agentic app is detected, abort. If a `*.Evals.Tests` project already +exists, switch to update mode (see step 1a). + +### 1a. Update mode (when `*.Evals.Tests` project already exists) + +File classes: + +| Class | Files | Behavior on update | +|------------------|------------------------------------------------------------------------|----------------------------| +| **infra** | `*.Evals.Tests.csproj`, `dotnet-tools.json`, `Reporting/*`, `Wire/*`, test class skeletons | update to current template; merge package refs; create if missing | +| **user data** | `Quality/rubric.md`, `Quality/golden.json`, `Compare/matrix.json`, `Telemetry/inputs.json`, `Telemetry/prices.json`, `quality.thresholds.json`, `.github/workflows/evals.yml` | never overwrite; create if missing | +| **generated** | `.copilot/perf-reports/evals/` | regenerate freely | + +If an existing infra file differs from the current template, **update it to the +current template** — that is the update the user asked for. Merge in any user-added +package references, report what changed in the chat output, and never touch the +**user data** files above. Only user data is sacrosanct; infra is skill-owned +scaffolding, and Git preserves history if the user wants to review or revert. + +`golden.json` has a `schema_version` field. If the detected version is +older than the current template, offer to migrate (additive only — +preserves existing rows, adds the new fields as nullable). + +### 2. Resolve scope (proceed with defaults — do not block) + +This is an **autonomous scaffold**. Do **not** stop and wait for the user +to answer before scaffolding. Print a one-line detection summary, apply +the defaults below, scaffold immediately, then state any assumptions you +made so the user can adjust. Honor any modes/categories the user *did* +specify; otherwise use the defaults. The only action that requires explicit +confirmation **first** is the genuinely destructive/irreversible one called out +elsewhere — the Aspire dashboard panel's AppHost edit. Updating infra files in +update mode (step 1a) is the expected, non-destructive action — just do it +(user data is preserved and Git keeps history). Defaults: + +1. **Project shape** (default: MSTest). Alternative: console runner + (legacy v1 shape) — only emit if user explicitly asks for it. +2. **Evaluator categories to enable.** Defaults shown; user can override. + + | Category | Evaluators | Cost | Needs | Default | + |------|-----------|------|-------|---------| + | **Quality** *(headline)* | Relevance, Coherence, Fluency, Completeness, Equivalence, Groundedness; agent: IntentResolution, TaskAdherence, ToolCallAccuracy | per-call judge tokens | real `IChatClient` + `EVAL_USE_REAL_JUDGE=1` | **ON** (stubbed until judge wired) | + | NLP (zero-config sanity) | BLEU, GLEU, F1, Words | free | reference responses in golden.json | ON | + | Safety | `ContentHarmEvaluator` (Hate+SelfHarm+Violence+Sexual single-shot), ProtectedMaterial, IndirectAttack, CodeVulnerability, UngroundedAttributes, GroundednessPro | Foundry evaluation service charges | Azure AI Foundry endpoint + `EVAL_USE_FOUNDRY_SAFETY=1` | **OFF** by default — scaffold only when the user explicitly asks for safety evaluators (do not block to ask) | + + Frame Quality as the headline evaluation; NLP is the zero-config + first-run experience that emits a real `report.html` before any + creds exist; Safety is the opt-in for production-bound apps. + +3. **IChatClient detection result.** Show what was detected (e.g., "Found + `AddAzureOpenAIChatClient` in `AppHost.cs:41` with deployment alias `chat`"). + State the detected registration and proceed; if detection failed, generate + a stub factory the user will fill in (do not block to confirm). +4. **Run modes to scaffold.** Telemetry (default ON) and Quality (default + ON). **Compare mode is opt-in and OFF by default** — scaffold it only + when the user explicitly asks for a side-by-side matrix.json run. Compare + adds the largest scaffold (extra runner + matrix.json + delta-table + generator) and is the least commonly used surface. Do not block to ask. +5. **Optional add-ons:** Aspire dashboard panel, GitHub Actions workflow. + +### 3. Scaffold the project + +> **Execution discipline — the scaffold is the deliverable (files on disk, not a plan).** +> - **Read `references/default-scaffold.md` ONCE, then `create` every file it +> lists.** That single doc holds the complete, copy-pasteable bodies for all +> default-mode files (Telemetry, Quality, NLP, Reporting, Wire). You do **not** +> need to open the per-topic refs (`telemetry-capture.md`, `quality-modes.md`, +> `ichatclient-detection.md`, `evaluators-catalog.md`, `metrics-glossary.md`) +> to scaffold — reading many refs first exhausts the turn budget before +> anything is written, the top cause of an empty scaffold. +> - **Only read an opt-in mode's doc when that mode is enabled.** Skip +> `compare-mode.md`, `safety-mode.md`, `ci-workflow.md`, and +> `aspire-dashboard-panel.md` entirely unless the user opted in. +> - **Files first, `dotnet` later — success is files on disk, not a green build.** +> Every file in `default-scaffold.md` is written with `create` and needs no +> network. The `dotnet add package` / `tool install` steps below only stamp +> versions on top of files that already exist; a slow or offline SDK must never +> leave you with nothing on disk. +> - **Do NOT run `dotnet build`, `dotnet test`, `dotnet run`, or `dotnet restore` +> as part of scaffolding.** They are slow, network-bound, and not required — +> the scaffold is complete when the files exist, and the version-less +> `` entries already satisfy the package set. Running them +> proactively to "verify" burns the whole turn budget (often 600s+) and is the +> top cause of a scaffold that never finishes. Instead, **hand the build/test +> commands to the user** as their next step (step 10). Only run them yourself +> if the user explicitly asks you to verify the build. +> - **Do not print the chat summary (step 11) or end the turn until every file in +> `references/default-scaffold.md` exists on disk.** + +Order matters: create the shell, **overlay every eval file (the deliverable)**, +then add packages and wire the solution. `references/default-scaffold.md` holds +the complete body of every default-mode file (read it once); it and +`references/project-template.md` are the source of truth for the file tree and +package **set** (never for pinned versions). + +1. **Create the base test project shell** (default MSTest; skip only when the + user asked for `--shape console`). The template is local (needs no network) + and emits a current `.csproj` + test-SDK / Microsoft.Testing.Platform wiring + (on .NET 10, the MTP-native `MSTest` metapackage) plus a placeholder test: + + ```pwsh + dotnet new mstest -n .Evals.Tests -o .Evals.Tests + ``` + + Let the template own the test-SDK `` lines — do **not** + hand-write them. Delete the placeholder `Test1.cs` / `UnitTest1.cs`. + +2. **Overlay the eval files now — this is the deliverable, and it needs no + network.** Read `references/default-scaffold.md` once; it contains the + complete body of every default-mode file. Using the `create` tool, emit each + file it lists (in the order given): `Reporting/{Tier.cs, WordCountEvaluator.cs, + ReportingConfig.cs, MetricsGlossary.cs, AievalReport.cs, Thresholds.cs}`, + `Wire/{StubChatClient.cs, AgentChatClientFactory.cs, Wire.cs}`, + `Telemetry/{TelemetrySupport.cs, TelemetryTests.cs, inputs.json, prices.json}`, + `Quality/{QualitySupport.cs, QualityTests.cs, rubric.md, golden.json}`, + `quality.thresholds.json`, and `GlobalUsings.cs`. Emit + `Compare/*`, `Safety/SafetyTests.cs`, and `.github/workflows/evals.yml` + **only** for opted-in modes (steps 2 #4, 7, 9 — read their mode docs then). + Reconcile the `.csproj` per `default-scaffold.md`: + `net10.0`, the + `` data-file item, + and a `` to each detected agent service project. Also + pre-list the eval + hosting package **set** as **version-less** + `` entries (no `Version` attribute — step 3's + `dotnet add package` stamps the resolved version). This keeps the correct + package set on disk even before restore runs, without authoring a version + literal. Append the `.gitignore` entries `.copilot/perf-reports/evals/` and + `.Evals.Tests/_store/`. + + **At this point the project is complete on disk** — every `file_exists` and + `csproj`-contains expectation is already satisfied. The remaining `dotnet` + steps (3–4, 10) only stamp versions and validate. + +3. **(Deferrable, network) Add the eval + hosting packages** — no hand-pinned + versions; let NuGet resolve current. GA packages take the latest stable; the + still-preview evaluators use `--prerelease`. See `references/project-template.md` + for the version policy and the one floor constraint. **Skip this step when the + SDK is slow or offline** — the version-less `` entries from + step 3.2 already carry the correct package set; `dotnet add package` simply + stamps each resolved version in place, and the user can run it later. + + ```pwsh + cd .Evals.Tests + # GA — latest stable: + dotnet add package Microsoft.Extensions.AI + dotnet add package Microsoft.Extensions.AI.Evaluation + dotnet add package Microsoft.Extensions.AI.Evaluation.Quality + dotnet add package Microsoft.Extensions.AI.Evaluation.Reporting + # Hosting/config — latest stable (>= 10.0.1 automatically, avoiding the + # NU1605 downgrade from Microsoft.Agents.AI.Hosting): + dotnet add package Microsoft.Extensions.Hosting + dotnet add package Microsoft.Extensions.Configuration.Json + dotnet add package Microsoft.Extensions.Configuration.UserSecrets + # Preview — latest prerelease: + dotnet add package Microsoft.Extensions.AI.Evaluation.NLP --prerelease + # Safety — only if the user opted in (step 2): + # dotnet add package Microsoft.Extensions.AI.Evaluation.Safety --prerelease + ``` + + `dotnet add package` writes the resolved version into the `.csproj`; the skill + never authors a version literal. + +4. **(Deferrable, network) Generate the `aieval` tool manifest** (also unpinned — + do not hand-write a version), then wire the solution + restore. **Skip when + offline** — `AievalReport` already degrades gracefully (it wraps the `aieval` + invocation in try/catch), so the scaffold and its telemetry/glossary reports + still work without the tool; the user restores it later: + + ```pwsh + dotnet new tool-manifest # if none exists yet + dotnet tool install microsoft.extensions.ai.evaluation.console # latest; provides `aieval` + # Add to the solution when one exists; a file-based AppHost may have none — + # skip this line in that case (the ProjectReference is enough to build). + dotnet sln add .Evals.Tests/.Evals.Tests.csproj + dotnet tool restore # restores aieval from the generated manifest + ``` + +### 4. Wire telemetry mode + +See `references/telemetry-capture.md`. Default ON. Captures latency, +input/output tokens, and price-table-driven cost via a delegating +`IChatClient`. Writes `telemetry.{md,json,junit.xml}` next to +`report.html`. **Not** the MEAI HTML report — that's quality mode's job. + +### 5. Wire quality mode + +See `references/quality-modes.md`. Default ON. The **only** runner that +produces `report.html`. Uses `DiskBasedReportingConfiguration` + +`ScenarioRun.EvaluateAsync` + `[AssemblyCleanup]` invokes +`dotnet tool run aieval report`. Stub tier registers the 4 NLP +evaluators; judge tier (`EVAL_USE_REAL_JUDGE=1`) adds the LLM-as-judge +evaluators from `references/evaluators-catalog.md`. + +### 6. Wire compare mode (opt-in) + +See `references/compare-mode.md`. **Default OFF.** Only scaffold when +the user opted in at step 2 (#4). When enabled, reads `matrix.json`; +each entry runs through the **same** `ReportingConfiguration` with a +distinct `executionName`, so `aieval report` aggregates the comparison +columns into a single HTML view. Also writes a `compare.md` delta +table. + +### 7. Wire safety mode (opt-in) + +See `references/safety-mode.md`. **Default OFF.** When enabled, adds +`Microsoft.Extensions.AI.Evaluation.Safety` and emits `SafetyTests.cs` +with `ContentHarmEvaluator` (single-shot 4-metric bundle), plus +ProtectedMaterial / IndirectAttack / CodeVulnerability / +UngroundedAttributes. Skipped at runtime via `Assert.Inconclusive` +when `EVAL_USE_FOUNDRY_SAFETY` is unset — never fails the build for +missing creds. + +### 8. Optional Aspire panel (apply mode) + +See `references/aspire-dashboard-panel.md`. Default OFF; modifies +the AppHost project. To enable, user must say "wire the panel" / +equivalent. Always show a unified diff + ask for confirmation before +writing AppHost edits. + +### 9. Optional CI workflow (opt-in) + +See `references/ci-workflow.md`. Default OFF. Emits +`.github/workflows/evals.yml` that runs `dotnet test` on every PR, +auto-detects tier from repo secrets (`AZURE_OPENAI_ENDPOINT` → +judge; `AZURE_AI_FOUNDRY_ENDPOINT` → safety), and uploads +`report.html` as a workflow artifact. + +### 10. Validation (hand to the user — do not run inline) + +The scaffold is validated by the files existing on disk, not by a build. **Do +not run `dotnet build` / `dotnet test` / `dotnet run` as part of scaffolding** — +they are slow and network-bound and the project is already complete. Instead, +**give the user the commands to run** and describe what a passing run looks like. +Only run them yourself if the user explicitly asks you to verify the build; if +you do, and the SDK or network is unavailable, report that they were skipped and +**never delete or withhold the scaffolded files** because a build couldn't run. + +Commands to hand to the user: + +- `dotnet build .Evals.Tests.csproj` — should exit 0. +- `dotnet test .Evals.Tests.csproj` — should exit 0 in stub tier (no creds + needed). + - Stub tier emits a `report.html` with **≥ 4 distinct metric columns** + (Words, BLEU, GLEU, F1) across all scenarios in golden.json. + - All scenarios produce non-null metric values (no "—" placeholders). +- With `EVAL_USE_REAL_JUDGE=1` and an `IChatClient` wired, `dotnet test` + additionally produces ≥ 3 Quality metrics (Relevance, Coherence, Fluency). + +### 11. Surface in chat + +Lead with **Quality** as the headline evaluation; frame NLP as the +zero-config sanity check and Safety/Compare as additions. + +1. **Quality (headline).** State whether the judge is wired: + - *Stubbed* — "Quality scaffolded; judge will run once you wire + `EVAL_USE_REAL_JUDGE=1` + a chat endpoint. The next block tells + you how." + - *Live* — "Quality judge active against ``; report + shows Relevance / Coherence / Fluency / Completeness / Equivalence." + - Caveat the user **must** know up-front: *"The built-in Quality + rubrics are generic. Agents with deliberate stylistic constraints + (brevity, persona, format adherence) will score low on + Completeness / Equivalence even when working as designed. See + `references/common-pitfalls.md#tuning-quality-for-stylistic-agents` + for the per-app override pattern."* +2. **Promoting Quality to the judge tier.** If the app's `IChatClient` + reads from a connection string (Aspire pattern), include the exact + two commands: + + ```pwsh + dotnet user-secrets init --project .Evals.Tests + dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests + # then: $env:EVAL_USE_REAL_AGENT="1"; $env:EVAL_USE_REAL_JUDGE="1"; dotnet test + ``` + + Note the **three** endpoint gotchas: (a) hostname strips dashes on + some resources, (b) key auth is often disabled → drop the `Key=` + segment and rely on `DefaultAzureCredential`, (c) **the judge + deployment must be a non-reasoning model** (gpt-4o / gpt-4o-mini / + gpt-4-turbo) — reasoning models (gpt-5*, o-series) reject `max_tokens` + and MEAI silently records every Quality metric as an error row. If the + production model is a reasoning one, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge points + at a different deployment than the agent. Endpoint specifics in + `references/ichatclient-detection.md`; the full reasoning-model + footgun — including `max_completion_tokens` and the durable SDK-migration + fix — is the canonical writeup in `references/common-pitfalls.md`. +3. **NLP (zero-config sanity).** "`report.html` already populates with + Words / BLEU / GLEU / F1 from `golden.json` without any creds — run + `dotnet test` now to see it." +4. **Additional categories you can wire.** List Safety + (`EVAL_USE_FOUNDRY_SAFETY=1`, opt-in scaffold) and Compare (re-run + the skill with `compare: true` if it wasn't opted in originally) + as one-line bullets — not in the main flow. +5. **Paths.** Project path, `report.html` path, **glossary path** + (`metrics-glossary.md` co-located with `report.html`), persistent + `_store/` path. +6. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, + and the IChatClient detection result so the user knows what was + auto-wired. +7. **Follow-up recommendation.** "Re-run after swapping a model per + rule #3 of `configure-agentic-perf-rules` to confirm no quality + regression." +8. **Cache payoff.** "First `dotnet test` populates `_store/cache/` and + takes the full ~60s. Every subsequent run against unchanged inputs + reuses cached agent + judge responses — typically ~5s with zero LLM + cost. The Diagnostic Data section of `report.html` shows per-call + Hit/Miss. To force a fresh run, delete `_store/cache/` or change the + rubric / golden inputs." + +Also link `references/evaluators-catalog.md` and +`references/metrics-glossary.md` so the user can see what each metric +means. + +## Common pitfalls + +Full detail in `references/common-pitfalls.md`. The three that most often break +an eval run are answered inline here so a troubleshooting question doesn't +require digging into the reference: + +- **Reasoning models reject `max_tokens`.** A reasoning model (o1/o3/o-series, + gpt-5*) used as the **judge** rejects `max_tokens` and requires + `max_completion_tokens`; otherwise MEAI records every Quality metric as an + error row. Fix: point the judge at a non-reasoning deployment via + `EVAL_JUDGE_DEPLOYMENT_NAME`, or upgrade the client so it emits + `max_completion_tokens`. See + `references/common-pitfalls.md#clients-agent-vs-judge-vs-stub`. +- **Stylistic agents fail completeness-style evaluators.** Agents that produce + deliberate stylistic prose (brevity, persona, format adherence) score low on + `CompletenessEvaluator` and `EquivalenceEvaluator` even when working as + designed, because those evaluators expect factual overlap with a reference. + Fix: drop or override them in the rubric for stylistic agents rather than + chasing the score. See + `references/common-pitfalls.md#tuning-quality-for-stylistic-agents`. +- **Compare mode is opt-in.** The A/B model-matrix (compare) mode is **opt-in** + and OFF by default; scaffold it only when the user explicitly asks. Telemetry + and Quality are the defaults. + +## References + +- **`references/default-scaffold.md` — the one-read, create-first file set for + the default modes (Telemetry + Quality + NLP + Reporting + Wire). Read this + first; it has the complete body of every default-mode file so you create the + whole scaffold from a single doc.** +- `references/project-template.md` — file tree + `.csproj` layout + version policy. +- `references/ichatclient-detection.md` — registration scan + factory emission. +- `references/evaluators-catalog.md` — NLP + Quality + Safety catalog with required `EvaluationContext` types. +- `references/metrics-glossary.md` — per-run glossary content + `MetricsGlossary.cs` template. +- `references/telemetry-capture.md` — per-call hook + cost report format. +- `references/quality-modes.md` — `DiskBasedReportingConfiguration` wiring + `aieval report` invocation. +- `references/compare-mode.md` — `matrix.json` + per-entry `executionName`. +- `references/safety-mode.md` — opt-in safety scaffold + `ContentHarmEvaluator` default. +- `references/ci-workflow.md` — `.github/workflows/evals.yml` template. +- `references/aspire-dashboard-panel.md` — optional static-file panel. +- `references/common-pitfalls.md` — known footguns to avoid when scaffolding. +- [MEAI.Evaluation libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) | [Tutorial: evaluate with reporting](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) | [dotnet/ai-samples](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md new file mode 100644 index 0000000000..943e43a871 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md @@ -0,0 +1,66 @@ +# Aspire dashboard panel (optional, v1 = static file) + +A minimal way to surface per-agent token/latency live during +`dotnet run` without extending the Aspire dashboard itself. + +## V1 — static file served from AppHost + +1. Telemetry mode (when running long-lived) writes + `wwwroot/eval-panel/telemetry.json` every N calls. +2. AppHost serves a static HTML page at `/eval-panel/` that fetches + the JSON every 2 seconds and renders a table. + +### `wwwroot/eval-panel/index.html` + +```html + +Agentic perf panel + +

Agentic perf panel

+

Last update: —

+ + + + + +
AgentModelCallsAvg msp95 msIn tokOut tok$/1K
+ +``` + +## V1 caveats + +- This is **not** an embedded Aspire dashboard panel; the dashboard + panel API is out of scope here. +- The panel reads only the latest `telemetry.json`; it does not + retain history across runs. +- If the AppHost project does not already enable static files, the + skill adds `app.UseStaticFiles()` (in apply mode only, with a + standard diff-preview-and-confirm flow). + +## Future v2 + +A proper Aspire dashboard contribution is tracked separately. The +v1 static panel buys most of the benefit (live per-agent visibility) +without depending on the dashboard extension story. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md new file mode 100644 index 0000000000..892a86f29e --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md @@ -0,0 +1,121 @@ +# CI workflow (opt-in) + +Off by default. Enabled in step 2 when the user picks "scaffold CI +workflow" / "add GitHub Actions" / equivalent. + +When enabled, emits `.github/workflows/evals.yml`. The workflow: + +- Runs on every PR + on push to `main`. +- Always runs Tier 1 (NLP) — no creds needed, fast smoke test. +- Promotes to Tier 2 (Quality, real judge) when the repo has secrets + `AZURE_OPENAI_ENDPOINT` + `AZURE_TENANT_ID`. +- Promotes to Tier 3 (Foundry Safety) when the repo has secret + `AZURE_AI_FOUNDRY_ENDPOINT`. +- Uploads `report.html` as a workflow artifact. + +## Template + +```yaml +name: evals + +on: + pull_request: + branches: [main] + push: + branches: [main] + workflow_dispatch: + +permissions: + id-token: write # for OIDC -> DefaultAzureCredential + contents: read + +jobs: + evaluate: + runs-on: ubuntu-latest + timeout-minutes: 20 + + env: + EVAL_REPORT_FOLDER: ${{ github.run_id }}-${{ github.run_attempt }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-dotnet@v4 + with: + dotnet-version: 10.0.x + + - name: Restore tools + working-directory: {{AppName}}.Evals.Tests + run: dotnet tool restore + + - name: Azure login (only if creds present) + if: env.AZURE_TENANT_ID != '' + uses: azure/login@v2 + with: + client-id: ${{ secrets.AZURE_CLIENT_ID }} + tenant-id: ${{ secrets.AZURE_TENANT_ID }} + subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} + env: + AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} + + - name: Tier selection + id: tier + run: | + if [ -n "${{ secrets.AZURE_OPENAI_ENDPOINT }}" ]; then + echo "EVAL_USE_REAL_AGENT=1" >> $GITHUB_ENV + echo "EVAL_USE_REAL_JUDGE=1" >> $GITHUB_ENV + echo "AZURE_OPENAI_ENDPOINT=${{ secrets.AZURE_OPENAI_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=judge" >> $GITHUB_OUTPUT + else + echo "tier=stub" >> $GITHUB_OUTPUT + fi + if [ -n "${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" ]; then + echo "EVAL_USE_FOUNDRY_SAFETY=1" >> $GITHUB_ENV + echo "AZURE_AI_FOUNDRY_ENDPOINT=${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=safety" >> $GITHUB_OUTPUT + fi + + - name: Run evals (dotnet test) + run: dotnet test {{AppName}}.Evals.Tests/{{AppName}}.Evals.Tests.csproj --logger "trx;LogFileName=evals.trx" + + - name: Generate report (already invoked by [AssemblyCleanup], this is a safety net) + if: always() + working-directory: {{AppName}}.Evals.Tests + run: | + mkdir -p $GITHUB_WORKSPACE/.copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }} + dotnet tool run aieval report \ + --path _store \ + --output $GITHUB_WORKSPACE/.copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html + + - name: Upload report + if: always() + uses: actions/upload-artifact@v4 + with: + name: eval-report-${{ steps.tier.outputs.tier }} + path: .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html + + - name: Upload trx + if: always() + uses: actions/upload-artifact@v4 + with: + name: trx + path: '**/evals.trx' +``` + +## Optional: PR comment with summary + +A second job can comment on the PR with a link to the artifact and a +one-line summary scraped from `compare.md`. Not in the default +template — too much variance across teams' bot/preference choices. + +## Required secrets + +| Secret | Tier unlocked | Required for OIDC? | +|--------|---------------|--------------------| +| `AZURE_TENANT_ID` | needed for any Azure auth | yes | +| `AZURE_CLIENT_ID` | OIDC federated identity | yes (if using `azure/login@v2`) | +| `AZURE_SUBSCRIPTION_ID` | scope | yes | +| `AZURE_OPENAI_ENDPOINT` | Tier 2 (judge) | no | +| `AZURE_AI_FOUNDRY_ENDPOINT` | Tier 3 (safety) | no | + +Document these clearly in the chat output when the workflow is scaffolded. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md new file mode 100644 index 0000000000..b9fcb98ed4 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -0,0 +1,214 @@ +# Common pitfalls + +Footguns that turned up while building and dogfooding this skill. +Avoid them when scaffolding `.Evals.Tests`. + +## Reporting pipeline + +- **Multiple `[AssemblyCleanup]` methods.** MSTest forbids more than + one `[AssemblyCleanup]` per assembly (UTA014 at discovery time — + every test in the assembly fails to load). The + `MetricsGlossary.WriteGlossary` write must be **chained from + `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`**, not + declared as its own. Wrap the chained call in a `try / catch` so a + glossary-write failure never masks the report. +- **Hand-rolling reports instead of using the Reporting pipeline.** + The whole point of the GA `Microsoft.Extensions.AI.Evaluation.Reporting` + package is `DiskBasedReportingConfiguration` + `aieval`. Never write + a hand-rolled markdown report and call it the "quality report" — + that's an MEAI report (HTML) vs a cost/latency capture (markdown). +- **Treating telemetry / compare / quality as separate report + streams.** Compare mode goes through `ReportingConfiguration` with + a distinct `executionName` per matrix entry, so `aieval report` + aggregates them into the same HTML. +- **Forgetting the per-run `metrics-glossary.md`.** The aieval HTML is + data-bound JSON; it shows numbers but no definitions. Co-locate + `metrics-glossary.md` (tier-aware) so a first-time reader can decode + the columns. The `Reporting/MetricsGlossary.cs` template handles + this — don't strip it. +- **Misreading "Cache Miss" in the Diagnostic Data section.** With the + default template (`enableResponseCaching: true`, agent resolved via + `run.ChatConfiguration!.ChatClient`, no per-run `executionName`), + the **first** `dotnet test` run shows Miss everywhere (cache empty) + and **subsequent runs against unchanged inputs show Hit everywhere** + in ~5s with zero LLM cost. Persistent Miss after run 1 means one of: + (a) the rubric / golden / scenario inputs changed, (b) the judge + model or chat options changed, (c) `_store/cache/` was deleted, or + (d) something is bypassing the cache — most commonly calling + `Wire.ResolveAgentClient()` (uncached) instead of + `run.ChatConfiguration!.ChatClient`, OR passing a fresh + `executionName` to `DiskBasedReportingConfiguration.Create(...)`. + Hit/Miss never affects correctness; it just tells you whether the + LLM was actually called this run. +- **Snake_case JSON deserialized with default STJ options.** `inputs.json`, + `matrix.json`, `prices.json`, and `golden.json` all use snake_case keys + by template convention, but the C# records use PascalCase. The loaders + (`InputsLoader`, `MatrixLoader`, `PriceTable.Load`, `GoldenLoader`) + **must** specify + `JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, PropertyNameCaseInsensitive = true }`, + or properties bind to `null`. Quality mode fails loudly (rubric/golden + evaluator throws); telemetry mode can fail **silently** (zeroed records + whose null fields are never read), which corrupts cost rollups. See + `references/telemetry-capture.md` and `references/compare-mode.md` for + the canonical loader sketches. + +## Clients (agent vs judge vs stub) + +- **Calling real models from the default test run.** Stub tier uses + `StubChatClient`; the report banner is clearly marked + `(stub IChatClient)`. Real-model runs are opt-in via three + independent env vars. +- **Conflating agent and judge clients.** Two different `IChatClient` + roles. The skill exposes them as two independent env vars + (`EVAL_USE_REAL_AGENT`, `EVAL_USE_REAL_JUDGE`) — one can be real + while the other is stubbed. +- **Auto-detected factory throwing a generic NRE on missing config.** + When the app uses Aspire orchestration, `dotnet test` runs outside + the AppHost and `ConnectionStrings:` is unset. The factory + template in `ichatclient-detection.md` wraps DI resolution in a + `try / catch` that throws a friendly `InvalidOperationException` + naming the connection-string key and the user-secrets command. Don't + strip this when adapting the template. +- **User-secrets silently not loading in `dotnet test`.** `dotnet test` + runs under `testhost.exe` as the entry assembly, so + `Host.CreateApplicationBuilder()` does NOT pick up the secrets store + bound to your test project's `UserSecretsId`. The factory must call + `builder.Configuration.AddUserSecrets(typeof(...).Assembly, optional: true)` + explicitly. Without it the secret is set on disk but the friendly NRE + still fires. +- **`services.ai.azure.com` hostname strips dashes.** Resource + `foundry-abc` resolves to host `foundryabc.services.ai.azure.com` (no + dash). Authoritative endpoints are in + `az cognitiveservices account show -n -g --query properties.endpoints`. + Use the `AI Foundry API` or `Azure AI Model Inference API` entries — + not the legacy `properties.endpoint` value, which points at the + `cognitiveservices.azure.com` hostname that 404s for the `/models` route. +- **Key-based auth disabled (`403`).** Foundry resources provisioned by + Aspire/azd usually set `disableLocalAuth=true`. Drop the `Key=` + segment from the connection string and rely on `DefaultAzureCredential` + (`az login` + a Cognitive Services User role assignment on the + resource). The Aspire `AddAzureChatCompletionsClient` registration + picks the credential automatically when the key is absent. +- **Reasoning models (gpt-5, gpt-5-mini, o1, o3) reject `max_tokens`.** + The `Azure.AI.Inference` beta SDK still sends `max_tokens`; reasoning + models require `max_completion_tokens` and return 400 + `unsupported_parameter`. **The MEAI Quality evaluators swallow the + 400 and record it as a per-metric error row, so tests pass but every + Quality column is an error.** Two workarounds: + - **Short-term:** pick a non-reasoning judge model (gpt-4o, + gpt-4o-mini, gpt-4-turbo). Tip: when picking a Foundry deployment + to point `ConnectionStrings:` at, check + `az cognitiveservices account deployment list -n -g + --query "[].{name:name, model:properties.model.name}" -o tsv` + and avoid any deployment whose model is `gpt-5*` or `o*`. When the + agent uses a reasoning model in production, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge + client points at a compatible deployment while the agent client + keeps the production model. + - **Durable (recommended for new apps):** migrate the AppHost off + `AddAzureChatCompletionsClient` (Azure.AI.Inference) onto the + OpenAI/v1 + stable OpenAI SDK path (`AddOpenAIClient` against an + OpenAI/v1 endpoint). The OpenAI SDK natively uses + `max_completion_tokens` against reasoning models. The + `Azure.AI.Inference` beta SDK is retiring on **August 26, 2026**; + see the + [Foundry migration guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/model-inference-to-openai-migration) + and the related issue + [dotnet/extensions#7580](https://github.com/dotnet/extensions/issues/7580). + The skill detects both registration patterns; see + `references/ichatclient-detection.md`. + +## Evaluators + +- **Wiring 4 separate safety evaluators.** Use `ContentHarmEvaluator` + for the Hate / SelfHarm / Violence / Sexual bundle — single Foundry + call, 4 metrics back. The 4 individual evaluators + (`HateAndUnfairnessEvaluator` etc.) are a strict subset. +- **Putting `RelevanceTruthAndCompletenessEvaluator` in the default + set.** Marked experimental upstream; not part of the v2 default. +- **Forgetting `EvaluationContext` for NLP evaluators.** BLEU / GLEU + need `BLEUEvaluatorContext(IEnumerable)`; F1 needs + `F1EvaluatorContext(string)`. If `golden.json` lacks + `reference_response`, NLP evaluators emit `(no reference)` and the + scenario shows blanks in the report. + +### Tuning Quality for stylistic agents + +The built-in Quality evaluators (`RelevanceEvaluator`, +`CoherenceEvaluator`, `FluencyEvaluator`, `CompletenessEvaluator`, +`EquivalenceEvaluator`) use **generic rubrics baked into the +evaluator's judge prompt**. They don't know about *your* agent's +contract. + +- **`CompletenessEvaluator` explicitly rewards thoroughness.** Any + agent whose contract is "compress / summarize / dumb-down" (ELI5, + TL;DR summarizers, tweet-length responders, structured-extractor + agents) will score 1-2 even when working perfectly. The score is + measuring conformance to a generic "answer thoroughly" rubric, not + conformance to your agent's contract. +- **`EquivalenceEvaluator` rewards lexical/semantic match to a + reference.** If your `golden.json` references are full-form + explanations but your agent emits paraphrases or stylized variants, + Equivalence will flag drift that isn't a real regression. +- **`CoherenceEvaluator` and `FluencyEvaluator`** penalize + fragmentary / one-line / heavily-structured outputs (JSON-only, + bullet-only, single-sentence). Agents that respond in a strict + format will be marked down. +- **Only `RelevanceEvaluator` is largely intent-agnostic** — it + checks "did the response address the query" without preferring + long-form. It's the one Quality metric that's mostly safe to leave + on for any agent. + +**Three remediation patterns, in increasing order of effort:** + +1. **Drop the offending evaluators per app.** Edit + `Reporting/ReportingConfig.cs` and remove `CompletenessEvaluator` + / `EquivalenceEvaluator` from the judge-tier evaluator list. + Keep Relevance + Coherence + Fluency. Document the choice in a + `// per-app: ELI5 contract → drop Completeness` comment so future + re-scaffolds don't add them back. +2. **Rewrite goldens in the agent's voice.** Edit `Quality/golden.json` + so `reference_response` is itself ELI5-style (or tweet-style, or + bullet-style). Equivalence becomes meaningful again; Completeness + still complains. +3. **Add a custom rubric-driven evaluator.** Write a `RubricEvaluator` + that reads `Quality/rubric.md` and asks the judge to score the + response against *your* criteria ("Is the explanation appropriate + for a 5-year-old? Score 1-5."). See + `references/evaluators-catalog.md#custom-rubric-driven-evaluator` + for the template. This is the right long-term answer for any + non-generic agent. + +> **Surface this in the chat output on first scaffold.** The user +> shouldn't have to interpret bad Quality scores against a bad-fit +> rubric and conclude their agent is broken. Step 11 of `SKILL.md` +> calls this out explicitly in the Quality headline block. + +## Configuration + +- **Hard-coding a price table.** Lives in `Telemetry/prices.json`, + user-editable. Costs change. +- **Auto-failing the build on quality regressions.** Quality mode is + informational by default. Users opt into a hard-fail by editing + `quality.thresholds.json` (which maps to real MEAI metric names like + `Relevance` / `BLEU` / `F1` and `EvaluationRating` levels), then + setting `hard_fail: true`. +- **Hand-pinning `Microsoft.Extensions.Hosting` too low.** Add it with + `dotnet add package Microsoft.Extensions.Hosting` (no `--version`) so it + resolves to the latest stable, which is always `>= 10.0.1`. A hand-pinned + `10.0.0` produces `NU1605` (treated as error in the agentic-app project + graph) because `Microsoft.Agents.AI.Hosting` 1.x requires `>= 10.0.1`. Do not + author a version literal here — let NuGet satisfy the floor. +- **Forgetting `.gitignore` entries.** Must include both + `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/`. + Otherwise reports pollute history and the persistent `_store/` + blocks PR diffs. + +## Update mode + +- **Overwriting `Quality/rubric.md` or `Quality/golden.json`.** These + are user data — never overwrite. The skill's update-mode behaviour + table in step 1a of `SKILL.md` is the source of truth. +- **Migrating `golden.json` schema destructively.** Migration is + additive only: new fields go in as nullable, existing rows are + preserved. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md new file mode 100644 index 0000000000..44909c5408 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md @@ -0,0 +1,132 @@ +# Compare mode + +Compare mode runs the quality + telemetry pipeline against **multiple +model assignments** and produces a side-by-side report. Crucially, it +goes through the **same `DiskBasedReportingConfiguration`** so +`aieval report` aggregates the comparison into a single HTML view. + +## `matrix.json` + +```json +{ + "schema_version": 2, + "entries": [ + { + "name": "baseline-all-mini", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o-mini", + "technical": "gpt-4o-mini", + "summariser": "gpt-4o-mini" + } + }, + { + "name": "interviewers-upgraded", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o", + "technical": "gpt-4o", + "summariser": "gpt-4o-mini" + } + } + ] +} +``` + +## Test shape + +Each matrix entry becomes one row of a parameterised test. Each entry +uses a **distinct `executionName`** so `aieval report` shows the +comparison side-by-side. + +```csharp +[TestClass] +public sealed class CompareTests +{ + public static IEnumerable Matrix() => + MatrixLoader.Load().Select(e => new object[] { e }); + + [TestMethod, DynamicData(nameof(Matrix), DynamicDataSourceType.Method)] + public async Task RunEntry(MatrixEntry entry) + { + // executionName scoped per entry so aieval report groups columns by it. + var reporting = DiskBasedReportingConfiguration.Create( + storageRootPath: ReportingConfig.StorageRoot, + evaluators: ReportingConfig.EvaluatorList(), + chatConfiguration: new ChatConfiguration( + Wire.ResolveJudgeClient(Wire.ResolveAgentClient(entry.ModelAssignments))), + enableResponseCaching: true, + // Stable per-entry name (NOT prefixed with a per-run timestamp) + // so re-running compare reuses the cache for unchanged entries. + executionName: $"compare-{entry.Name}"); + + foreach (var g in GoldenLoader.Load()) + { + var scenarioName = $"Compare.{entry.Name}.{g.Id}"; + await using var run = await reporting.CreateScenarioRunAsync(scenarioName); + // ... same shape as QualityTests + } + } +} +``` + +## Override the per-agent model id + +`Wire.ResolveAgentClient(IDictionary overrides)` is the +extension point. The generated factory (`AgentChatClientFactory`) +exposes an overload accepting per-agent model assignments — useful when +the app uses multiple deployment aliases or supports model swapping +via `ChatOptions.ModelId`. + +## `MatrixLoader` (snake_case JSON, PascalCase records) + +`matrix.json` uses snake_case keys (`schema_version`, `model_assignments`) +that bind to PascalCase C# properties (`SchemaVersion`, `ModelAssignments`). +The loader **must** opt into snake_case-aware deserialization or the +properties come back `null` and `CompareTests` crashes on enumeration: + +```csharp +internal static class MatrixLoader +{ + private static readonly JsonSerializerOptions s_options = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, + PropertyNameCaseInsensitive = true, + }; + + public static List Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "Compare", "matrix.json"); + var root = JsonSerializer.Deserialize( + File.ReadAllText(path), s_options); + return root?.Entries ?? new(); + } +} + +internal sealed record MatrixRoot(int SchemaVersion, List Entries); +internal sealed record MatrixEntry(string Name, Dictionary ModelAssignments); +``` + +Reuse the same `s_options` for `InputsLoader`, `PriceTable.Load()`, and +`GoldenLoader.Load()` — STJ defaults will silently null-out PascalCase +properties bound to snake_case JSON. + +## Compare-specific report + +`compare.md` (still emitted, in addition to the aggregated +`report.html`): + +| name | avg ms | in tok | out tok | $ | mean quality | mean BLEU | +|------|--------|--------|---------|---|--------------|-----------| +| baseline-all-mini | 432 | 380 | 210 | 0.0021 | 3.8 | 0.31 | +| interviewers-upgraded | 891 | 380 | 245 | 0.0210 | 4.4 | 0.42 | + +| Recommendation | +|----------------| +| `interviewers-upgraded`: +0.6 quality at +9.4× cost. Promote only if quality bar requires it. | + +The recommendation row is rule-based: + +- If cost increases > 3× and quality delta < 0.3 → **do not promote**. +- If cost increases ≤ 1.5× and quality delta ≥ 0.5 → **promote**. +- Otherwise → **manual review**. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/default-scaffold.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/default-scaffold.md new file mode 100644 index 0000000000..7b2fadd48c --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/default-scaffold.md @@ -0,0 +1,944 @@ +# Default scaffold — one-read, create-first file set + +**Read this file once, then create every file below with the `create` tool.** +This is the single source of truth for the **default** modes (Telemetry ON, +Quality ON, NLP ON). It exists so the agent does **not** have to open six +separate reference docs before writing anything — reading many refs first is +the top cause of an empty scaffold (the turn budget is gone before a single +file is written). + +- **Create files first; run `dotnet` later.** Every file here is authored with + the `create` tool and needs **no network**. The `dotnet` commands in SKILL.md + step 3 (`dotnet add package`, `dotnet tool install/restore`, `dotnet build`, + `dotnet test`) only **stamp versions and validate** on top of files that + already exist. If the SDK is slow or offline, a complete project is still on + disk. Success = files on disk, not a green build. +- Substitute `{{AppName}}` with the detected app name (e.g. `CodeReviewBuddy`) + and set the `` to each detected agent service project. +- Only for **opt-in** modes (Compare, Safety, CI workflow, Aspire panel) read + the corresponding mode doc — those files are **not** in this list. +- Deeper rationale for any file lives in the per-topic refs + (`telemetry-capture.md`, `quality-modes.md`, `ichatclient-detection.md`, + `evaluators-catalog.md`, `metrics-glossary.md`, `common-pitfalls.md`); you do + **not** need them to create the default scaffold. + +## Create order + +``` +.Evals.Tests/ + .Evals.Tests.csproj + GlobalUsings.cs + Reporting/Tier.cs + Reporting/WordCountEvaluator.cs + Reporting/ReportingConfig.cs + Reporting/MetricsGlossary.cs + Reporting/AievalReport.cs + Reporting/Thresholds.cs + Wire/StubChatClient.cs + Wire/AgentChatClientFactory.cs + Wire/Wire.cs + Telemetry/TelemetrySupport.cs + Telemetry/TelemetryTests.cs + Telemetry/inputs.json + Telemetry/prices.json + Quality/QualitySupport.cs + Quality/QualityTests.cs + Quality/rubric.md + Quality/golden.json + quality.thresholds.json +``` + +Then append the `.gitignore` entries (last section). + +--- + +## `.Evals.Tests.csproj` + +The test-SDK line(s) are owned by `dotnet new mstest` (they track the installed +SDK — on .NET 10 the MTP-native `MSTest` metapackage, no `Microsoft.NET.Test.Sdk`). +The block below is the reconciliation target: set the TFM, the data-file +`CopyToOutputDirectory` item, the agent `ProjectReference`, and the **version-less** +eval + hosting package set (no `Version` attribute — step 3.3's `dotnet add package` +stamps each resolved version in place). Do **not** hand-author a version literal. + +```xml + + + net10.0 + enable + enable + false + {{AppName}}.Evals.Tests + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +## `GlobalUsings.cs` + +```csharp +global using System.Diagnostics; +global using System.Text; +global using System.Text.Json; +global using Microsoft.Extensions.AI; +global using Microsoft.Extensions.AI.Evaluation; +global using Microsoft.Extensions.AI.Evaluation.NLP; +global using Microsoft.Extensions.AI.Evaluation.Quality; +global using Microsoft.Extensions.AI.Evaluation.Reporting; +global using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; +global using Microsoft.Extensions.Configuration; +global using Microsoft.Extensions.DependencyInjection; +global using Microsoft.Extensions.Hosting; +global using Microsoft.VisualStudio.TestTools.UnitTesting; +``` + +## `Reporting/Tier.cs` + +Tier enum, environment reader, and the repo/project path helpers every other +file uses. `ProjectRoot` = the directory holding the test `.csproj` +(and `.config/dotnet-tools.json`); `RepoRoot` = the nearest ancestor with a +`.git` folder or a `*.sln`/`*.slnx`. + +```csharp +namespace {{AppName}}.Evals.Tests; + +internal static class EvalEnv +{ + public static bool UseRealAgent => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_AGENT") == "1"; + + public static bool UseRealJudge => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_JUDGE") == "1"; + + public static bool UseFoundrySafety => + Environment.GetEnvironmentVariable("EVAL_USE_FOUNDRY_SAFETY") == "1"; + + public static string Tier => + UseFoundrySafety ? "Safety" : UseRealJudge ? "Judge" : "Stub"; + + // Per-run timestamp ONLY for the report output folder under + // .copilot/perf-reports/evals//. NOT passed to MEAI's + // ReportingConfiguration (that would scope the response cache per run and + // defeat caching). Override with EVAL_REPORT_FOLDER in CI. + public static readonly string ReportFolder = + Environment.GetEnvironmentVariable("EVAL_REPORT_FOLDER") + ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); +} + +internal static class ProjectRoot +{ + private static string? s_cached; + + public static string Find() + { + if (s_cached is not null) return s_cached; + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null) + { + if (dir.GetFiles("*.csproj").Length > 0 || + Directory.Exists(Path.Combine(dir.FullName, ".config"))) + return s_cached = dir.FullName; + dir = dir.Parent; + } + return s_cached = AppContext.BaseDirectory; + } +} + +internal static class RepoRoot +{ + private static string? s_cached; + + public static string Find() + { + if (s_cached is not null) return s_cached; + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null) + { + if (Directory.Exists(Path.Combine(dir.FullName, ".git")) || + dir.GetFiles("*.sln").Length > 0 || + dir.GetFiles("*.slnx").Length > 0) + return s_cached = dir.FullName; + dir = dir.Parent; + } + return s_cached = ProjectRoot.Find(); + } +} +``` + +## `Reporting/WordCountEvaluator.cs` + +Canonical Learn-doc pattern. Runs in stub tier with no API key. + +```csharp +namespace {{AppName}}.Evals.Tests; + +public sealed class WordCountEvaluator : IEvaluator +{ + public const string MetricName = "Words"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + public ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + var text = modelResponse?.Text ?? string.Empty; + var count = text.Split( + [' ', '\t', '\r', '\n'], + StringSplitOptions.RemoveEmptyEntries).Length; + + var metric = new NumericMetric(MetricName, value: count) + { + Interpretation = count switch + { + < 5 => new EvaluationMetricInterpretation(EvaluationRating.Poor, reason: "Response too short"), + > 500 => new EvaluationMetricInterpretation(EvaluationRating.Average, reason: "Response very long"), + _ => new EvaluationMetricInterpretation(EvaluationRating.Good), + } + }; + return new ValueTask(new EvaluationResult(metric)); + } +} +``` + +## `Reporting/ReportingConfig.cs` + +One `IChatClient` serves both the agent and the judge call so MEAI's response +cache covers both (see `quality-modes.md` for the caching rules). The +`executionName` is deliberately omitted to keep the cache scope stable across +runs. + +```csharp +namespace {{AppName}}.Evals.Tests; + +internal static class ReportingConfig +{ + // _store lives next to the test project (the directory that holds + // .config/dotnet-tools.json) so the AssemblyCleanup report call and + // `dotnet tool run aieval` resolve the same store. + public static readonly string StorageRoot = + Path.Combine(ProjectRoot.Find(), "_store"); + + public static ReportingConfiguration ForQuality() + { + var agent = Wire.ResolveAgentClient(); + var judge = Wire.ResolveJudgeClient(agent); + + var evaluators = new List + { + // Tier 1 — always on, deterministic (no LLM). + new WordCountEvaluator(), + new BLEUEvaluator(), + new GLEUEvaluator(), + new F1Evaluator(), + }; + + if (EvalEnv.UseRealJudge) + { + // Tier 2 — needs a real judge client. + evaluators.Add(new RelevanceEvaluator()); + evaluators.Add(new CoherenceEvaluator()); + evaluators.Add(new FluencyEvaluator()); + evaluators.Add(new CompletenessEvaluator()); + evaluators.Add(new EquivalenceEvaluator()); + evaluators.Add(new GroundednessEvaluator()); + // Agentic-only evaluators (IntentResolution / TaskAdherence / + // ToolCallAccuracy) are added when an *.AppHost.csproj is detected; + // see evaluators-catalog.md. + } + + // executionName deliberately omitted (stable cache scope). Report + // folder timestamp lives separately in EvalEnv.ReportFolder. + return DiskBasedReportingConfiguration.Create( + storageRootPath: StorageRoot, + evaluators: evaluators, + chatConfiguration: new ChatConfiguration(judge), + enableResponseCaching: true); + } +} +``` + +## `Reporting/MetricsGlossary.cs` + +Writes the tier-relevant slice of the metrics glossary next to `report.html`. +Plain static class (no `[TestClass]`) — MSTest allows only one +`[AssemblyCleanup]`, so this is chained from `AievalReport` (next file). + +```csharp +namespace {{AppName}}.Evals.Tests; + +internal static class MetricsGlossary +{ + public static void WriteGlossary() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var path = Path.Combine(outDir, "metrics-glossary.md"); + + var sb = new StringBuilder(); + sb.AppendLine($"# Metrics glossary — {EvalEnv.Tier} tier"); + sb.AppendLine(); + sb.AppendLine($"Generated: {DateTime.UtcNow:O}"); + sb.AppendLine(); + sb.AppendLine("Companion to `report.html` in this folder. The aieval HTML report shows numbers; this file explains them."); + sb.AppendLine(); + + sb.AppendLine(NlpEntries); + if (EvalEnv.UseRealJudge) sb.AppendLine(QualityEntries); + if (EvalEnv.UseFoundrySafety) sb.AppendLine(SafetyEntries); + + sb.AppendLine(); + sb.AppendLine("> Source: setup-maf-evals references/metrics-glossary.md"); + + File.WriteAllText(path, sb.ToString()); + Console.WriteLine($"[MetricsGlossary] {path}"); + } + + private const string NlpEntries = """ + ## NLP tier (deterministic, no LLM) + - **Words** (int): response length sanity check. <5 too short, 5-500 ok, >500 long. + - **BLEU** (0-1): n-gram overlap with reference(s). 0.1-0.3 normal, >0.3 strong, >0.5 near-quotation. *Lexical, not semantic.* + - **GLEU** (0-1): sentence-level BLEU; better for short outputs. Same buckets as BLEU. + - **F1** (0-1): unigram token F1 vs ground-truth. 0.3-0.5 typical, >0.6 strong word-level match. Order-insensitive. + + > Headline: NLP metrics measure wording similarity, not correctness. Use for regression early-warning, not as quality verdicts. + """; + + private const string QualityEntries = """ + ## Quality tier (LLM-as-judge) + Each rated 1-5 (Poor -> Excellent) with a free-text rationale. + - **Relevance**: addresses the user's query. Catches off-topic regressions. + - **Coherence**: logically structured. Catches rambling/contradictory outputs. + - **Fluency**: grammar/readability. Catches broken-English outputs. + - **Completeness** (needs reference): comprehensive and accurate. + - **Equivalence** (needs reference): semantic similarity in context of the query. + - **Groundedness** (needs context): aligned with supplied source-of-truth. + - **Intent Resolution / Task Adherence / Tool Call Accuracy** (agentic only). + + > Headline: judge scores drift across model versions. Pin the judge model for comparable runs. + """; + + private const string SafetyEntries = """ + ## Safety tier (Foundry) + Each rated 1-5 severity (1 safe -> 5 severe). + - **ContentHarm bundle** (single-shot, 4 metrics): Hate-And-Unfairness, Self-Harm, Violence, Sexual. + - **Protected Material**: copyrighted text reproduced. + - **Indirect Attack**: prompt-injection content from retrieved/tool data. + - **Code Vulnerability**: vulnerable code patterns (SQLi, weak crypto, etc.). + - **Ungrounded Attributes**: inferred human attributes not in input. + - **Groundedness Pro**: Foundry-hosted fine-tuned groundedness check. + + > Headline: all safety metrics inspect *outputs*, not inputs. Pair with Azure AI Content Safety on the request side for full coverage. + """; +} +``` + +## `Reporting/AievalReport.cs` + +The assembly's **single** `[AssemblyCleanup]`. Generates `report.html` via the +`aieval` tool, then chains the glossary write in a `try/catch` so a glossary +failure never masks the report. + +```csharp +namespace {{AppName}}.Evals.Tests; + +[TestClass] +public static class AievalReport +{ + [AssemblyCleanup] + public static void GenerateReport() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var html = Path.Combine(outDir, "report.html"); + + try + { + var psi = new ProcessStartInfo("dotnet", + $"tool run aieval report --path \"{ReportingConfig.StorageRoot}\" --output \"{html}\"") + { + WorkingDirectory = ProjectRoot.Find(), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + using var p = Process.Start(psi)!; + p.WaitForExit(); + Console.WriteLine($"Eval report: {html}"); + } + catch (Exception ex) + { + Console.Error.WriteLine($"[AievalReport] Report generation skipped: {ex.Message}"); + } + + try { MetricsGlossary.WriteGlossary(); } + catch (Exception ex) { Console.Error.WriteLine($"[MetricsGlossary] Failed: {ex.Message}"); } + } +} +``` + +## `Reporting/Thresholds.cs` + +Reads `quality.thresholds.json` and, when `hard_fail: true`, fails the test on a +below-threshold metric. Default (`hard_fail: false`) is informational — failures +show in the report only. + +```csharp +namespace {{AppName}}.Evals.Tests; + +internal static class Thresholds +{ + private static readonly JsonSerializerOptions s_options = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, + PropertyNameCaseInsensitive = true, + }; + + private sealed record Rule(string? MinRating, double? MinValue, double? MaxValue); + private sealed record Config(int SchemaVersion, bool HardFail, Dictionary Thresholds); + + private static Config Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "quality.thresholds.json"); + if (!File.Exists(path)) return new Config(2, false, new()); + return JsonSerializer.Deserialize(File.ReadAllText(path), s_options) + ?? new Config(2, false, new()); + } + + // Logs every metric; when hard_fail is set, Assert.Fail on any breach. + public static void ApplyOrLog(EvaluationResult result, string scenarioId) + { + var cfg = Load(); + var breaches = new List(); + + foreach (var metric in result.Metrics.Values) + { + if (!cfg.Thresholds.TryGetValue(metric.Name, out var rule)) continue; + + if (metric is NumericMetric num && num.Value is double v) + { + if (rule.MinValue is double lo && v < lo) + breaches.Add($"{metric.Name}={v} < min {lo}"); + if (rule.MaxValue is double hi && v > hi) + breaches.Add($"{metric.Name}={v} > max {hi}"); + } + + if (rule.MinRating is not null && + Enum.TryParse(rule.MinRating, out var minRating) && + metric.Interpretation?.Rating is EvaluationRating actual && + actual > minRating) // enum: Poor(1) < ... < Excellent(5) but rating order is reversed + { + breaches.Add($"{metric.Name} rating {actual} below {minRating}"); + } + } + + if (breaches.Count > 0) + { + var msg = $"[{scenarioId}] threshold breaches: {string.Join("; ", breaches)}"; + if (cfg.HardFail) Assert.Fail(msg); + else Console.WriteLine($"(informational) {msg}"); + } + } +} +``` + +## `Wire/StubChatClient.cs` + +Deterministic, offline `IChatClient` used when `EVAL_USE_REAL_AGENT` is unset. +The report banner is marked `(stub IChatClient)`. + +```csharp +namespace {{AppName}}.Evals.Tests; + +internal sealed class StubChatClient : IChatClient +{ + public Task GetResponseAsync( + IEnumerable messages, + ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + var last = messages.LastOrDefault()?.Text ?? string.Empty; + var text = $"(stub IChatClient) Echoing: {last}"; + var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, text)) + { + ModelId = "stub", + Usage = new UsageDetails { InputTokenCount = 0, OutputTokenCount = 0 }, + }; + return Task.FromResult(response); + } + + public async IAsyncEnumerable GetStreamingResponseAsync( + IEnumerable messages, + ChatOptions? options = null, + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) + { + var response = await GetResponseAsync(messages, options, cancellationToken); + yield return new ChatResponseUpdate(ChatRole.Assistant, response.Text); + } + + public object? GetService(Type serviceType, object? serviceKey = null) => null; + + public void Dispose() { } +} +``` + +## `Wire/AgentChatClientFactory.cs` + +Emit the **Case A** template below when detection found exactly one +`IChatClient` registration; replace `{{InsertDetectedRegistrationCallVerbatim}}` +with the literal call from the app and `{{ConnStrName}}` with the connection +string alias. For **multiple** registrations (Case B) emit the same shape with a +comment listing candidates and ask the user to pick before writing. For **no +registration** (Case C) emit a `Create()` that throws `NotImplementedException` +with a wire-your-client message. See `ichatclient-detection.md` for B/C bodies +and the Foundry connection-string setup notes. + +```csharp +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Resolves the same IChatClient the app uses, by building a minimal host + /// that mirrors the app's DI registration. + /// Detected: {{DetectionSummary}} at {{File}}:{{Line}} + /// + public static IChatClient Create() + { + var builder = Host.CreateApplicationBuilder(); + + // In test hosts (`dotnet test`), the entry assembly is testhost.exe, so + // user-secrets are NOT auto-loaded. Add them explicitly from THIS assembly. + builder.Configuration.AddUserSecrets(typeof(AgentChatClientFactory).Assembly, optional: true); + + // {{InsertDetectedRegistrationCallVerbatim}} + var host = builder.Build(); + try + { + return host.Services.GetRequiredService(); + } + catch (InvalidOperationException ex) + { + throw new InvalidOperationException( + "EVAL_USE_REAL_AGENT=1 but IChatClient could not be resolved. The " + + "detected registration ({{DetectionSummary}}) reads connection string " + + "\"{{ConnStrName}}\" from configuration. Aspire's AppHost populates this " + + "at runtime, but `dotnet test` runs standalone. Set it via:\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;DeploymentId={{ConnStrName}}\" " + + "--project {{AppName}}.Evals.Tests\n" + + "(Drop `Key=` and use DefaultAzureCredential when key auth is disabled; " + + "the hostname strips dashes from the resource name.)", + ex); + } + } +} +``` + +## `Wire/Wire.cs` + +Central resolver. Agent client is real (`EVAL_USE_REAL_AGENT=1`) or the stub. +The judge defaults to the **same** instance as the agent (one credential setup, +one shared response cache). `EVAL_JUDGE_DEPLOYMENT_NAME` is honored by +`QualityTests` (see `quality-modes.md`). + +```csharp +namespace {{AppName}}.Evals.Tests; + +internal static class Wire +{ + public static IChatClient ResolveAgentClient() => + EvalEnv.UseRealAgent ? AgentChatClientFactory.Create() : new StubChatClient(); + + // Judge == agent by default. When EVAL_USE_REAL_JUDGE is set without a real + // agent, still use the resolved agent client (stub) so the pipeline runs + // offline; a separate judge deployment is opt-in via + // EVAL_JUDGE_DEPLOYMENT_NAME (handled in QualityTests). + public static IChatClient ResolveJudgeClient(IChatClient agent) => agent; +} +``` + +## `Telemetry/TelemetrySupport.cs` + +Input record + loader, price table, per-call record + store, and the delegating +capture client. **All JSON loaders use snake_case options** — the JSON files use +snake_case keys but the records are PascalCase; default STJ options bind them to +`null` (telemetry fails *silently* with zeroed records). See +`common-pitfalls.md`. + +```csharp +namespace {{AppName}}.Evals.Tests; + +public sealed record TelemetryInput(string Agent, string Text); + +internal sealed record TelemetryRecord( + string AgentName, string Model, + long InputTokens, long OutputTokens, + long LatencyMs, decimal CostUsd); + +internal static class JsonOpts +{ + public static readonly JsonSerializerOptions SnakeCase = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, + PropertyNameCaseInsensitive = true, + }; +} + +internal static class InputsLoader +{ + public static List Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "Telemetry", "inputs.json"); + return JsonSerializer.Deserialize>( + File.ReadAllText(path), JsonOpts.SnakeCase) ?? new(); + } +} + +internal sealed class PriceTable +{ + private sealed record Price(double InputPer1k, double OutputPer1k); + private readonly Dictionary _prices; + + private PriceTable(Dictionary prices) => _prices = prices; + + public static PriceTable Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "Telemetry", "prices.json"); + var prices = File.Exists(path) + ? JsonSerializer.Deserialize>( + File.ReadAllText(path), JsonOpts.SnakeCase) ?? new() + : new(); + return new PriceTable(prices); + } + + public decimal Cost(string? modelId, long inputTokens, long outputTokens) + { + if (modelId is null || !_prices.TryGetValue(modelId, out var p)) return 0m; + return (decimal)(inputTokens / 1000.0 * p.InputPer1k + + outputTokens / 1000.0 * p.OutputPer1k); + } +} + +internal static class TelemetryStore +{ + private static readonly List s_records = new(); + + public static void Record(TelemetryRecord r) + { + lock (s_records) s_records.Add(r); + } + + public static void FlushTo(string outDir) + { + Directory.CreateDirectory(outDir); + List snapshot; + lock (s_records) snapshot = new(s_records); + + // telemetry.json + File.WriteAllText( + Path.Combine(outDir, "telemetry.json"), + JsonSerializer.Serialize(snapshot, new JsonSerializerOptions { WriteIndented = true })); + + // telemetry.md + var sb = new StringBuilder(); + sb.AppendLine($"# Telemetry capture — {EvalEnv.Tier} tier"); + sb.AppendLine(); + sb.AppendLine("| Agent | Model | In tok | Out tok | Latency (ms) | Cost (USD) |"); + sb.AppendLine("|-------|-------|-------:|--------:|-------------:|-----------:|"); + foreach (var r in snapshot) + sb.AppendLine($"| {r.AgentName} | {r.Model} | {r.InputTokens} | {r.OutputTokens} | {r.LatencyMs} | {r.CostUsd:0.######} |"); + File.WriteAllText(Path.Combine(outDir, "telemetry.md"), sb.ToString()); + + Console.WriteLine($"[Telemetry] {Path.Combine(outDir, "telemetry.md")}"); + } +} + +internal sealed class TelemetryCapturingChatClient(IChatClient inner, PriceTable prices) : IChatClient +{ + public decimal LastCostUsd { get; private set; } + + public async Task GetResponseAsync( + IEnumerable messages, ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + var resp = await inner.GetResponseAsync(messages, options, cancellationToken); + var usage = resp.Usage; + LastCostUsd = prices.Cost(resp.ModelId, + usage?.InputTokenCount ?? 0, usage?.OutputTokenCount ?? 0); + return resp; + } + + public async IAsyncEnumerable GetStreamingResponseAsync( + IEnumerable messages, ChatOptions? options = null, + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) + { + await foreach (var update in inner.GetStreamingResponseAsync(messages, options, cancellationToken)) + yield return update; + } + + public object? GetService(Type serviceType, object? serviceKey = null) => + inner.GetService(serviceType, serviceKey); + + public void Dispose() => inner.Dispose(); +} +``` + +## `Telemetry/TelemetryTests.cs` + +```csharp +namespace {{AppName}}.Evals.Tests; + +[TestClass] +public sealed class TelemetryTests +{ + public static IEnumerable Inputs() => + InputsLoader.Load().Select(i => new object[] { i }); + + [TestMethod, DynamicData(nameof(Inputs), DynamicDataSourceType.Method)] + public async Task Capture(TelemetryInput input) + { + var inner = Wire.ResolveAgentClient(); + var wrapped = new TelemetryCapturingChatClient(inner, PriceTable.Load()); + + var messages = new List { new(ChatRole.User, input.Text) }; + var sw = Stopwatch.StartNew(); + var response = await wrapped.GetResponseAsync(messages); + sw.Stop(); + + TelemetryStore.Record(new TelemetryRecord( + AgentName: input.Agent, + Model: response.ModelId ?? "unknown", + InputTokens: response.Usage?.InputTokenCount ?? 0, + OutputTokens: response.Usage?.OutputTokenCount ?? 0, + LatencyMs: sw.ElapsedMilliseconds, + CostUsd: wrapped.LastCostUsd)); + } + + [ClassCleanup] + public static void WriteReports() => TelemetryStore.FlushTo( + Path.Combine(RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ReportFolder)); +} +``` + +## `Telemetry/inputs.json` + +5 starter inputs the user customizes. Replace the agent names / prompts with the +target app's agents. + +```json +[ + { "agent": "receptionist", "text": "Hi" }, + { "agent": "behavioural", "text": "Tell me about a tough project" }, + { "agent": "technical", "text": "How would you throttle requests?" }, + { "agent": "summariser", "text": "Wrap up the interview" }, + { "agent": "receptionist", "text": "What roles can I practice for?" } +] +``` + +## `Telemetry/prices.json` + +Edit freely — costs change. Never bake prices into source. + +```json +{ + "gpt-4o-mini": { "input_per_1k": 0.00015, "output_per_1k": 0.0006 }, + "gpt-4o": { "input_per_1k": 0.0025, "output_per_1k": 0.01 }, + "o4-mini": { "input_per_1k": 0.003, "output_per_1k": 0.012 } +} +``` + +## `Quality/QualitySupport.cs` + +Golden item + loaders. Snake_case JSON options again (see `common-pitfalls.md`). + +```csharp +namespace {{AppName}}.Evals.Tests; + +public sealed record GoldenItem( + string Id, + string UserMessage, + string? ReferenceResponse, + string? Context, + string[]? ExpectedTraits, + string[]? ExpectedToolCalls); + +internal static class GoldenLoader +{ + private sealed record GoldenFile(int SchemaVersion, List Scenarios); + + public static List Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "Quality", "golden.json"); + var file = JsonSerializer.Deserialize( + File.ReadAllText(path), JsonOpts.SnakeCase); + return file?.Scenarios ?? new(); + } +} + +internal static class RubricLoader +{ + public static string SystemPrompt() + { + var path = Path.Combine(AppContext.BaseDirectory, "Quality", "rubric.md"); + return File.Exists(path) ? File.ReadAllText(path) : string.Empty; + } +} +``` + +## `Quality/QualityTests.cs` + +Uses `run.ChatConfiguration!.ChatClient` for the agent call (the cached wrapper) +unless `EVAL_JUDGE_DEPLOYMENT_NAME` splits judge from agent — then falls back to +the uncached agent factory so the agent call doesn't silently use the judge +model. See `quality-modes.md` for the caching contract. + +```csharp +namespace {{AppName}}.Evals.Tests; + +[TestClass] +public sealed class QualityTests +{ + private static ReportingConfiguration s_reporting = null!; + + [ClassInitialize] + public static void Init(TestContext _) => + s_reporting = ReportingConfig.ForQuality(); + + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); + + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task Evaluate(GoldenItem g) + { + var scenarioName = $"{nameof(QualityTests)}.{g.Id}"; + await using var run = await s_reporting.CreateScenarioRunAsync(scenarioName); + + var agent = string.IsNullOrEmpty( + Environment.GetEnvironmentVariable("EVAL_JUDGE_DEPLOYMENT_NAME")) + ? run.ChatConfiguration!.ChatClient + : Wire.ResolveAgentClient(); + + var messages = new List + { + new(ChatRole.System, RubricLoader.SystemPrompt()), + new(ChatRole.User, g.UserMessage), + }; + var response = await agent.GetResponseAsync(messages); + + var contexts = new List(); + if (!string.IsNullOrEmpty(g.ReferenceResponse)) + { + contexts.Add(new BLEUEvaluatorContext([g.ReferenceResponse])); + contexts.Add(new GLEUEvaluatorContext([g.ReferenceResponse])); + contexts.Add(new F1EvaluatorContext(g.ReferenceResponse)); + contexts.Add(new EquivalenceEvaluatorContext(g.ReferenceResponse)); + contexts.Add(new CompletenessEvaluatorContext(g.ReferenceResponse)); + } + if (!string.IsNullOrEmpty(g.Context)) + contexts.Add(new GroundednessEvaluatorContext(g.Context)); + + var result = await run.EvaluateAsync(messages, response, contexts); + Thresholds.ApplyOrLog(result, g.Id); + } +} +``` + +## `Quality/rubric.md` + +The LLM-judge rubric the user edits to describe *their* agent's contract. + +```markdown +# Quality rubric + +Score the assistant response against the following criteria. Adjust these to +match your agent's contract. + +- **On-topic:** the response addresses the user's actual request. +- **Safe:** no harmful, biased, or policy-violating content. +- **Format:** the response follows the expected shape (length, structure, tone). +- **Accurate:** claims are correct and, where a reference/context is supplied, + grounded in it. +``` + +## `Quality/golden.json` + +Golden conversations (schema v2). `reference_response` feeds BLEU/GLEU/F1/ +Equivalence/Completeness; `context` feeds Groundedness. Both may be `null`. + +```json +{ + "schema_version": 2, + "scenarios": [ + { + "id": "g-receptionist-greeting", + "user_message": "Hi, I'd like to start an interview.", + "reference_response": "Hello! I'd be happy to start an interview with you. What role are you preparing for?", + "context": null, + "expected_traits": ["on_topic", "safe", "format_correct"], + "expected_tool_calls": null + } + ] +} +``` + +## `quality.thresholds.json` + +Maps **real MEAI metric names** to minimum ratings / values. `hard_fail: false` +(default) is informational; set `true` to fail the test on a breach. + +```json +{ + "schema_version": 2, + "hard_fail": false, + "thresholds": { + "Relevance": { "min_rating": "Good" }, + "Coherence": { "min_rating": "Good" }, + "Fluency": { "min_rating": "Average" }, + "Groundedness": { "min_rating": "Good" }, + "BLEU": { "min_value": 0.20 }, + "F1": { "min_value": 0.30 }, + "Words": { "min_value": 5, "max_value": 500 } + } +} +``` + +## `.gitignore` additions (idempotent) + +Append to the repo `.gitignore` if not already present: + +``` +# setup-maf-evals +.copilot/perf-reports/evals/ +.Evals.Tests/_store/ +``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md new file mode 100644 index 0000000000..487524eb74 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md @@ -0,0 +1,70 @@ +# `dotnet-tools.json` manifest + +The skill scaffolds a **local** tool manifest. Global install is +explicitly avoided — pinning the tool version to the project makes +evals reproducible across machines and CI runs. + +## Why local + +- Reproducible: every clone gets the exact same `aieval` version. +- CI-friendly: `dotnet tool restore` in the workflow is one line. +- No PATH conflicts with developers who may have other versions installed. + +## Generation (do not hand-write the version) + +Generate the manifest instead of authoring a pinned version literal: + +```pwsh +cd .Evals.Tests +dotnet new tool-manifest # if none exists +dotnet tool install microsoft.extensions.ai.evaluation.console # latest; provides `aieval` +``` + +`dotnet tool install` (no `--version`) records the **current** tool version into +the manifest and defaults to `rollForward: false`, so you still get a pinned, +reproducible entry — but NuGet chose the number, not the skill. The resulting +manifest looks like this (version shown is an illustrative snapshot): + +```json +{ + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } +} +``` + +`dotnet new tool-manifest` places the file per the SDK default — conventionally +`.Evals.Tests/.config/dotnet-tools.json`, though newer SDKs may write a +repo-root-style `dotnet-tools.json` in the current directory. Either way, run +`dotnet tool restore` / `dotnet tool run` **from the `.Evals.Tests/` +directory** so discovery finds the manifest. Running them from the repo root +would walk *up* and could miss a manifest that lives in a child directory, so the +CI `Restore tools` step, the `aieval report` safety-net step, and the runtime +`AssemblyCleanup` report call all use the test-project directory as their working +directory. The skill emits `dotnet tool restore` as the next step in the chat +output after scaffolding. + +## Why `rollForward: false` + +The output of `aieval report` is consumed by humans visually and by +artifact comparison in CI. A silent rollForward of the tool to a +newer version can change report layout, charts, or metric grouping — +breaking trend comparisons and visual diffs. + +If a user wants to upgrade the tool, the skill should emit a +`dotnet tool update microsoft.extensions.ai.evaluation.console` +command in the chat output (with a note: "this may change report +layout"). + +## Conflict with existing manifest + +If `dotnet-tools.json` already exists at the project (or any parent +directory), the skill **merges** the `aieval` entry rather than +overwriting. If a different version of `aieval` is already pinned, +surface the diff in chat output and require user confirmation. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md new file mode 100644 index 0000000000..232ffb7445 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md @@ -0,0 +1,271 @@ +# Evaluators catalog + +Full catalog of evaluators wired by `setup-maf-evals`, grouped by tier. +The skill defaults to Tier 1 (NLP) always on, Tier 2 (Quality) on but +gated by `EVAL_USE_REAL_JUDGE`, Tier 3 (Safety) off unless opted in. + +Source: [learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) + +## Tier 1 — NLP (deterministic, no LLM) + +Package: `Microsoft.Extensions.AI.Evaluation.NLP` (preview — added via `--prerelease`). + +| Evaluator | Metric | Context type needed | Notes | +|-----------|--------|---------------------|-------| +| `BLEUEvaluator` | `BLEU` | `BLEUEvaluatorContext(IEnumerable references)` | n-gram overlap with one or more reference strings | +| `GLEUEvaluator` | `GLEU` | `GLEUEvaluatorContext(IEnumerable references)` | sentence-level BLEU variant | +| `F1Evaluator` | `F1` | `F1EvaluatorContext(string groundTruth)` | unigram-level F1 | + +Plus a built-in custom evaluator the skill always scaffolds: + +| Evaluator | Metric | Why | +|-----------|--------|-----| +| `WordCountEvaluator` (custom) | `Words` | Sanity check: response is non-empty and reasonable length. Same pattern as the Learn doc tutorial. | + +### `WordCountEvaluator` reference implementation + +Scaffold this file verbatim (the Learn-doc canonical pattern) into +`Reporting/WordCountEvaluator.cs`. It runs in stub tier with no API key. + +```csharp +public sealed class WordCountEvaluator : IEvaluator +{ + public const string MetricName = "Words"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + public ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + var text = modelResponse?.Text ?? string.Empty; + var count = text.Split( + new[] { ' ', '\t', '\r', '\n' }, + StringSplitOptions.RemoveEmptyEntries).Length; + + var metric = new NumericMetric(MetricName, value: count) + { + Interpretation = count switch + { + < 5 => new EvaluationMetricInterpretation(EvaluationRating.Poor, reason: "Response too short"), + > 500 => new EvaluationMetricInterpretation(EvaluationRating.Average, reason: "Response very long"), + _ => new EvaluationMetricInterpretation(EvaluationRating.Good), + } + }; + return new ValueTask(new EvaluationResult(metric)); + } +} +``` + +## Tier 2 — Quality (LLM-as-judge) + +Package: `Microsoft.Extensions.AI.Evaluation.Quality` (GA — latest stable). + +Requires `EVAL_USE_REAL_JUDGE=1` and a real `IChatClient`. The skill +wires the following by default: + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `RelevanceEvaluator` | `Relevance` | none | how relevant is the response to the query | +| `CoherenceEvaluator` | `Coherence` | none | logical, orderly presentation | +| `FluencyEvaluator` | `Fluency` | none | grammar, readability | +| `CompletenessEvaluator` | `Completeness` | `CompletenessEvaluatorContext(string groundTruth)` | comprehensive and accurate | +| `EquivalenceEvaluator` | `Equivalence` | `EquivalenceEvaluatorContext(string groundTruth)` | similarity vs ground truth wrt query | +| `GroundednessEvaluator` | `Groundedness` | `GroundednessEvaluatorContext(string context)` | alignment with given context | + +Agent-focused (added when `*.AppHost.csproj` is detected, indicating +this is an agentic app and not just a chat completion app): + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `IntentResolutionEvaluator` | `Intent Resolution` | none | identifies + resolves user intent | +| `TaskAdherenceEvaluator` | `Task Adherence` | none | sticks to assigned task | +| `ToolCallAccuracyEvaluator` | `Tool Call Accuracy` | `ToolCallAccuracyEvaluatorContext(...)` | uses tools correctly | + +**Not wired by default:** `RelevanceTruthAndCompletenessEvaluator` +(marked experimental in upstream docs), `RetrievalEvaluator` (specific +to RAG pipelines — separate skill territory). + +## Tier 3 — Safety (Foundry Evaluation service) + +Package: `Microsoft.Extensions.AI.Evaluation.Safety` (preview — added via `--prerelease`). + +Off by default. Enabled when user opts in during step 2 of the +workflow. Requires `EVAL_USE_FOUNDRY_SAFETY=1` and an Azure AI Foundry +endpoint (the skill prompts for `AZURE_AI_FOUNDRY_ENDPOINT` + Entra credentials). + +**Always wire the bundle, not the 4 separate evaluators:** + +| Evaluator | Metrics produced | Notes | +|-----------|------------------|-------| +| `ContentHarmEvaluator` | `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` | **Single-shot — one Foundry call returns all 4 metrics.** Always prefer this over the 4 separate evaluators below. | + +Additional safety evaluators (each wired as separate `[TestMethod]`): + +| Evaluator | Metric | Notes | +|-----------|--------|-------| +| `ProtectedMaterialEvaluator` | `Protected Material` | copyrighted material in output | +| `IndirectAttackEvaluator` | `Indirect Attack` | prompt-injection-style indirect attacks | +| `CodeVulnerabilityEvaluator` | `Code Vulnerability` | vulnerable code in output | +| `UngroundedAttributesEvaluator` | `Ungrounded Attributes` | inferred human attributes | +| `GroundednessProEvaluator` | `Groundedness Pro` | fine-tuned Foundry-hosted groundedness check | + +The 4 separate evaluators (`HateAndUnfairnessEvaluator`, +`SelfHarmEvaluator`, `ViolenceEvaluator`, `SexualEvaluator`) are +**not** scaffolded — they're a strict subset of `ContentHarmEvaluator` +and cost 4× more Foundry calls for the same metrics. + +## Threshold mapping + +`quality.thresholds.json` maps **real MEAI metric names** to minimum +`EvaluationRating` enum values: + +```json +{ + "schema_version": 2, + "hard_fail": false, + "thresholds": { + "Relevance": { "min_rating": "Good" }, + "Coherence": { "min_rating": "Good" }, + "Fluency": { "min_rating": "Average" }, + "Groundedness":{ "min_rating": "Good" }, + "BLEU": { "min_value": 0.20 }, + "F1": { "min_value": 0.30 }, + "Words": { "min_value": 5, "max_value": 500 } + } +} +``` + +`hard_fail: true` makes any below-threshold metric fail the test. +Default `false` makes the test pass-through informational — failures +show in the report only. + +## Custom rubric-driven evaluator + +The built-in Quality evaluators (Relevance, Coherence, Fluency, +Completeness, Equivalence) judge against generic rubrics baked into +the evaluator's judge prompt. They cannot read `Quality/rubric.md`. +That makes them ill-fitting for agents with deliberate stylistic +constraints (ELI5 / summarizer / strict-format / persona-bound +agents) — see `common-pitfalls.md#tuning-quality-for-stylistic-agents`. + +For those agents, scaffold a `RubricEvaluator` that reads +`Quality/rubric.md` and asks the judge to score against *your* +criteria. The pattern is shape-identical to `WordCountEvaluator` — +it just delegates to the judge chat client. + +```csharp +// Reporting/RubricEvaluator.cs +// Custom rubric-driven evaluator. Reads Quality/rubric.md and asks +// the judge to score the response against per-app criteria. Emits a +// single "RubricFit" metric (numeric 1-5) plus a free-text rationale. +public sealed class RubricEvaluator : IEvaluator +{ + public const string MetricName = "RubricFit"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + private readonly string _rubric; + + public RubricEvaluator(string rubricMarkdown) => _rubric = rubricMarkdown; + + public static RubricEvaluator FromFile(string path) => + new(File.ReadAllText(path)); + + public async ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + if (chatConfiguration?.ChatClient is null) + { + // Judge tier not active — return a stubbed Inconclusive metric. + return new EvaluationResult(new NumericMetric(MetricName) + { + Interpretation = new EvaluationMetricInterpretation( + EvaluationRating.Inconclusive, reason: "Judge not wired.") + }); + } + + var userQuery = string.Join("\n", messages.Select(m => m.Text)); + var responseText = modelResponse.Text ?? string.Empty; + + // NOTE: $$"""...""" (double-$) raw string. Inside, single { } is literal + // (matters for the JSON braces below), and {{var}} is interpolation. + // Plain $"""...""" would reject `{{ ... }}` as literal with CS9006. + var prompt = $$""" + You are scoring an assistant response against the rubric below. + + ## Rubric + {{_rubric}} + + ## User query + {{userQuery}} + + ## Assistant response + {{responseText}} + + Respond with strict JSON: { "score": <1-5 int>, "rationale": "<1-2 sentences>" }. + 5 = perfectly satisfies every rubric clause. 1 = ignores the rubric. + """; + + var judge = await chatConfiguration.ChatClient.GetResponseAsync( + prompt, cancellationToken: cancellationToken).ConfigureAwait(false); + + // Tolerant parse: fall back to Inconclusive on bad JSON. + var (score, rationale) = TryParse(judge.Text ?? ""); + + var metric = new NumericMetric(MetricName, value: score) + { + Interpretation = new EvaluationMetricInterpretation( + rating: score switch { >= 4 => EvaluationRating.Good, + 3 => EvaluationRating.Average, + _ => EvaluationRating.Poor }, + reason: rationale) + }; + return new EvaluationResult(metric); + } + + private static (int score, string rationale) TryParse(string raw) + { + try + { + using var doc = JsonDocument.Parse(raw.Trim().Trim('`', ' ', '\n', '\r')); + return (doc.RootElement.GetProperty("score").GetInt32(), + doc.RootElement.GetProperty("rationale").GetString() ?? ""); + } + catch { return (3, "Judge response was unparseable; defaulted to Average."); } + } +} +``` + +Wire it from `Reporting/ReportingConfig.cs`'s judge-tier evaluator +list **alongside** Relevance/Coherence/Fluency (drop Completeness + +Equivalence per the pitfall guidance): + +```csharp +// In ReportingConfig.ForQuality(), when EvalEnv.UseRealJudge: +evaluators.Add(new RelevanceEvaluator()); +evaluators.Add(new CoherenceEvaluator()); +evaluators.Add(new FluencyEvaluator()); +evaluators.Add(RubricEvaluator.FromFile( + Path.Combine(AppContext.BaseDirectory, "Quality", "rubric.md"))); +// CompletenessEvaluator + EquivalenceEvaluator deliberately dropped +// for this app — see common-pitfalls.md tuning section. +``` + +Ensure `Quality/rubric.md` is copied to output by adding to the csproj: + +```xml + + + +``` + +This `RubricFit` metric will appear in `report.html` alongside the +built-ins, and the rationale shows in the per-scenario detail drawer. +Update `Reporting/MetricsGlossary.cs`'s `QualityEntries` constant to +add a one-line `RubricFit` definition pointing at `Quality/rubric.md`. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md new file mode 100644 index 0000000000..ad7ac0b3ec --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -0,0 +1,226 @@ +# IChatClient detection + +The skill auto-detects how the target app registers its `IChatClient` +and emits `Wire/AgentChatClientFactory.cs` so `EVAL_USE_REAL_AGENT=1` +works without further code. Detection result is surfaced in step 2 +(scope confirmation) and can be overridden by the user. + +## Patterns to scan for + +Scan **`*.AppHost.csproj` directory** and **all `*.Agent*.csproj` +directories**. Match (case-insensitive, multi-line): + +| Pattern (regex-ish) | Inferred client | Example file:line | +|---------------------|-----------------|--------------------| +| `AddAzureOpenAIChatClient\s*\(` or `AddAzureOpenAIClient\s*\(` | Azure OpenAI | `AppHost.cs`, `Program.cs` | +| `AddOpenAIChatClient\s*\(` | OpenAI direct | `Program.cs` | +| `AddOllamaChatClient\s*\(` | Ollama | `Program.cs` | +| `AddAIInference\s*\(` (Foundry deployment alias) | Azure AI Foundry | `AppHost.cs` | +| `AddAzureChatCompletionsClient\s*\([^)]*\)\s*\.AddChatClient\s*\(` | Aspire `Aspire.Azure.AI.Inference` (Foundry-routed, **legacy — see deprecation note below**) | `Program.cs` | +| `AddOpenAIClient\s*\([^)]*\)\s*\.AddChatClient\s*\(` | Aspire OpenAI/v1 (Foundry-routed via the OpenAI SDK) | `Program.cs` | +| `services\.AddSingleton` (any explicit registration) | custom | varies | +| `\.AsIChatClient\(\)` (after an SDK client) | manual wrap | varies | + +> The `AddAzureChatCompletionsClient(...).AddChatClient(...)` chain is the +> Aspire 13.2 way of wiring an `IChatClient` against a Foundry chat +> deployment via the `Azure.AI.Inference` beta SDK. The argument is the +> **connection-string name**, which Aspire's AppHost populates +> automatically (`AddDeployment("chat", ...)` -> connection string `chat`). +> The factory mirrors both calls verbatim. +> +> ⚠️ **Deprecation note (as of 2026):** the `Azure.AI.Inference` beta SDK +> is being retired on **August 26, 2026** in favor of the GA OpenAI/v1 +> API + stable OpenAI SDK. New apps should prefer the +> `AddOpenAIClient(...).AddChatClient(...)` pattern against an OpenAI/v1 +> endpoint, which avoids several quirks (notably the reasoning-model +> `max_tokens` rejection documented in +> `references/common-pitfalls.md`). The skill still detects and supports +> the legacy pattern unchanged. See the +> [Foundry migration guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/model-inference-to-openai-migration). + +Capture the deployment alias / model id literal if present (e.g., +`builder.AddAIInference("chat", "gpt-4o-mini")` → alias `chat`). + +## What to emit + +### Case A — exactly one registration found + +Emit a factory that resolves from the host: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Resolves the same IChatClient the app uses, by building a minimal + /// host that mirrors the app's DI registration. + /// Detected: {{DetectionSummary}} at {{File}}:{{Line}} + /// + public static IChatClient Create() + { + var builder = Host.CreateApplicationBuilder(); + + // In test hosts (`dotnet test`), the entry assembly is testhost.exe so + // user-secrets are NOT auto-loaded by CreateApplicationBuilder. + // Add them explicitly from THIS assembly's UserSecretsId. + builder.Configuration.AddUserSecrets(typeof(AgentChatClientFactory).Assembly, optional: true); + + // {{InsertDetectedRegistrationCallVerbatim}} + var host = builder.Build(); + try + { + return host.Services.GetRequiredService(); + } + catch (InvalidOperationException ex) + { + throw new InvalidOperationException( + "EVAL_USE_REAL_AGENT=1 but IChatClient could not be resolved. " + + "The detected registration ({{DetectionSummary}}) reads connection " + + "string \"{{ConnStrName}}\" from configuration. Aspire's AppHost " + + "populates this at runtime, but `dotnet test` runs standalone. " + + "Wire one of:\n" + + " # Key-based auth (only if the resource has it enabled):\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + " # Entra-ID auth (DefaultAzureCredential — works when key auth is disabled):\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + "Note: the hostname strips dashes from the resource name " + + "(resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`).\n" + + "Get the exact endpoint from `azd env get-values` or " + + "`az cognitiveservices account show -n -g --query properties.endpoints`.", + ex); + } + } +} +``` + +Where `{{InsertDetectedRegistrationCallVerbatim}}` is the literal call +copied from the detection source (with any required `using`s in scope +via `GlobalUsings.cs`), and `{{ConnStrName}}` is the connection-string +literal extracted from the call (e.g., the `"chat"` argument). + +### Case B — multiple registrations found + +Emit the same factory but with a comment listing all candidates, and +have the chat output ask the user to pick one before write. Do **not** +write the file until confirmation. + +### Case C — no registration found + +Emit a stub factory the user fills in: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Auto-detection failed: no IChatClient registration found in + /// AppHost or agent projects. Wire your client manually. + /// + public static IChatClient Create() => + throw new NotImplementedException( + "Wire your IChatClient here. See https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai"); +} +``` + +## Runtime selection + +`ReportingConfig` picks agent client based on tier: + +```csharp +internal static IChatClient ResolveAgentClient() => + EvalEnv.UseRealAgent ? AgentChatClientFactory.Create() : new StubChatClient(); +``` + +And by default the judge client is the **same** instance as the agent +client. This saves a duplicate Azure credential setup AND lets the +MEAI response cache serve both the agent call and the judge call from +one cache (because `QualityTests` uses `run.ChatConfiguration!.ChatClient` +for the agent call — that's the cached wrapper around the shared +instance). The user can override by setting `EVAL_JUDGE_DEPLOYMENT_NAME` +to a different deployment alias when (e.g.) the production model is a +reasoning model that can't be used as a judge. Trade-off: with the +override active, `run.ChatConfiguration!.ChatClient` becomes the +**judge** client, so the agent call would silently use the judge +model. To avoid that, `QualityTests` falls back to +`Wire.ResolveAgentClient()` (uncached) for agent calls whenever the +override is set; cache hits then apply only to judge calls. + +## Connection-string setup for standalone test runs + +When the target app uses Aspire orchestration, the AppHost populates +`ConnectionStrings:` automatically. **`dotnet test` runs outside +the AppHost and does not get this for free.** Surface this in the chat +output any time the detected pattern reads from a connection string +(`AddAzureChatCompletionsClient`, `AddAIInference`, `AddOllamaChatClient`, +or `AddAzureOpenAIChatClient` without an explicit endpoint). + +> **Important — the factory must opt into user-secrets explicitly.** In a +> `dotnet test` process the entry assembly is `testhost.exe`, so the +> `dotnet user-secrets` payload tied to *your* `UserSecretsId` is NOT auto +> loaded by `Host.CreateApplicationBuilder()`. The Case A template above +> calls `builder.Configuration.AddUserSecrets(typeof(...).Assembly)` for +> this reason. Without that line, the secret is set on disk but never read. + +Recommend the user wire one of: + +```pwsh +# 0 — one-time: bind the test project to a secrets store +dotnet user-secrets init --project .Evals.Tests + +# Option A — Key-based auth (only if the resource has key auth enabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId=" ` + --project .Evals.Tests + +# Option B — Entra-ID auth (DefaultAzureCredential; works when key auth disabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests +# requires `az login` and a Cognitive Services User / Azure AI User role +# on the resource for the signed-in identity. + +# Option C — env var (works in CI without a secrets file) +$env:ConnectionStrings__ = "Endpoint=;DeploymentId=" +``` + +**Two endpoint gotchas to call out in chat:** + +1. The `services.ai.azure.com/models` hostname **strips dashes** from the + resource name. Resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`. + Use `az cognitiveservices account show -n -g --query properties.endpoints` + to see all valid endpoint hostnames (`AI Foundry API` / `Azure AI Model Inference API`). +2. If the resource has `disableLocalAuth=true` (common on Foundry resources + provisioned by Aspire/azd), key-based auth returns `403 Key based authentication + is disabled for this resource`. Drop the `Key=` segment and use Entra (Option B). + +For Foundry-routed clients the connection string is what `azd env get-values` +prints for `connectionString` against the deployment resource. Document this +in the chat output along with the tier banner so the user doesn't see a +silent NRE on first real-agent run. + +## Chat output (step 2) + +When detection succeeds, surface as: + +``` +IChatClient detection: + - AddAzureOpenAIChatClient at AppHost.cs:41 + - Deployment alias: "chat" → gpt-4o-mini + - Will generate Wire/AgentChatClientFactory.cs that resolves it from the host. + +Override [y/N]? +``` + +When detection fails: + +``` +IChatClient detection: no registration found. + - Will generate a stub factory you'll need to fill in. + - Tier 2 (Quality) and Tier 3 (Safety) will be skipped until wired. +``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md new file mode 100644 index 0000000000..c212a732cd --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md @@ -0,0 +1,247 @@ +# Metrics glossary + +Source of truth for the per-run `metrics-glossary.md` artifact that the +scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes next +to `report.html`. The aieval HTML report is data-bound JSON — it shows +numbers but no definitions — so we co-locate this glossary with the +report so a first-time reader has a one-page cheat-sheet. + +The skill emits **only the entries for evaluators that actually ran** in +the active tier, so a stub-tier user sees Words/BLEU/GLEU/F1 and +nothing else. + +## NLP tier (deterministic, no LLM) + +### `Words` +- **Custom evaluator** scaffolded by this skill (see `evaluators-catalog.md`). +- **What it measures:** word count of the response text (whitespace-separated tokens, per `WordCountEvaluator`). +- **Scale:** integer ≥ 0. +- **Interpretation:** `< 5` → Poor (response too short / empty). `5–500` → Good. `> 500` → Average (response unusually long). +- **Trust for:** sanity-checking that the model produced *anything* and isn't running away with a 10-page essay. +- **Don't trust for:** quality. A 50-word wrong answer scores the same as a 50-word correct answer. + +### `BLEU` — Bilingual Evaluation Understudy +- **What it measures:** n-gram (1-4) overlap between the response and one or more reference strings, with a brevity penalty. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.1 weak overlap, often paraphrased; ~0.1–0.3 normal for free-form generation; ~0.3–0.5 strong; > 0.5 near-quotation. +- **Trust for:** "is the response in the same lexical neighbourhood as the reference?" — useful as a regression signal when the reference is canonical. +- **Don't trust for:** semantic correctness. A correct paraphrase scores low; an incorrect copy-paste of reference fragments scores high. +- **Reference:** [`BLEUEvaluator` in MEAI.Evaluation.NLP](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.bleuevaluator). + +### `GLEU` — Google-BLEU +- **What it measures:** sentence-level BLEU variant. Symmetric — penalises both missing reference n-grams and extra invented ones. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** same buckets as BLEU; GLEU usually tracks BLEU but is less brittle on short outputs. +- **Trust for:** the same use cases as BLEU when individual scenarios are short (1-2 sentences) and BLEU's brevity penalty would be misleading. +- **Don't trust for:** any semantic claim. Same caveats as BLEU. +- **Reference:** [`GLEUEvaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.gleuevaluator). + +### `F1` — Token-level F1 +- **What it measures:** harmonic mean of unigram precision and recall against the ground-truth string. Order-insensitive. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.2 mostly-disjoint vocab; 0.3–0.5 typical for free-form generation; > 0.6 strong word-level match. +- **Trust for:** QA / extraction-style scenarios where the *set of words* matters more than phrasing (SQuAD-style benchmarks use this). +- **Don't trust for:** word-order-sensitive tasks (e.g., "yes" vs "no" answers buried in long output) — the F1 score will be high even if the polarity is wrong. +- **Reference:** [`F1Evaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.f1evaluator). + +> **NLP-tier headline:** all three of BLEU/GLEU/F1 are *lexical* metrics. +> They're cheap, deterministic, and free, but they cannot tell you the +> response is *correct* — only that it's *similar in wording*. Treat them +> as a regression early-warning, not a quality verdict. + +## Quality tier (LLM-as-judge — `EVAL_USE_REAL_JUDGE=1`) + +All Quality metrics produce a 1-5 `EvaluationRating` (Poor → Excellent) +plus a free-text rationale from the judge model. + +### `Relevance` +- **What it measures:** does the response actually address the user's query? +- **Trust for:** catching off-topic regressions (model started monologuing about a different topic). +- **Don't trust for:** factual correctness — a *relevantly wrong* answer can score high. + +### `Coherence` +- **What it measures:** is the response logically structured / orderly / easy to follow? +- **Trust for:** detecting rambling, contradictory, or logically broken outputs. +- **Don't trust for:** depth or correctness — coherent nonsense scores high. + +### `Fluency` +- **What it measures:** grammar, readability, naturalness. +- **Trust for:** detecting broken-English / token-soup outputs from undertrained or quantised models. +- **Don't trust for:** anything beyond surface text quality. A fluent lie scores high. + +### `Completeness` +- **What it measures:** how comprehensive and accurate the response is, given a reference (`CompletenessEvaluatorContext(groundTruth)`). +- **Trust for:** catching responses that are correct but partial (covered 2 of 4 required points). +- **Don't trust for:** brevity-as-a-feature scenarios — a short-but-correct answer can score lower than a long-and-padded one. + +### `Equivalence` +- **What it measures:** semantic similarity between response and ground truth in the context of the original query. +- **Trust for:** distinguishing "right answer, different words" (high) from "wrong answer" (low). Better than BLEU/F1 when paraphrasing is acceptable. +- **Don't trust for:** any case where the ground truth is itself ambiguous or one of multiple valid answers. + +### `Groundedness` +- **What it measures:** alignment between the response and a supplied source-of-truth context (`GroundednessEvaluatorContext(context)`). +- **Trust for:** RAG pipelines — flags when the model invented facts not in the retrieved context. +- **Don't trust for:** open-ended chat without a context document. + +### Agentic-only + +The skill wires these only when an `*.AppHost.csproj` is detected. + +- **`Intent Resolution`** — did the model identify and resolve the user's actual intent (vs answering a related-but-different question)? +- **`Task Adherence`** — did the model stick to the assigned task or wander into other agent territory? +- **`Tool Call Accuracy`** — did the model invoke the right tools with the right arguments? Requires `ToolCallAccuracyEvaluatorContext`. + +> **Quality-tier headline:** these are LLM-judge subjective scores. They +> drift across judge model versions. Pin the judge model in +> `quality.thresholds.json` if you want comparable scores across runs. + +## Safety tier (Foundry — `EVAL_USE_FOUNDRY_SAFETY=1`) + +All Safety evaluators return a 1-5 severity (1 = safe, 5 = severe harm) +plus a confidence score. Each metric is an *output classifier* — it +inspects the model's response, not the input prompt. + +### `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` +- Wired together as the single-shot `ContentHarmEvaluator` (one Foundry call → all 4 metrics). +- **Trust for:** detecting harmful content in agent outputs. +- **Don't trust for:** input-side filtering — these never see the user's prompt. Pair with input-side Azure AI Content Safety for full coverage. + +### `Protected Material` +- Detects copyrighted text / song lyrics / book passages reproduced in the output. + +### `Indirect Attack` +- Detects prompt-injection-style content that would indicate the model picked up an indirect attack from retrieved content or tool output. Closest thing to an input-side check available in this tier. + +### `Code Vulnerability` +- Flags vulnerable patterns in code the model emitted (SQL injection, hardcoded credentials, weak crypto, etc.). + +### `Ungrounded Attributes` +- Detects inferred human attributes (race, gender, age, religion, etc.) in the response that weren't in the input. + +### `Groundedness Pro` +- Foundry-hosted fine-tuned groundedness evaluator. More accurate than the open-source `GroundednessEvaluator` but costs a Foundry call per scenario. + +> **Safety-tier headline:** all safety metrics are *output classifiers*. +> They protect downstream consumers from the agent's outputs; they do not +> protect the agent from its inputs. + +## Quick reference card + +| Metric | Tier | Scale | Needs ground truth? | Needs LLM? | +|--------|------|-------|---------------------|-----------| +| Words | NLP | int | no | no | +| BLEU | NLP | 0-1 | yes (references) | no | +| GLEU | NLP | 0-1 | yes (references) | no | +| F1 | NLP | 0-1 | yes (one string) | no | +| Relevance / Coherence / Fluency | Quality | 1-5 | no | yes (judge) | +| Completeness / Equivalence | Quality | 1-5 | yes (one string) | yes (judge) | +| Groundedness | Quality | 1-5 | yes (context) | yes (judge) | +| Intent Resolution / Task Adherence | Quality (agentic) | 1-5 | no | yes (judge) | +| Tool Call Accuracy | Quality (agentic) | 1-5 | yes (expected_tool_calls) | yes (judge) | +| Content Harm bundle (4 metrics) | Safety | 1-5 severity | no | yes (Foundry) | +| Protected / Indirect / Code Vuln. / Ungrounded | Safety | 1-5 severity | varies | yes (Foundry) | +| Groundedness Pro | Safety | 1-5 | yes (context) | yes (Foundry) | + +## `MetricsGlossary.cs` template + +The scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes +the tier-relevant slice of this glossary next to `report.html` after +each `dotnet test` run. + +> **MSTest constraint:** an assembly may declare **only one** +> `[AssemblyCleanup]` method. The skill emits `MetricsGlossary` as a +> plain static class (no `[TestClass]`, no `[AssemblyCleanup]`) and +> chains `MetricsGlossary.WriteGlossary()` from +> `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`. +> Wrap the call in a `try/catch` so a glossary-write failure never +> masks the report. + +```csharp +internal static class MetricsGlossary +{ + public static void WriteGlossary() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var path = Path.Combine(outDir, "metrics-glossary.md"); + + var sb = new StringBuilder(); + sb.AppendLine($"# Metrics glossary — {EvalEnv.Tier} tier"); + sb.AppendLine(); + sb.AppendLine($"Generated: {DateTime.UtcNow:O}"); + sb.AppendLine(); + sb.AppendLine($"Companion to `report.html` in this folder. The aieval HTML report shows numbers; this file explains them."); + sb.AppendLine(); + + sb.AppendLine(NlpEntries); + if (EvalEnv.UseRealJudge) sb.AppendLine(QualityEntries); + if (EvalEnv.UseFoundrySafety) sb.AppendLine(SafetyEntries); + + sb.AppendLine(); + sb.AppendLine("> Source: setup-maf-evals references/metrics-glossary.md"); + + File.WriteAllText(path, sb.ToString()); + Console.WriteLine($"[MetricsGlossary] {path}"); + } + + private const string NlpEntries = """ + ## NLP tier (deterministic, no LLM) + - **Words** (int): response length sanity check. <5 too short, 5-500 ok, >500 long. + - **BLEU** (0-1): n-gram overlap with reference(s). 0.1-0.3 normal, >0.3 strong, >0.5 near-quotation. *Lexical, not semantic.* + - **GLEU** (0-1): sentence-level BLEU; better for short outputs. Same buckets as BLEU. + - **F1** (0-1): unigram token F1 vs ground-truth. 0.3-0.5 typical, >0.6 strong word-level match. Order-insensitive. + + > Headline: NLP metrics measure wording similarity, not correctness. Use for regression early-warning, not as quality verdicts. + """; + + private const string QualityEntries = """ + ## Quality tier (LLM-as-judge) + Each rated 1-5 (Poor → Excellent) with a free-text rationale. + - **Relevance**: addresses the user's query. Catches off-topic regressions. + - **Coherence**: logically structured. Catches rambling/contradictory outputs. + - **Fluency**: grammar/readability. Catches broken-English outputs. + - **Completeness** (needs reference): comprehensive and accurate. + - **Equivalence** (needs reference): semantic similarity in context of the query. + - **Groundedness** (needs context): aligned with supplied source-of-truth. + - **Intent Resolution / Task Adherence / Tool Call Accuracy** (agentic only). + + > Headline: judge scores drift across model versions. Pin the judge model for comparable runs. + """; + + private const string SafetyEntries = """ + ## Safety tier (Foundry) + Each rated 1-5 severity (1 safe → 5 severe). + - **ContentHarm bundle** (single-shot, 4 metrics): Hate-And-Unfairness, Self-Harm, Violence, Sexual. + - **Protected Material**: copyrighted text reproduced. + - **Indirect Attack**: prompt-injection content from retrieved/tool data. + - **Code Vulnerability**: vulnerable code patterns (SQLi, weak crypto, etc.). + - **Ungrounded Attributes**: inferred human attributes not in input. + - **Groundedness Pro**: Foundry-hosted fine-tuned groundedness check. + + > Headline: all safety metrics inspect *outputs*, not inputs. Pair with Azure AI Content Safety on the request side for full coverage. + """; +} +``` + +Then in `Reporting/AievalReport.cs` (the assembly's single +`[AssemblyCleanup]` host): + +```csharp +[TestClass] +public static class AievalReport +{ + [AssemblyCleanup] + public static void GenerateReport() + { + // ... aieval invocation ... + + try { MetricsGlossary.WriteGlossary(); } + catch (Exception ex) { Console.Error.WriteLine($"[MetricsGlossary] Failed: {ex.Message}"); } + } +} +``` + +The skill should emit both files verbatim — change the `private const` +strings only if upstream MEAI changes a metric definition. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md new file mode 100644 index 0000000000..76d87b11ef --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md @@ -0,0 +1,203 @@ +# Project template (MSTest shape) + +> **For the default scaffold, read `references/default-scaffold.md` instead** — +> it consolidates the complete, copy-pasteable body of every default-mode file +> (Telemetry + Quality + NLP + Reporting + Wire) into one doc so you create the +> whole project from a single read. This file remains the reference for the file +> tree, the `.csproj` layout, and the version policy. + +The scaffold creates `.Evals.Tests` as a **MSTest** project. This +matches the +[upstream Learn doc tutorial](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) +and the [dotnet/ai-samples evaluation unit tests](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/), +which is the canonical pattern for `Microsoft.Extensions.AI.Evaluation`. + +A console-runner shape is available behind an explicit +`--shape console` flag for users who can't take an MSTest dependency, +but is no longer the default — every CI system understands `dotnet test` +out of the box, and Test Explorer integration is automatic. + +## File tree + +``` +.Evals.Tests/ + .Evals.Tests.csproj + .config/dotnet-tools.json + GlobalUsings.cs + Reporting/ + ReportingConfig.cs # DiskBasedReportingConfiguration factory; tier-aware evaluator list + Tier.cs # EvalTier enum + EvalEnv reader + AievalReport.cs # [AssemblyCleanup] that invokes the dotnet tool + Wire/ + AgentChatClientFactory.cs # auto-generated from IChatClient detection + StubChatClient.cs # used when EVAL_USE_REAL_AGENT is unset + Telemetry/ + TelemetryTests.cs + inputs.json + prices.json + Quality/ + QualityTests.cs + rubric.md + golden.json + Compare/ + CompareTests.cs + matrix.json + Safety/ # only if user opted in + SafetyTests.cs + quality.thresholds.json +.github/ + workflows/ + evals.yml # optional, opt-in +``` + +`.gitignore` additions (idempotent): + +``` +# setup-maf-evals +.copilot/perf-reports/evals/ +.Evals.Tests/_store/ +``` + +## `.csproj` template + +The `.csproj` is **not hand-written**. SKILL.md step 3 generates the shell with +`dotnet new mstest` (which owns the MSTest / Microsoft.Testing.Platform +references at their SDK-current versions) and then adds the eval + hosting +packages with `dotnet add package` **without hand-authored versions** (see the +version policy below). The listing here is an **illustrative expected result** — +the source of truth for the *eval + hosting* package **set** (not their exact +versions) — and the reconciliation target (TFM, data-file +`CopyToOutputDirectory`, and the agent `ProjectReference`) after `dotnet new` +runs. Do not paste it verbatim over the generated file; let the template own the +test-SDK lines and only add/adjust what is shown. + +**The test-SDK line(s) vary by installed SDK — do not hand-maintain them.** On +.NET 10 (`dotnet new mstest`) the template emits a single MTP-native +`MSTest` metapackage (e.g. `4.0.1`) and **no** `Microsoft.NET.Test.Sdk` +reference; on older SDKs it emits `MSTest` 3.x **plus** `Microsoft.NET.Test.Sdk`. +The block below shows the .NET 10 single-package form. Whatever the template +produces is authoritative; the skill never rewrites it. + +```xml + + + net10.0 + enable + enable + false + {{AppName}}.Evals.Tests + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +**Version policy (no hand-authored version literals):** + +- **GA** packages — `Microsoft.Extensions.AI`, `Microsoft.Extensions.AI.Evaluation`, + `.Quality`, `.Reporting` — are added with `dotnet add package ` (no + `--version`), so they resolve to the **latest stable**. +- **Preview** packages — `.NLP` (opt-in-on) and `.Safety` (opt-in-off) — have no + stable release yet, so they are added with `dotnet add package --prerelease` + (**latest prerelease**). No specific preview build is pinned. +- **Hosting/config** — `Microsoft.Extensions.Hosting` and + `Microsoft.Extensions.Configuration.*` — are also added at latest stable. They + must resolve to **>= 10.0.1** (not `10.0.0`) to satisfy the transitive + constraint from `Microsoft.Agents.AI.Hosting`; a pin at `10.0.0` produces + `NU1605`. Latest stable is always >= that floor, so no explicit pin is needed — + this is a *floor*, not a hand-maintained version. + +`dotnet add package` writes the concrete resolved version into the `.csproj`; the +skill never authors those numbers. The versions shown in the block above are an +illustrative snapshot from one scaffold run, not values to keep in sync. + +> **Version-less pre-listing (scaffold determinism).** SKILL.md step 3.2 may +> pre-list the eval + hosting package **IDs** as version-less +> `` entries (no `Version` attribute) while +> creating files, so the correct package *set* is on disk before the +> network-bound `dotnet add package` (step 3.3) stamps each resolved version. +> This is still "no hand-authored version" — a version-less ref carries no +> literal, and `dotnet add package ` updates the existing entry in place +> rather than duplicating it. + +## Tool manifest (`dotnet-tools.json`) + +Do **not** hand-write this file with a pinned version. Generate it (unpinned): + +```pwsh +dotnet new tool-manifest # if none exists +dotnet tool install microsoft.extensions.ai.evaluation.console # latest; provides `aieval` +``` + +`dotnet tool install` (no `--version`) records the current tool version in the +manifest. `dotnet new tool-manifest` places it per the SDK default (a +`.config/dotnet-tools.json`, or a repo-root `dotnet-tools.json` on newer SDKs); +`dotnet tool restore` finds it either way. The illustrative manifest below shows +the resulting shape: + +```json +{ + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } +} +``` + +After scaffold: `dotnet tool restore` (the skill runs this automatically). + +## `GlobalUsings.cs` + +```csharp +global using Microsoft.Extensions.AI; +global using Microsoft.Extensions.AI.Evaluation; +global using Microsoft.Extensions.AI.Evaluation.Reporting; +global using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; +global using Microsoft.Extensions.AI.Evaluation.NLP; +global using Microsoft.Extensions.AI.Evaluation.Quality; +global using Microsoft.Extensions.Configuration; // AddUserSecrets in AgentChatClientFactory +global using Microsoft.Extensions.DependencyInjection; // GetRequiredService in AgentChatClientFactory +global using Microsoft.Extensions.Hosting; // Host.CreateApplicationBuilder +global using Microsoft.VisualStudio.TestTools.UnitTesting; +``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md new file mode 100644 index 0000000000..128dcc3a5b --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md @@ -0,0 +1,258 @@ +# Quality mode + +`Quality/QualityTests.cs` is the MSTest class that actually drives +`Microsoft.Extensions.AI.Evaluation.Reporting`. It's the **only** +runner that produces `report.html`. + +## Response cache (read first) + +`DiskBasedReportingConfiguration.Create(..., enableResponseCaching: true)` +wraps the supplied `IChatClient` with a content-addressable cache stored +under `_store/cache/`. The first `dotnet test` run populates it; every +subsequent run against the same scenarios is **near-instant with zero +LLM cost** because both the agent call and the judge call are served +from disk. + +Two rules MUST be followed to make the cache work: + +1. **The agent call must go through `run.ChatConfiguration!.ChatClient`** + (NOT `Wire.ResolveAgentClient()` directly). The run-scoped client is + the cached wrapper. Calling the factory directly bypasses the cache. +2. **Do not pass a per-run `executionName`** to `Create(...)`. The + `executionName` is part of the cache scope; a fresh timestamp per run + guarantees misses. Use a separate `EvalEnv.ReportFolder` value for + the report-output directory if you want per-run history. + +If both rules are followed, the only legitimate reasons to see Miss in +the report's Diagnostic Data section are: (a) the cache directory was +deleted, (b) the rubric / golden / scenario input changed, (c) the +judge model or chat options changed, or (d) it's the first run. + +## Pipeline + +``` +[ClassInitialize] → build ReportingConfig (tier-aware evaluator list) +[TestMethod] → one per golden.json entry + ├─ CreateScenarioRunAsync(scenarioName) + ├─ get cached IChatClient from run.ChatConfiguration + ├─ get agent response (cached) + ├─ build per-evaluator EvaluationContext (BLEU refs, F1 ground truth, ...) + └─ scenarioRun.EvaluateAsync(messages, response, contexts) // judge calls cached +[AssemblyCleanup] → dotnet tool run aieval report --path _store --output /report.html +``` + +## Reporting config (sketch) + +```csharp +// Reporting/ReportingConfig.cs +internal static class ReportingConfig +{ + public static readonly string StorageRoot = + // _store lives next to the test project (the directory that holds + // .config/dotnet-tools.json) so the runtime AssemblyCleanup report + // call, the CI safety-net step, and `dotnet tool run aieval` — all + // executed from the project directory — resolve the same store. + public static readonly string StorageRoot = + Path.Combine(ProjectRoot.Find(), "_store"); + + public static ReportingConfiguration ForQuality() + { + // ONE client serves both the agent call and the judge call. MEAI wraps + // it with the response cache when handed to ChatConfiguration, so the + // *agent* call gets cached too when QualityTests calls it through + // `run.ChatConfiguration!.ChatClient` (see "Test class" below). + // On re-runs against unchanged inputs, the entire run is a cache hit + // and finishes in seconds with zero LLM cost. Override the judge with + // a separate model via EVAL_JUDGE_DEPLOYMENT_NAME (advanced). + var agent = Wire.ResolveAgentClient(); + var judge = Wire.ResolveJudgeClient(agent); + + var evaluators = new List + { + // Tier 1 — always on, deterministic + new WordCountEvaluator(), + new BLEUEvaluator(), + new GLEUEvaluator(), + new F1Evaluator(), + }; + + if (EvalEnv.UseRealJudge) + { + // Tier 2 — needs real judge + evaluators.Add(new RelevanceEvaluator()); + evaluators.Add(new CoherenceEvaluator()); + evaluators.Add(new FluencyEvaluator()); + evaluators.Add(new CompletenessEvaluator()); + evaluators.Add(new EquivalenceEvaluator()); + evaluators.Add(new GroundednessEvaluator()); + + if (AgenticAppDetected) + { + evaluators.Add(new IntentResolutionEvaluator()); + evaluators.Add(new TaskAdherenceEvaluator()); + evaluators.Add(new ToolCallAccuracyEvaluator()); + } + } + + // executionName: deliberately omitted. MEAI's default keeps the cache + // scope stable across runs so re-runs hit. The report folder + // timestamp lives separately in EvalEnv.ReportFolder. + return DiskBasedReportingConfiguration.Create( + storageRootPath: StorageRoot, + evaluators: evaluators, + chatConfiguration: new ChatConfiguration(judge), + enableResponseCaching: true); + } +} +``` + +## Test class (sketch) + +```csharp +[TestClass] +public sealed class QualityTests +{ + private static ReportingConfiguration s_reporting = null!; + + [ClassInitialize] + public static void Init(TestContext _) => + s_reporting = ReportingConfig.ForQuality(); + + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); + + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task Evaluate(GoldenItem g) + { + var scenarioName = $"{nameof(QualityTests)}.{g.Id}"; + await using var run = await s_reporting.CreateScenarioRunAsync(scenarioName); + + // IMPORTANT: use the run's ChatClient, NOT Wire.ResolveAgentClient(), + // for the agent call. The run-scoped client is wrapped with MEAI's + // response cache (set up by ReportingConfig.ForQuality), so identical + // inputs across runs return cached responses instead of paying for + // a fresh LLM call. Calling AgentChatClientFactory directly bypasses + // the cache and guarantees a cache miss on every judge call too + // (judge cache key includes the agent response, which then varies). + // + // EDGE CASE: when EVAL_JUDGE_DEPLOYMENT_NAME splits judge from agent, + // run.ChatConfiguration.ChatClient IS the judge client — using it as + // the agent would silently call the wrong model. Fall back to the + // uncached agent factory in that case (judge calls still cache). + var agent = string.IsNullOrEmpty( + Environment.GetEnvironmentVariable("EVAL_JUDGE_DEPLOYMENT_NAME")) + ? run.ChatConfiguration!.ChatClient + : Wire.ResolveAgentClient(); + var messages = new List + { + new(ChatRole.System, RubricLoader.SystemPrompt()), + new(ChatRole.User, g.UserMessage), + }; + var response = await agent.GetResponseAsync(messages); + + var contexts = new List(); + if (!string.IsNullOrEmpty(g.ReferenceResponse)) + { + contexts.Add(new BLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new GLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new F1EvaluatorContext(g.ReferenceResponse)); + contexts.Add(new EquivalenceEvaluatorContext(g.ReferenceResponse)); + contexts.Add(new CompletenessEvaluatorContext(g.ReferenceResponse)); + } + if (!string.IsNullOrEmpty(g.Context)) + contexts.Add(new GroundednessEvaluatorContext(g.Context)); + + var result = await run.EvaluateAsync(messages, response, contexts); + Thresholds.ApplyOrLog(result, g.Id); // hard_fail in JSON => Assert.Fail + } +} +``` + +## Report generation + +```csharp +// Reporting/AievalReport.cs +[TestClass] +public static class AievalReport +{ + [AssemblyCleanup] + public static void GenerateReport() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", + EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var html = Path.Combine(outDir, "report.html"); + + var psi = new ProcessStartInfo("dotnet", + $"tool run aieval report --path \"{ReportingConfig.StorageRoot}\" --output \"{html}\"") + { + WorkingDirectory = ProjectRoot.Find(), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + using var p = Process.Start(psi)!; + p.WaitForExit(); + Console.WriteLine($"Eval report: {html}"); + } +} +``` + +## EvalEnv (sketch, in `Reporting/Tier.cs`) + +```csharp +internal static class EvalEnv +{ + public static bool UseRealAgent => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_AGENT") == "1"; + + public static bool UseRealJudge => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_JUDGE") == "1"; + + public static bool UseFoundrySafety => + Environment.GetEnvironmentVariable("EVAL_USE_FOUNDRY_SAFETY") == "1"; + + public static string Tier => + UseFoundrySafety ? "Safety" : UseRealJudge ? "Judge" : "Stub"; + + // Per-run timestamp ONLY for the report output folder under + // .copilot/perf-reports/evals//. NOT passed to MEAI's + // ReportingConfiguration — that would scope the response cache per + // run and defeat caching. Override with EVAL_REPORT_FOLDER in CI to + // align with a build number. + public static readonly string ReportFolder = + Environment.GetEnvironmentVariable("EVAL_REPORT_FOLDER") + ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); +} +``` + +## `golden.json` schema (v2) + +```json +{ + "schema_version": 2, + "scenarios": [ + { + "id": "g-receptionist-greeting", + "user_message": "Hi, I'd like to start an interview.", + "reference_response": "Hello! I'd be happy to start an interview with you. What role are you preparing for?", + "context": null, + "expected_traits": ["on_topic", "safe", "format_correct"], + "expected_tool_calls": null + } + ] +} +``` + +- `reference_response`: required for BLEU/GLEU/F1/Equivalence/Completeness. +- `context`: required for Groundedness. Free text providing the + source-of-truth context the response should be grounded in. +- `expected_traits`: free-form labels surfaced in the report rubric + view. Read by the LLM judge. +- `expected_tool_calls`: required for `ToolCallAccuracyEvaluator`. + +Migration from v1 (no `schema_version`, no `reference_response`): the +skill adds the fields as `null` so existing tests don't fail. NLP +evaluators emit `(no reference)` when null. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md new file mode 100644 index 0000000000..e8e61106e8 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md @@ -0,0 +1,90 @@ +# Safety mode (opt-in) + +Off by default. Enabled in step 2 of the workflow when the user picks +"include Safety tier" / says "wire safety" / equivalent. + +When enabled, the skill: + +1. Adds `Microsoft.Extensions.AI.Evaluation.Safety` (preview) to the csproj via + `dotnet add package Microsoft.Extensions.AI.Evaluation.Safety --prerelease`. +2. Generates `Safety/SafetyTests.cs` using `ContentHarmEvaluator` as the + default bundle (4 metrics in 1 Foundry call), plus + `ProtectedMaterialEvaluator`, `IndirectAttackEvaluator`, + `CodeVulnerabilityEvaluator`, `UngroundedAttributesEvaluator`, and + optionally `GroundednessProEvaluator`. +3. Adds the required Azure AI Foundry endpoint config keys to + `quality.thresholds.json` and surfaces them in the chat output. + +## Runtime gating + +Safety tests must **never** fail the build when Foundry creds are +missing — they're an opt-in capability. The pattern: + +```csharp +[TestClass] +public sealed class SafetyTests +{ + [ClassInitialize] + public static void Init(TestContext _) + { + if (!EvalEnv.UseFoundrySafety) + Assert.Inconclusive( + "Safety tier disabled. Set EVAL_USE_FOUNDRY_SAFETY=1 and " + + "AZURE_AI_FOUNDRY_ENDPOINT to enable."); + } + + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); + + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task ContentHarm(GoldenItem g) + { + var reporting = ReportingConfig.ForSafety(); // separate config — Foundry chat client + await using var run = await reporting.CreateScenarioRunAsync($"Safety.ContentHarm.{g.Id}"); + var agent = Wire.ResolveAgentClient(); + var messages = new List { new(ChatRole.User, g.UserMessage) }; + var response = await agent.GetResponseAsync(messages); + await run.EvaluateAsync(messages, response); // ContentHarmEvaluator returns all 4 metrics + } + + // Repeat for ProtectedMaterial / IndirectAttack / CodeVulnerability / etc. +} +``` + +## Why `ContentHarmEvaluator` (not 4 separate) + +From the upstream docs: + +> ContentHarmEvaluator provides single-shot evaluation for the four +> metrics supported by HateAndUnfairnessEvaluator, SelfHarmEvaluator, +> ViolenceEvaluator, and SexualEvaluator. + +That's **1 Foundry call instead of 4** for the same metric set. Always +wire `ContentHarmEvaluator` unless the user has a strict reason to +isolate one harm category. + +## Config keys surfaced in chat + +When Safety tier is enabled, the skill output adds: + +``` +Safety tier enabled. To activate at runtime: + export EVAL_USE_FOUNDRY_SAFETY=1 + export AZURE_AI_FOUNDRY_ENDPOINT=https://.cognitiveservices.azure.com + az login --tenant # DefaultAzureCredential + +Safety tests are MARKED INCONCLUSIVE (not failed) when the env vars are unset, +so your default `dotnet test` run will not break. +``` + +## What Safety evaluators do **not** cover + +Document this explicitly in the rubric: safety evaluators are *output* +classifiers, not *input* classifiers. They do not protect the agent +from receiving harmful prompts — for that, use a separate input filter +(e.g., Azure AI Content Safety on the request side). + +`IndirectAttackEvaluator` is the closest to an input-side check; it +looks for prompt-injection-style content in the *response* that would +indicate the model picked up an indirect attack from retrieved content +or tool output. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md new file mode 100644 index 0000000000..fab6e2eb7f --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md @@ -0,0 +1,143 @@ +# Telemetry mode + +Telemetry mode captures **latency, input tokens, output tokens, and +cost** per agent call across a fixed input set. It is **not** the same +as the MEAI eval report — it produces a separate cost/latency capture. + +## Why separate from quality + +Quality mode answers "is the response any good?" via +`Microsoft.Extensions.AI.Evaluation.Reporting` and produces +`report.html`. + +Telemetry mode answers "how much does it cost and how slow is it?" +via a delegating `IChatClient` wrapper. Different question, different +artifact. Conflating them is the #1 most common scaffolding mistake. + +## Artifacts (each in `.copilot/perf-reports/evals//`) + +- `telemetry.md` — human-readable per-input table. +- `telemetry.json` — machine-readable for CI scraping. +- `telemetry.junit.xml` — for test-result dashboards. + +These are **distinct** from `report.html` (which only quality mode +writes). The skill output should never refer to `telemetry.md` as +"the eval report." + +## Test shape + +```csharp +[TestClass] +public sealed class TelemetryTests +{ + public static IEnumerable Inputs() => + InputsLoader.Load().Select(i => new object[] { i }); + + [TestMethod, DynamicData(nameof(Inputs), DynamicDataSourceType.Method)] + public async Task Capture(TelemetryInput input) + { + var inner = Wire.ResolveAgentClient(); + var wrapped = new TelemetryCapturingChatClient(inner, PriceTable.Load()); + + var messages = new List { new(ChatRole.User, input.Text) }; + var sw = Stopwatch.StartNew(); + var response = await wrapped.GetResponseAsync(messages); + sw.Stop(); + + TelemetryStore.Record(new TelemetryRecord( + AgentName: input.Agent, + Model: response.ModelId ?? "unknown", + InputTokens: response.Usage?.InputTokenCount ?? 0, + OutputTokens: response.Usage?.OutputTokenCount ?? 0, + LatencyMs: sw.ElapsedMilliseconds, + CostUsd: wrapped.LastCostUsd)); + } + + [ClassCleanup] + public static void WriteReports() => TelemetryStore.FlushTo( + Path.Combine(RepoRoot.Find(), ".copilot", "perf-reports", "evals", + ReportingConfig.ExecutionName)); +} +``` + +## Delegating client (sketch) + +```csharp +internal sealed class TelemetryCapturingChatClient(IChatClient inner, PriceTable prices) : IChatClient +{ + public decimal LastCostUsd { get; private set; } + + public async Task GetResponseAsync( + IEnumerable messages, ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + var resp = await inner.GetResponseAsync(messages, options, cancellationToken); + var usage = resp.Usage; + LastCostUsd = prices.Cost(resp.ModelId, + usage?.InputTokenCount ?? 0, usage?.OutputTokenCount ?? 0); + return resp; + } + + // GetStreamingResponseAsync delegates similarly; usage parsed off the last update. + + public object? GetService(Type serviceType, object? serviceKey = null) => + inner.GetService(serviceType, serviceKey); + + public void Dispose() => inner.Dispose(); +} +``` + +## `inputs.json` + +```json +[ + { "agent": "receptionist", "text": "Hi" }, + { "agent": "behavioural", "text": "Tell me about a tough project" }, + { "agent": "technical", "text": "How would you throttle requests?" }, + { "agent": "summariser", "text": "Wrap up the interview" } +] +``` + +## `prices.json` + +```json +{ + "gpt-4o-mini": { "input_per_1k": 0.00015, "output_per_1k": 0.0006 }, + "gpt-4o": { "input_per_1k": 0.0025, "output_per_1k": 0.01 }, + "o4-mini": { "input_per_1k": 0.003, "output_per_1k": 0.012 } +} +``` + +Edit freely — costs change. The price table is **never** baked into source. + +## `InputsLoader` (the deserializer the test uses) + +`inputs.json` is snake_case; the C# records (`TelemetryInput`, etc.) are +PascalCase. The loader **must** specify a `JsonSerializerOptions` with +`PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower` and +`PropertyNameCaseInsensitive = true`, or properties deserialize to +`null` and tests fail (or, worse, silently produce zeroed records when +the property is never read — telemetry mode is especially prone to +this because it never accesses some fields). + +```csharp +internal static class InputsLoader +{ + private static readonly JsonSerializerOptions s_options = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, + PropertyNameCaseInsensitive = true, + }; + + public static List Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "Telemetry", "inputs.json"); + return JsonSerializer.Deserialize>( + File.ReadAllText(path), s_options) ?? new(); + } +} +``` + +Same applies to `PriceTable.Load()` (`prices.json`), `MatrixLoader.Load()` +(`matrix.json`), and `GoldenLoader.Load()` (`golden.json`). Use a shared +options instance — do NOT rely on STJ defaults. diff --git a/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml new file mode 100644 index 0000000000..a9adff6066 --- /dev/null +++ b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml @@ -0,0 +1,240 @@ +config: + max_parallel_scenarios: 1 + max_parallel_runs: 2 + +scenarios: + - name: "Fresh install — creates copilot-instructions.md with managed block" + prompt: "I have a new MAF/Aspire/Foundry agentic .NET app and Copilot keeps missing perf issues. Install the agentic-perf rules into this project." + setup: + files: + - path: "MyAgentApp.sln" + content: | + Microsoft Visual Studio Solution File, Format Version 12.00 + - path: "MyAgentApp.Agent/MyAgentApp.Agent.csproj" + content: | + + + net10.0 + enable + + + + + + + - path: "MyAgentApp.Agent/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + var app = builder.Build(); + app.Run(); + assertions: + - type: "file_exists" + path: ".github/copilot-instructions.md" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "BEGIN: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "END: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Agent count" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Handoff edges" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Model selection" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Message-history strategy" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Token / cost surfacing" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Post-change measurement" + - type: "exit_success" + rubric: + - "Created .github/copilot-instructions.md (or detected an existing one) and added a managed block delimited by sentinel HTML comments" + - "The managed block contains all six rule categories in the canonical order: agent count, handoff edges, model selection, message-history strategy, token/cost surfacing, post-change measurement" + - "The managed block contains a thresholds YAML map with default values" + - "Did not modify content outside the sentinel-delimited block" + - "Embedded the skill version in the BEGIN sentinel comment" + timeout: 360 + + - name: "Append to existing copilot-instructions.md without disturbing user content" + prompt: "Install the agentic-perf rules into this project. There is already a copilot-instructions.md with team conventions — preserve them." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + # Team conventions + + - Always use `IChatClient` from `Microsoft.Extensions.AI`, never the provider SDK directly. + - Commit messages follow conventional-commits. + - Run `dotnet format` before pushing. + - path: "MyAgentApp.Agent/MyAgentApp.Agent.csproj" + content: | + + net10.0 + + + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Team conventions" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "conventional-commits" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "BEGIN: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Agentic Performance Rules" + - type: "exit_success" + rubric: + - "Preserved every line of the existing 'Team conventions' content unchanged" + - "Appended the managed block after the existing content (or in a clearly delimited section)" + - "Did not duplicate or restate the user's existing rules inside the managed block" + timeout: 360 + + - name: "Idempotent re-install — current-version managed block is a no-op" + prompt: "The agentic-perf rules block is already installed in this project — re-run the install to confirm it's a no-op (no diff on second run)." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + # Project notes + + + + ## Agentic Performance Rules + + ```yaml + thresholds: + per_turn_input_token_warn: 8000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (rule sections elided for fixture brevity) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "v0.3.0" + - type: "exit_success" + rubric: + - "Detected the existing managed block at the current skill version" + - "Reported 'already current' (or equivalent) and made no changes to the file" + - "Did not duplicate the managed block" + timeout: 360 + + - name: "Update older version — preserves user-edited threshold values" + prompt: "Update the agentic-perf rules in this project to the latest version. I had to bump the input-token warn level for our RAG workflow — keep that override." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + + + ## Agentic Performance Rules + + ```yaml + thresholds: + per_turn_input_token_warn: 25000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (older content) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn: 25000" + - type: "exit_success" + rubric: + - "Detected the existing managed block at an older version (v0.0.1)" + - "Replaced the block with the current-version template" + - "Preserved the user-edited per_turn_input_token_warn value of 25000 in the new block — did NOT reset to the 8000 default" + - "Showed the user a diff or summary of what changed before applying" + timeout: 360 + + - name: "Update from v0.2.0 — drops deprecated agent_count_max and llm_routed_edges_max_per_turn with warning" + prompt: "Update the agentic-perf rules in this project to the latest version." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + + + ## Agentic Performance Rules + + ```yaml + thresholds: + agent_count_max: 5 + llm_routed_edges_max_per_turn: 3 + per_turn_input_token_warn: 12000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (older content) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn: 12000" + - type: "exit_success" + rubric: + - "Detected the existing managed block at v0.2.0" + - "Replaced the block with the current-version template (v0.3.0+)" + - "Preserved per_turn_input_token_warn: 12000 (user override) — did NOT reset to default" + - "Dropped the deprecated `agent_count_max` key from the rendered YAML (no longer a threshold in current schema)" + - "Dropped the deprecated `llm_routed_edges_max_per_turn` key from the rendered YAML" + - "Emitted a chat warning naming each dropped key (`agent_count_max`, `llm_routed_edges_max_per_turn`) so the user has an audit trail" + - "Did NOT silently retain the dropped keys in the new managed block" + timeout: 360 + + - name: "AGENTS.md stub when both instructions files exist" + prompt: "Install the agentic-perf rules. The project uses both AGENTS.md and .github/copilot-instructions.md." + setup: + files: + - path: "AGENTS.md" + content: | + # Agent guidelines for this repository + + See `.github/copilot-instructions.md` for repository conventions. + - path: ".github/copilot-instructions.md" + content: | + # Repository conventions + + - Use `IChatClient` for all LLM calls. + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "BEGIN: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: "AGENTS.md" + value: "managed by" + - type: "exit_success" + rubric: + - "Wrote the managed block into .github/copilot-instructions.md (the GitHub-native primary location)" + - "Added a single-line stub to AGENTS.md pointing readers to .github/copilot-instructions.md for the agentic-perf rules" + - "Did NOT duplicate the rule prose into AGENTS.md" + - "Preserved the existing content of both files outside the managed block / stub line" + timeout: 360 diff --git a/tests/dotnet-ai/scan-agentic-app-perf/eval.yaml b/tests/dotnet-ai/scan-agentic-app-perf/eval.yaml new file mode 100644 index 0000000000..067b34315e --- /dev/null +++ b/tests/dotnet-ai/scan-agentic-app-perf/eval.yaml @@ -0,0 +1,155 @@ +name: scan-agentic-app-perf +required_skills: + - scan-agentic-app-perf + +scenarios: + - name: clean-project-zero-criticals + timeout: 360 + prompt: | + Audit the agentic .NET app at ./fixture for perf, cost, and reliability issues. + Read-only — don't edit any source files. + setup: + files: + - path: fixture/MyApp.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + var openai = builder.AddConnectionString("openai"); + builder.AddProject("coach") + .WithReference(openai) + .WaitFor(openai); + builder.Build().Run(); + - path: fixture/MyApp.Coach/Program.cs + content: | + var builder = WebApplication.CreateBuilder(args); + builder.AddServiceDefaults(); + builder.AddAzureOpenAIClient("openai"); + builder.Services.AddSingleton(sp => + sp.GetRequiredService() + .GetChatClient(builder.Configuration["Model"]) + .AsIChatClient()); + builder.Services.AddSingleton(sp => new ChatClientAgent( + sp.GetRequiredService(), + instructions: "You are a concise coach.")); + var app = builder.Build(); + app.MapDefaultEndpoints(); + app.MapPost("/chat", async (ChatClientAgent agent, string message) => + await agent.RunAsync(message)); + app.Run(); + - path: fixture/MyApp.Coach/appsettings.json + content: | + { "Model": "gpt-4o-mini" } + assertions: + - type: file_exists + path: fixture/.copilot/perf-reports/scan.md + - type: file_contains + path: fixture/.copilot/perf-reports/scan.md + value: "## Findings" + - type: file_contains + path: fixture/.copilot/perf-reports/scan.md + value: "critical: 0" + + - name: sprawl-fixture-flags-topology + timeout: 360 + prompt: | + My agent app at ./fixture feels too complex. Find the perf and topology issues. + setup: + files: + - path: fixture/Sprawl.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddProject("a"); + builder.AddProject("b"); + builder.AddProject("c"); + builder.AddProject("d"); + builder.AddProject("e"); + builder.AddProject("f"); + builder.Build().Run(); + - path: fixture/Agent.A/Program.cs + content: | + var agent = new ChatClientAgent(client, instructions: "router"); + agent.AddHandoff("b"); agent.AddHandoff("c"); + - path: fixture/Agent.B/Program.cs + content: | + var agent = new ChatClientAgent(client, instructions: "router"); + agent.AddHandoff("c"); agent.AddHandoff("d"); + assertions: + - type: file_exists + path: fixture/.copilot/perf-reports/scan.md + - type: file_contains + path: fixture/.copilot/perf-reports/scan.md + value: "topology" + - type: output_contains + value: "critical" + + - name: full-history-fixture-flags-message-history + timeout: 360 + prompt: | + Why is the agent app at ./fixture slow on multi-turn conversations? Find the perf issues. + setup: + files: + - path: fixture/HistoryHog.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddProject("coach"); + builder.AddProject("critic"); + builder.Build().Run(); + - path: fixture/Coach/Program.cs + content: | + await critic.RunAsync(history, cancellationToken); + assertions: + - type: file_contains + path: fixture/.copilot/perf-reports/scan.md + value: "history" + + - name: prompt-bloat-fixture-flags-prompt-weight + timeout: 360 + prompt: | + Audit the agent app at ./fixture for perf and cost issues. + setup: + files: + - path: fixture/PromptHog.AppHost/Program.cs + content: | + var agent = new ChatClientAgent(client, instructions: HUGE_PROMPT); + - path: fixture/PromptHog.AppHost/HugePrompt.cs + content: | + const string HUGE_PROMPT = @"...PLACEHOLDER 12000 CHARS OF RULES AND EXAMPLES..."; + assertions: + - type: file_contains + path: fixture/.copilot/perf-reports/scan.md + value: "prompt" + + - name: missing-otel-fixture-flags-otel-coverage + timeout: 360 + prompt: | + Find the observability and perf issues in the agent app at ./fixture. + setup: + files: + - path: fixture/NoOtel.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddProject("coach"); + builder.Build().Run(); + - path: fixture/Coach/Program.cs + content: | + var builder = WebApplication.CreateBuilder(args); + // No builder.AddServiceDefaults() and no OpenTelemetry wiring at all: + // no tracing, no metrics, no OTLP exporter, no gen_ai.* token telemetry. + builder.AddAzureOpenAIClient("openai"); + builder.Services.AddSingleton(sp => + sp.GetRequiredService() + .GetChatClient(builder.Configuration["Model"]) + .AsIChatClient()); + builder.Services.AddSingleton(sp => new ChatClientAgent( + sp.GetRequiredService(), + instructions: "You are a helpful coach.")); + var app = builder.Build(); + app.MapPost("/chat", async (ChatClientAgent agent, string message) => + await agent.RunAsync(message)); + app.Run(); + - path: fixture/Coach/appsettings.json + content: | + { "Model": "gpt-4o-mini" } + assertions: + - type: file_contains + path: fixture/.copilot/perf-reports/scan.md + value: "otel" diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml new file mode 100644 index 0000000000..bde5c1fcc6 --- /dev/null +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -0,0 +1,373 @@ +name: setup-maf-evals +required_skills: + - setup-maf-evals + +scenarios: + - name: scaffold-evals-tests-project-fresh + timeout: 600 + prompt: | + Set up evals for the agentic app at ./fixture. Wire telemetry, + quality, and compare modes. Skip Safety, Aspire panel, and CI workflow. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + builder.Build().Run(); + - path: fixture/MyApp.Coach/MyApp.Coach.csproj + content: | + net10.0 + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + - type: file_exists + path: fixture/MyApp.Evals.Tests/.config/dotnet-tools.json + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/ReportingConfig.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Quality/golden.json + - type: file_exists + path: fixture/MyApp.Evals.Tests/Telemetry/TelemetryTests.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Compare/CompareTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + value: "Microsoft.Extensions.AI.Evaluation.Reporting" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + value: "Microsoft.Extensions.AI.Evaluation.NLP" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + value: "MSTest" + - type: file_contains + path: fixture/MyApp.Evals.Tests/.config/dotnet-tools.json + value: "microsoft.extensions.ai.evaluation.console" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/ReportingConfig.cs + value: "DiskBasedReportingConfiguration" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + value: "BLEUEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + value: "reference_response" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + value: "schema_version" + - type: file_contains + path: fixture/.gitignore + value: ".copilot/perf-reports/evals/" + - type: file_contains + path: fixture/.gitignore + value: "_store/" + + - name: scaffold-with-safety-tier + timeout: 600 + prompt: | + Add an evaluation harness for the agentic app at ./fixture, including safety evaluators. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + value: "ContentHarmEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + value: "Assert.Inconclusive" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + value: "Microsoft.Extensions.AI.Evaluation.Safety" + + - name: scaffold-with-ci-workflow + timeout: 600 + prompt: | + Add evals for the agentic app at ./fixture, and include a GitHub Actions workflow that runs them on every PR. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/.github/workflows/evals.yml + - type: file_contains + path: fixture/.github/workflows/evals.yml + value: "aieval report" + - type: file_contains + path: fixture/.github/workflows/evals.yml + value: "upload-artifact" + + - name: ichatclient-detection-azure-openai + timeout: 600 + prompt: | + Add evaluation coverage for the agentic app at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + value: "AddAzureOpenAIChatClient" + - type: output_contains + value: "AppHost.cs" + + - name: ichatclient-detection-missing-emits-stub + timeout: 600 + prompt: | + Add an evaluation harness for the agentic app at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.Build().Run();" + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + value: "NotImplementedException" + - type: output_contains + value: "no registration found" + + - name: skip-when-no-app-host + timeout: 600 + prompt: | + Add evaluation coverage for the .NET project at ./fixture. + setup: + files: + - path: fixture/PlainApi/PlainApi.csproj + content: | + net10.0 + assertions: + - type: output_contains + value: "agentic" + + - name: update-mode-preserves-user-data + timeout: 600 + prompt: | + Refresh the evaluation harness for the agentic app at ./fixture. A MyApp.Evals.Tests project + already exists with a custom rubric and golden.json. Do not overwrite either. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + content: | + net10.0 + - path: fixture/MyApp.Evals.Tests/Quality/rubric.md + content: | + # Custom rubric — DO NOT OVERWRITE + - my_trait: ... + - path: fixture/MyApp.Evals.Tests/Quality/golden.json + content: | + [{"id": "custom", "user_message": "preserve me", "expected_traits": ["unique"]}] + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/rubric.md + value: "DO NOT OVERWRITE" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + value: "preserve me" + + - name: tier-banner-surfaces-in-chat-output + timeout: 600 + prompt: | + Add an evaluation harness for the agentic app at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: output_contains + value: "EVAL_USE_REAL_AGENT" + - type: output_contains + value: "EVAL_USE_REAL_JUDGE" + - type: output_contains + value: "EVAL_USE_FOUNDRY_SAFETY" + + - name: scaffold-emits-metrics-glossary + timeout: 600 + prompt: | + Set up evals for the agentic app at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + value: "metrics-glossary.md" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/AievalReport.cs + value: "[AssemblyCleanup]" + - type: output_contains + value: "metrics-glossary.md" + + - name: factory-emits-friendly-secrets-diagnostic + timeout: 600 + prompt: | + Add evaluation coverage for the agentic app at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + value: "dotnet user-secrets" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + value: "ConnectionStrings" + + - name: factory-loads-user-secrets-explicitly + timeout: 600 + prompt: | + Set up an evaluation harness for the agentic app at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); + assertions: + # Without explicit AddUserSecrets, dotnet test never reads the user-secrets store + # because the entry assembly is testhost.exe, not the test project. + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + value: "AddUserSecrets" + + - name: pitfalls-doc-warns-reasoning-models-reject-max-tokens + timeout: 180 + reject_tools: ["bash"] + prompt: | + I'm adding an evaluation harness to my .NET agentic app, and the agent under test + uses a reasoning model (o1-style). What do I need to know about the + max_tokens / MaxOutputTokens parameter so the evals don't silently break? + assertions: + # Assert on the agent's answer, not on out-of-sandbox repo files. + - type: output_contains + value: "max_completion_tokens" + - type: output_matches + pattern: "(?i)reasoning model" + + - name: pitfalls-doc-warns-stylistic-agents-fail-completeness + timeout: 180 + reject_tools: ["bash"] + prompt: | + I'm setting up evals for my .NET agentic app. It mostly produces stylistic prose + with no factual claims — which quality evaluators will systematically fail it, and + how should I account for that? + assertions: + # Assert on the agent's answer, not on out-of-sandbox repo files. + - type: output_contains + value: "CompletenessEvaluator" + - type: output_matches + pattern: "(?i)(equivalence|rubric|drop)" + + - name: compare-mode-is-opt-in-not-default + timeout: 180 + reject_tools: ["bash"] + prompt: | + I'm adding an evaluation harness to my .NET agentic app. Is compare mode + (A/B model matrix) on by default, or is it opt-in? + assertions: + # Assert on the agent's answer, not on out-of-sandbox repo files. + - type: output_matches + pattern: "(?i)opt[- ]?in" +