Make performance eval criteria outcome-focused

[artl93]

Part of #886

Purpose and result

This PR is only an evaluator de-overfitting change. Its intended effect is to preserve the existing skill outcome while removing vocabulary-dependent grading, so a separate future performance-skill update can be measured against a frozen, comparable evaluator. It does not attempt to improve the skill or its score.

A fresh matched A/B evaluated only dotnet-diag/analyzing-dotnet-performance, with three repetitions across Opus, GPT, Sonnet 4.6, Haiku, and MAI:

• current eval control: main eval, head 4cf1d89
• revised eval treatment: this PR's eval, head 5a59430
• byte-identical skill in both arms: SHA-256 9980c08c82a1f0a961892472ea773d6b5342b080e46d92e9db3c109814e1afad
• byte-identical workflow in both arms: SHA-256 c6e979a63e0096469cd7ca1f3b3253f9f265f65e9899dc46354b4611768d9a22
• only eval.yaml and eval.vally.yaml differed

Outcome parity

Comparison	Current eval	Revised eval	Interpretation
All scenarios, mean skilled-vs-baseline preference	+0.258 over 149 pairs	+0.265 over 156 pairs	+0.007: effectively unchanged
Unchanged scenarios, mean preference	+0.278 over 123 pairs	+0.305 over 126 pairs	+0.027 empirical stochastic-noise floor
Unchanged scenarios, W/T/L	88/22/13	90/24/12	Same overall signal; family movement occurred in both directions
Changed scenarios 8–9, W/T/L	15/5/6 over 26 pairs	16/7/7 over 30 pairs	Comparable signal; revised eval completed four additional pairs

Both runs emitted all 330 expected raw records. All ten family jobs succeeded and there were zero pairwise-judge errors. Aggregate family conclusions varied where raw trajectories hit the three-minute limit; those timeout, activation, and tool-call differences also appeared in unchanged scenarios and are treated as run noise, not evaluator or skill impact.

Overfitting removed

In the 56 completed current-eval outputs for scenarios 8–9, literal LINQ, All, and Where checks passed 84/85 times. The sole miss was a demonstrated false negative: MAI's skilled scenario 9 trial 2 passed exit_success and all 5/5 substantive rubric criteria, but failed overall only because its final prose omitted Where.

The revised eval:

• passed exit_success for all 60 scenario 8/9 outputs
• removed the demonstrated Where false negative and the same structural risk from LINQ/All
• produced independent semantic discrimination across all 11 revised criteria
• had per-criterion pass counts ranging 7–15/15 for baseline and 9–15/15 for skilled outputs
• used judge explanations that distinguished behavior preservation, culture safety, hot-path prioritization, repeated projection, params fast paths, and preservation of existing span helpers rather than terminology
• reduced unrequested fixture editing from 58 edit calls across 56 completed current-eval outputs to 3 across 60 revised-eval outputs

Conclusion: the revised evaluator preserves the existing aggregate skill outcome while removing observed and structural vocabulary overfit. That is the sole purpose of this PR and prepares a stable measuring instrument for the separate skill update.

Changes

• replace literal LINQ, All, and Where output gates with exit_success
• rewrite scenario 8 and 9 rubric items as independently evaluable diagnosis/recommendation outcomes
• clarify that both prompts request analysis only and must not modify fixtures
• keep canonical eval.yaml and Vally eval.vally.yaml semantics aligned

Validation

• targeted skill-validator check passes for dotnet-diag
• canonical/Vally parity passes across all 11 scenarios
• git diff --check passes
• branch diff contains only eval.yaml and eval.vally.yaml

The PR remains draft.

[Copilot]

Pull request overview

This PR updates the dotnet-diag performance skill evaluation to reduce overfitting in scenarios 8 and 9 by making rubrics more outcome-focused, and adjusts the skill guidance to reduce unnecessary reference loading and tool-call overhead during focused reviews.

Changes:

• Rewrote scenarios 8 and 9 prompts/rubrics in both canonical and Vally evals toward independently evaluable, outcome-focused criteria.
• Updated analyzing-dotnet-performance skill workflow guidance to emphasize “read source first”, load references only when needed, and batch confirmation searches.

Show a summary per file

File	Description
tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml	Updates scenario 8/9 prompts and rubrics to be less technique/vocabulary-specific.
tests/dotnet-diag/analyzing-dotnet-performance/eval.vally.yaml	Mirrors the scenario 8/9 prompt/rubric updates for Vally parity.
plugins/dotnet-diag/skills/analyzing-dotnet-performance/SKILL.md	Adjusts workflow guidance to reduce reference loading and tool-call overhead.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 3/3 changed files
• Comments generated: 6

[Copilot]

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 0 new

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
analyzing-dotnet-performance	Detects compiled regex startup budget and regex chain allocations	4.0/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅
analyzing-dotnet-performance	Detects CurrentCulture comparer and compiled regex budget in inflection rules	5.0/5 → 5.0/5	✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	❌ [1]
analyzing-dotnet-performance	Finds per-call Dictionary allocation not hoisted to static	3.7/5 ⏰ → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [2]
analyzing-dotnet-performance	Catches compound allocations in recursive number converter with ToLower	3.7/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill, bash, grep	✅ 0.13	✅
analyzing-dotnet-performance	Finds StringComparison.Ordinal missing and FrozenDictionary opportunities	4.7/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	❌ [3]
analyzing-dotnet-performance	Detects Aggregate+Replace chain and struct missing IEquatable	3.7/5 → 4.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep, bash / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [4]
analyzing-dotnet-performance	Finds branched Replace chain in format string manipulation	3.3/5 → 4.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [5]
analyzing-dotnet-performance	Catches LINQ on hot-path string processing and All(char.IsUpper)	4.0/5 → 4.7/5 🟢	✅ analyzing-dotnet-performance; tools: skill, bash, grep / ✅ analyzing-dotnet-performance; tools: skill, grep, bash	✅ 0.13	✅ [6]
analyzing-dotnet-performance	Detects LINQ pipeline in TimeSpan formatting and collection processing	3.7/5 → 2.3/5 ⏰ 🔴	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	❌
analyzing-dotnet-performance	Flags Span inconsistencies and compound method chains in truncation library	3.0/5 ⏰ → 2.3/5 ⏰ 🔴	✅ analyzing-dotnet-performance; tools: report_intent, skill, view / ✅ analyzing-dotnet-performance; tools: report_intent, view, grep, skill	✅ 0.13	❌ [7]
analyzing-dotnet-performance	Identifies unsealed leaf classes and locale hierarchy patterns	3.0/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [8]

[1] ⚠️ High run-to-run variance (CV=67%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -6.0% due to: tokens (29261 → 80436), tool calls (2 → 7), time (19.5s → 49.4s)
[2] ⚠️ High run-to-run variance (CV=157%) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=402%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -9.0% due to: tokens (29583 → 80500), tool calls (2 → 6), time (26.8s → 53.9s)
[4] ⚠️ High run-to-run variance (CV=70%) — consider re-running with --runs 5
[5] ⚠️ High run-to-run variance (CV=107%) — consider re-running with --runs 5
[6] ⚠️ High run-to-run variance (CV=13524%) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=94%) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=61%) — consider re-running with --runs 5

⏰ timeout — run(s) hit the (120s, 180s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 905 in dotnet/skills, download eval artifacts with gh run download 29626270559 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/0b7ae60bec7c7fa37b98ae6c3fb4f286d70f426a/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for 0b7ae60. cc @dotnet/dotnet-diag @dotnet/skills-diag-reviewers — please review.

[Evangelink]

Could we add a rubric criterion that verifies the response only provides recommendations and does not modify the fixture? The prompt now makes non-modification an explicit requirement, but exit_success and the current rubric do not detect violations, so a run can edit the fixture and still pass. This would also match the enforcement used by comparable analysis-only evals.

[github-actions]

👋 @artl93 — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
❌	dotnet-diag	analyzing-dotnet-performance	0/21	0%

Uncovered: dotnet-diag/analyzing-dotnet-performance

• [Validation] All critical patterns were checked (from reference files or inline recipes) (line 178)
• [Validation] Topic-specific recipes run only when matching signals detected (line 179)
• [Validation] Each finding includes a concrete code fix (line 180)
• [Validation] Scan execution checklist is complete (all recipes run) (line 181)
• [Validation] Summary table included at end (line 182)
• [Pitfall] Flagging every Dictionary as needing FrozenDictionary (line 188)
• [Pitfall] Suggesting Span in async methods (line 189)
• [Pitfall] Reporting LINQ outside hot paths (line 190)
• [Pitfall] Suggesting ConfigureAwait(false) in app code (line 191)
• [Pitfall] Recommending ValueTask everywhere (line 192)
• [Pitfall] Flagging new HttpClient() in DI services (line 193)
• [Pitfall] Suggesting [GeneratedRegex] for dynamic patterns (line 194)
• [Pitfall] Suggesting CollectionsMarshal.AsSpan broadly (line 195)
• [Pitfall] Suggesting unsafe code for micro-optimizations (line 196)
• [WorkflowStep] Step 1: Load Reference Files (if available) (line 38)
• [WorkflowStep] Step 2: Detect Code Signals and Select Topic Recipes (line 44)
• [WorkflowStep] Step 3: Scan and Report (line 63)
• [WorkflowStep] Step 4: Classify and Prioritize Findings (line 117)
• [WorkflowStep] Step 5: Generate Findings (line 140)
• [CodePattern] sealed (line 70)
• [CodePattern] readonly (line 70)

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (2)

tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml:160

• Scenario name still includes technique/vocabulary-specific terms (LINQ/All) even though the grader gates were removed and the rubric is now outcome-focused. Renaming the scenario helps keep the evaluator aligned with the PR’s intent and avoids reintroducing vocabulary bias in reporting/diagnostics.

  - name: Catches LINQ on hot-path string processing and All(char.IsUpper)

tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml:178

• Scenario name still calls out LINQ specifically, but the updated evaluation criteria no longer depend on LINQ terminology. Consider renaming to describe the measured outcomes (avoidable intermediate enumeration/materialization) so the scenario label matches the new rubric.

  - name: Detects LINQ pipeline in TimeSpan formatting and collection processing

• Files reviewed: 1/1 changed files
• Comments generated: 0 new

[AbhitejJohn]

/evaluate

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — 1 improved, 0 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).

Skill	Result	Δ Preference [95% CI]	W/T/L	Quality	Baseline
analyzing-dotnet-performance	✅	+32.7% [+16.5%, +48.9%]	10/0/1	4.9/5	4.7/5

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

👋 @artl93 — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)