[skills-eval] dotnet-test: 14 skills, 37% pass — 1 P0, 1 P1, 6 to strengthen, 6 keep

[AbhitejJohn]

🔄 EDITED 2026-07-19 — correction per #909

This finding has been revised. The errored trials that drove the FIX-RELIABILITY (P0) flag(s) below were shown in #909 to be judge-side infrastructure failures — a disabled/throttled PAT (CAPIError 400 organization disabled) and Vally's session.idle judge timeout — not fixture nondeterminism. Verdict scores already exclude errored trials, so re-running the classifier with those judge errors dropped reclassifies the affected skill(s):

Skill	Was	Now
writing-mstest-tests	FIX-RELIABILITY (P0)	KEEP-POLISH (keep)
detect-static-dependencies	FIX-RELIABILITY (P0)	KEEP-POLISH (keep)

Struck-through text below is the superseded finding — in particular the "pin fixture SDK/tool versions" remediation does not apply. The title's P0 count has been updated. No eval re-run is required to correct the scores; recovering the lost judgments (optional) means re-judging those trials, not re-running the skill.

Context — cross-family skill evaluation: dotnet-test

This issue is self-contained: it captures everything a skill author needs to act on the dotnet-test plugin without opening the full report.

What this measures. Every runnable skill in dotnet/skills was run through Vally (0.7) on a cross-family matrix: 5 executor model families — opus-4.8, gpt-5.5, sonnet-4.6, haiku-4.5, mai-flash — each judged by a different family (judge ≠ executor; default judge = latest Opus, or GPT when Opus is the executor). For every skill, a skilled run is compared against a baseline (no-skill) run and scored per executor. This removes single-model and self-judging bias, so a skill that only helps one model family — or only its own family's judge — is visible.

• Data source: cross-family CI grid (run 29228914412 + backfills) — 5 executors × 85 runnable skills, 419 scored cells over 84 skills.
• Row grain: one row per skill, aggregated across its (up to 5) executor cells. avgN is the mean trial count behind the cells (trials 1–17; higher = more statistically trustworthy). thin-N flags directional-only rows.

dotnet-test at a glance (portfolio scorecard)

Plugin	Skills	Cells	Pass	Impact	Tie-trials	Err	avg ΔTok	avg ΔTurns	avg ΔTools	Headline
dotnet-test	14	70	37%	0.336	63	3	+50,897	+0.40	+1.20	🟠 Most inert of the large plugins (63 tie-trials / 70 cells)

Insights. 14 skill(s); mean impact 0.34; 7/14 help ≥1 frontier model. Exemplars (win incl. frontier): mtp-hot-reload, test-tagging, generate-testability-wrappers. Weakest: migrate-static-to-wrapper (-0.04).

Address first: writing-mstest-tests — reliability; detect-static-dependencies — reliability; migrate-static-to-wrapper — regression; code-testing-agent — cost.

Address first: migrate-static-to-wrapper — regression; code-testing-agent — cost. (writing-mstest-tests and detect-static-dependencies were reclassified out of P0 — see EDIT above.)

How to read the table

Each skill is scored skilled vs. baseline on these axes:

Signal	Column	What it means	Good
Breadth	Families ✓ (n/5)	How many of the 5 model families the skill helps (per-family pass count, max 5/5)	4–5 / 5
Where	Passed on	Which families passed. Frontier = latest Opus + latest GPT are bold; Sonnet 4.6 / Haiku 4.5 / MAI Flash are mid/low-weight	frontier ✓
Magnitude	Impact (−1…+1)	How strongly the judge prefers skilled over baseline	≥ 0.4
Decisiveness	Ties▵	Trials where the judge saw no difference → skill is inert	low
Safety	Loss▵	Trials where skilled was WORSE than baseline → skill misfires	~0
Reliability	Err	Trials that errored/crashed in setup or judging	0
Efficiency	ΔTok / ΔTurns / ΔTools	Extra tokens / agent turns / tool calls vs baseline	≤ 0
Confidence	avgN	Mean trials behind the verdict; low N = directional only	≥ 3
Invocation	Call%	Share of skilled trials where the model actually invoked the skill	~100%

Families ✓ is cell-level (max 5/5); Ties▵/Loss▵ are trial-level tallies summed across all families (including the ones where the skill failed). A high Families ✓ next to non-zero Loss▵ is not a contradiction — see Passed on and the Action text for where losses landed.

Action buckets (each skill has one primary action; [flags] note secondary concerns):

Bucket	Priority	Meaning
FIX-RELIABILITY	🔴 P0	Errored trials / no verdict — stabilize the harness before trusting the score
FIX-DISCOVERY	🔴 P0	Model doesn't invoke it (Call% < 50%) — a triggering/description problem
FIX-REGRESSION	🔴 P0	Skilled is worse than baseline on many trials — the skill misfires
ADD-DECISIVENESS	🟠 P1	Called ~100% but ties dominate, ~0 impact — inert; needs sharper behavioral steps
TRIM-COST	🟠 P1	Passes but with heavy token/turn overhead — trim verbosity
EXEMPLAR	🟢 keep	Broad, strong, reliable win — use as a template
EFFICIENT-WIN	🟢 protect	Wins and cuts turns/tools — the ideal shape
KEEP-POLISH	🟢	Solid majority win; minor polish + more trials
STRENGTHEN	🟡 P2	Marginal/mixed lift — sharpen triggers & success criteria

Per-skill actions

Skill	Families ✓	Passed on (frontier bold)	Impact	Ties▵	Loss▵	Err	avgN	Call%	ΔTok	ΔTurns	ΔTools	Action
mtp-hot-reload	3/5	Opus, GPT, Sonnet	0.69	0	4	0	6	94%	-7,703	+0.03	-0.57	EFFICIENT-WIN · ✅ Better and cheaper. Wins on Opus, GPT, Sonnet while saving 0.57 tools, 7,703 tok. What's good: it adds value without bloat — the ideal shape. Protect the brevity; don't let it grow. ([efficient])
coverage-analysis	0/5	—	0.64	0	0	0	3	100%	+114,632	+1.93	+2.73	STRENGTHEN · 🟡 Marginal/mixed (passes 0/5). Diagnosis: misses both frontier models — likely assumes context they solve unaided. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. ([both-frontier-miss], [costly])
test-tagging	5/5	Opus, GPT, Sonnet, Haiku, MAI	0.57	9	2	0	8	88%	+17,862	-1.02	+1.3	EXEMPLAR · ✅ Template-worthy. What's good: broad (passes 5/5, incl. frontier Opus, GPT); cheaper (1.02 turns). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ([efficient])
crap-score	2/5	Opus, MAI	0.41	0	2	0	3	100%	+35,905	+1.27	+2.27	STRENGTHEN · 🟡 Marginal/mixed (passes 2/5 — Opus, MAI). Diagnosis: model-dependent — helps some families, not others. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. ([frontier-miss])
writing-mstest-tests	4/5	GPT, Sonnet, Haiku, MAI	0.38	6	7	2	14.6	89%	+32,537	+0.51	+0.46	FIX-RELIABILITY · 🔴 2 errored trial(s) — verdict can't be trusted until setup is deterministic. This is an infra/harness fix, not a content one: (1) capture the failing trial's stderr; (2) pin tool/SDK versions in the fixture; (3) add a setup smoke-check before scoring. ([reliability], [frontier-miss]) 🔄 EDIT (see #909): KEEP-POLISH · 🟢 Solid majority win (4/5 — GPT, Sonnet, Haiku, MAI); losses/ties sit in the one non-passing cell (Opus). The 2 errored trials were judge-side session.idle timeouts (haiku, sonnet46), not fixture flakiness. Raise trials on Opus; minor wording polish only. ([frontier-miss])
generate-testability-wrappers	2/5	GPT, Sonnet	0.37	1	4	0	4	90%	-10,342	-2.15	-2.1	EFFICIENT-WIN · ✅ Better and cheaper. Wins on GPT, Sonnet while saving 2.15 turns, 2.1 tools, 10,342 tok. What's good: it adds value without bloat — the ideal shape. Protect the brevity; don't let it grow. ⚠️ Frontier miss on Opus — worth a look. ([frontier-miss], [frontier-regressed], [efficient])
run-tests	3/5	Opus, GPT, Sonnet	0.36	15	9	0	13.8	76%	+131,515	+3.5	+5.26	KEEP-POLISH · 🟢 Solid majority win on Opus, GPT, Sonnet. Losses/ties are concentrated in the non-passing cells (Haiku, MAI), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on Haiku, MAI; (2) add trials on the failing families to confirm; (3) minor wording polish only. ([costly])
test-smell-detection	1/5	Sonnet	0.29	7	0	0	4	75%	+25,487	-0.15	+0.7	STRENGTHEN · 🟡 Marginal/mixed (passes 1/5 — Sonnet). Diagnosis: mostly ties — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. ([both-frontier-miss])
detect-static-dependencies	3/5	GPT, Sonnet, MAI	0.27	7	2	1	5.8	90%	+20,012	+0.91	+1.03	FIX-RELIABILITY · 🔴 1 errored trial(s) — verdict can't be trusted until setup is deterministic. This is an infra/harness fix, not a content one: (1) capture the failing trial's stderr; (2) pin tool/SDK versions in the fixture; (3) add a setup smoke-check before scoring. ([reliability], [frontier-miss]) 🔄 EDIT (see #909): KEEP-POLISH · 🟢 Solid majority win (3/5 — GPT, Sonnet, MAI). The 1 errored trial was a judge-side disabled-PAT failure (haiku); this is a static-analysis skill that never builds, so a fixture SDK issue was never plausible. Investigate the Opus miss; minor polish. ([frontier-miss])
assertion-quality	1/5	Sonnet	0.25	3	6	0	4.8	80%	+39,846	+1.58	+2.14	STRENGTHEN · 🟡 Marginal/mixed (passes 1/5 — Sonnet). Diagnosis: misses both frontier models — likely assumes context they solve unaided. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. ([both-frontier-miss])
test-gap-analysis	1/5	Sonnet	0.23	2	4	0	4	75%	-126,010	-6.3	-6.75	STRENGTHEN · 🟡 Marginal/mixed (passes 1/5 — Sonnet). Diagnosis: misses both frontier models — likely assumes context they solve unaided. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. ([both-frontier-miss])
code-testing-agent	0/5	—	0.16	5	4	0	3.8	100%	+349,643	+2.83	+5.37	TRIM-COST · 🟠 Passes but heavy (+349,643 tok). Value is real; cost isn't justified. Trim: (1) cut redundant/whole-file reads — point to specific sections; (2) replace "explore everything" with a targeted checklist; (3) move deep reference material behind links instead of inlining it. ([both-frontier-miss], [costly])
test-anti-patterns	1/5	Sonnet	0.13	6	7	0	6	100%	+39,956	+1.13	+2.1	STRENGTHEN · 🟡 Marginal/mixed (passes 1/5 — Sonnet). Diagnosis: misses both frontier models — likely assumes context they solve unaided. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. ([both-frontier-miss], [frontier-regressed])
migrate-static-to-wrapper	0/5	—	-0.04	2	8	0	3	100%	+49,216	+1.47	+2.8	FIX-REGRESSION · 🔴 Retire candidate. Fails on both frontier models (Opus, GPT) and regresses (Opus, GPT, Sonnet, MAI) with non-positive impact (-0.04) across ~15 trials. A skill that makes the strongest models worse is net-negative. Strong case to remove unless a narrow, guarded rewrite (explicit stop-conditions) can be shown to stop the regressions. ([both-frontier-miss], [frontier-regressed])

⚠️ Coverage gap — 6 skills excluded from the cross-family run

The matrix scores a skill only if it has a tests/dotnet-test/<skill>/eval.vally.yaml. The following dotnet-test skills ship a SKILL.md but were never measured — so they carry zero evidence of impact (not counted in the pass rate or scorecard above):

• Has a base eval.yaml but no eval.vally.yaml — add a cross-family Vally config to fold them into the matrix: filter-syntax, grade-tests
• No eval harness at all — needs both eval.yaml + eval.vally.yaml: code-testing-extensions, find-untested-sources, platform-detection, test-analysis-extensions

Until these have an eval.vally.yaml, we cannot tell whether they help, hurt, or are inert. Recommended: add the cross-family eval configs and re-run so they appear in the next scorecard.

---

_{Generated from the cross-family Call-to-Action report (CALL-TO-ACTION.md §4–§5; companion IMPACT-ANALYSIS.md). Regenerate the underlying tables with node deep-metrics.mjs "$env:TEMP\cf-ci" agg-ci → node gen-cta-tables.mjs agg-ci. Numbers are directional where avgN is low; treat single-trial cells as hypotheses to confirm with more runs.}

[github-actions]

🎯 Issue Triage

Summary: This is a cross-family skill evaluation report for the dotnet-test plugin (14 skills, 37% pass rate). It identifies 3 P0 issues (reliability fixes for writing-mstest-tests, detect-static-dependencies; regression fix for migrate-static-to-wrapper), 1 P1 cost trim (code-testing-agent), 6 skills to strengthen, and 6 untested skills needing eval configs.

Type: task — actionable evaluation report with prioritized work items

Assigned owners: @AbhitejJohn and @webreidi assigned directly. Primary area owners are @dotnet/dotnet-testing (team — cannot be directly assigned).

Confidence: High — issue clearly maps to dotnet-test plugin with explicit per-skill action buckets.

Similar issues: Related eval issues exist for other plugins (e.g., dotnet-test-migration) in the same series.

Triage details

• Area mapping: area-test (already applied) — maps to /plugins/dotnet-test/ per CODEOWNERS
• CODEOWNERS match: /plugins/dotnet-test/ @dotnet/dotnet-testing``
• Confidence rationale: High — structured evaluation report with clear scope, type, and actionable items. No ambiguity about area or owners.
• Suggested next steps: Address P0 items first (FIX-RELIABILITY on writing-mstest-tests and detect-static-dependencies; FIX-REGRESSION on migrate-static-to-wrapper), then add eval configs for the 6 uncovered skills.

Generated by Issue Triage for issue #899 · 82.6 AIC · ⊞ 28.4K · ◷