[skills-eval] dotnet-msbuild: 14 skills, 56% pass — 2 P0, 1 P1, 2 to strengthen, 9 keep

[AbhitejJohn]

Context — cross-family skill evaluation: dotnet-msbuild

This issue is self-contained: it captures everything a skill author needs to act on the dotnet-msbuild plugin without opening the full report.

What this measures. Every runnable skill in dotnet/skills was run through Vally (0.7) on a cross-family matrix: 5 executor model families — opus-4.8, gpt-5.5, sonnet-4.6, haiku-4.5, mai-flash — each judged by a different family (judge ≠ executor; default judge = latest Opus, or GPT when Opus is the executor). For every skill, a skilled run is compared against a baseline (no-skill) run and scored per executor. This removes single-model and self-judging bias, so a skill that only helps one model family — or only its own family's judge — is visible.

• Data source: cross-family CI grid (run 29228914412 + backfills) — 5 executors × 85 runnable skills, 419 scored cells over 84 skills.
• Row grain: one row per skill, aggregated across its (up to 5) executor cells. avgN is the mean trial count behind the cells (trials 1–17; higher = more statistically trustworthy). thin-N flags directional-only rows.

dotnet-msbuild at a glance (portfolio scorecard)

Plugin	Skills	Cells	Pass	Impact	Tie-trials	Err	avg ΔTok	avg ΔTurns	avg ΔTools	Headline
dotnet-msbuild	14	70	56%	0.282	31	0	+8,450	+0.28	+0.01	Deepest, most reliable plugin; thin-N

Insights. 14 skill(s); mean impact 0.28; 9/14 help ≥1 frontier model. Exemplars (win incl. frontier): msbuild-server, build-perf-baseline, build-perf-diagnostics, directory-build-organization, including-generated-files. Weakest: eval-performance (-0.24).

Address first: msbuild-antipatterns — discovery; eval-performance — regression; check-bin-obj-clash — cost.

How to read the table

Each skill is scored skilled vs. baseline on these axes:

Signal	Column	What it means	Good
Breadth	Families ✓ (n/5)	How many of the 5 model families the skill helps (per-family pass count, max 5/5)	4–5 / 5
Where	Passed on	Which families passed. Frontier = latest Opus + latest GPT are bold; Sonnet 4.6 / Haiku 4.5 / MAI Flash are mid/low-weight	frontier ✓
Magnitude	Impact (−1…+1)	How strongly the judge prefers skilled over baseline	≥ 0.4
Decisiveness	Ties▵	Trials where the judge saw no difference → skill is inert	low
Safety	Loss▵	Trials where skilled was WORSE than baseline → skill misfires	~0
Reliability	Err	Trials that errored/crashed in setup or judging	0
Efficiency	ΔTok / ΔTurns / ΔTools	Extra tokens / agent turns / tool calls vs baseline	≤ 0
Confidence	avgN	Mean trials behind the verdict; low N = directional only	≥ 3
Invocation	Call%	Share of skilled trials where the model actually invoked the skill	~100%

Families ✓ is cell-level (max 5/5); Ties▵/Loss▵ are trial-level tallies summed across all families (including the ones where the skill failed). A high Families ✓ next to non-zero Loss▵ is not a contradiction — see Passed on and the Action text for where losses landed.

Action buckets (each skill has one primary action; [flags] note secondary concerns):

Bucket	Priority	Meaning
FIX-RELIABILITY	🔴 P0	Errored trials / no verdict — stabilize the harness before trusting the score
FIX-DISCOVERY	🔴 P0	Model doesn't invoke it (Call% < 50%) — a triggering/description problem
FIX-REGRESSION	🔴 P0	Skilled is worse than baseline on many trials — the skill misfires
ADD-DECISIVENESS	🟠 P1	Called ~100% but ties dominate, ~0 impact — inert; needs sharper behavioral steps
TRIM-COST	🟠 P1	Passes but with heavy token/turn overhead — trim verbosity
EXEMPLAR	🟢 keep	Broad, strong, reliable win — use as a template
EFFICIENT-WIN	🟢 protect	Wins and cuts turns/tools — the ideal shape
KEEP-POLISH	🟢	Solid majority win; minor polish + more trials
STRENGTHEN	🟡 P2	Marginal/mixed lift — sharpen triggers & success criteria

Per-skill actions

Skill	Families ✓	Passed on (frontier bold)	Impact	Ties▵	Loss▵	Err	avgN	Call%	ΔTok	ΔTurns	ΔTools	Action
binlog-generation	0/5	—	0.64	3	0	0	3	93%	+13,103	+0.87	+0.4	STRENGTHEN · 🟡 Marginal/mixed (passes 0/5). Diagnosis: mostly ties — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. ([both-frontier-miss])
msbuild-server	5/5	Opus, GPT, Sonnet, Haiku, MAI	0.52	0	0	0	1	100%	+12,929	+1	+0.2	EXEMPLAR · ✅ Template-worthy. What's good: broad (passes 5/5, incl. frontier Opus, GPT); zero ties or losses (clean). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ([thin-N])
build-perf-baseline	4/5	GPT, Sonnet, Haiku, MAI	0.44	1	0	0	1	100%	-161,315	-4.4	-5	EXEMPLAR · ✅ Template-worthy. What's good: broad (passes 4/5, incl. frontier GPT); cheaper (4.4 turns, 5 tools, 161,315 tok). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ⚠️ Caveat: frontier miss on Opus — confirm it isn't only lifting weaker models on cases frontier already handles. ([frontier-miss], [efficient], [thin-N])
build-perf-diagnostics	4/5	GPT, Sonnet, Haiku, MAI	0.44	1	0	0	1	80%	+30,394	-1.6	-0.6	EXEMPLAR · ✅ Template-worthy. What's good: broad (passes 4/5, incl. frontier GPT); cheaper (1.6 turns, 0.6 tools). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ⚠️ Caveat: frontier miss on Opus — confirm it isn't only lifting weaker models on cases frontier already handles. ([frontier-miss], [efficient], [thin-N])
directory-build-organization	5/5	Opus, GPT, Sonnet, Haiku, MAI	0.40	0	0	0	1	100%	-28,616	-2	-2.2	EXEMPLAR · ✅ Template-worthy. What's good: broad (passes 5/5, incl. frontier Opus, GPT); zero ties or losses (clean); cheaper (2 turns, 2.2 tools, 28,616 tok). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ([efficient], [thin-N])
including-generated-files	5/5	Opus, GPT, Sonnet, Haiku, MAI	0.40	0	0	0	1	100%	+28,420	+1.4	+1.4	EXEMPLAR · ✅ Template-worthy. What's good: broad (passes 5/5, incl. frontier Opus, GPT); zero ties or losses (clean). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ([thin-N])
build-parallelism	2/5	Sonnet, Haiku	0.28	3	0	0	1	100%	+21,857	+1	+2	STRENGTHEN · 🟡 Marginal/mixed (passes 2/5 — Sonnet, Haiku). Diagnosis: mostly ties — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. ([both-frontier-miss], [thin-N])
resolve-project-references	3/5	Opus, Sonnet, Haiku	0.28	1	1	0	1	100%	+4,776	+0.4	-2	KEEP-POLISH · 🟢 Solid majority win on Opus, Sonnet, Haiku. Losses/ties are concentrated in the non-passing cells (MAI, GPT), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on GPT, MAI; (2) investigate why a frontier model (GPT) didn't pass — that's the highest-value gap; (3) minor wording polish only. ([frontier-miss], [thin-N])
binlog-failure-analysis	3/5	GPT, Haiku, MAI	0.24	2	0	0	1	100%	-111,566	-2.2	-1.6	KEEP-POLISH · 🟢 Solid majority win on GPT, Haiku, MAI. Losses/ties are concentrated in the non-passing cells (Opus, Sonnet), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on Opus, Sonnet; (2) investigate why a frontier model (Opus) didn't pass — that's the highest-value gap; (3) minor wording polish only. ([frontier-miss], [thin-N])
incremental-build	3/5	GPT, Sonnet, MAI	0.24	2	0	0	1	100%	+56,173	+2.4	+2.6	KEEP-POLISH · 🟢 Solid majority win on GPT, Sonnet, MAI. Losses/ties are concentrated in the non-passing cells (Opus, Haiku), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on Opus, Haiku; (2) investigate why a frontier model (Opus) didn't pass — that's the highest-value gap; (3) minor wording polish only. ([frontier-miss], [thin-N])
check-bin-obj-clash	2/5	Opus, GPT	0.20	2	1	0	1	100%	+154,323	+4.4	+4.4	TRIM-COST · 🟠 Passes but heavy (+154,323 tok). Value is real; cost isn't justified. Trim: (1) cut redundant/whole-file reads — point to specific sections; (2) replace "explore everything" with a targeted checklist; (3) move deep reference material behind links instead of inlining it. ([costly], [thin-N])
msbuild-modernization	3/5	Sonnet, Haiku, MAI	0.16	1	1	0	1	100%	+37,125	+0	-1.4	KEEP-POLISH · 🟢 Solid majority win on Sonnet, Haiku, MAI. Losses/ties are concentrated in the non-passing cells (Opus, GPT), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on Opus, GPT; (2) investigate why a frontier model (Opus, GPT) didn't pass — that's the highest-value gap; (3) minor wording polish only. ([both-frontier-miss], [frontier-regressed], [thin-N])
msbuild-antipatterns	0/5	—	-0.05	13	4	0	4	30%	+8,217	-0.15	+0.3	FIX-DISCOVERY · 🔴 Barely invoked (30% of trials) — the model isn't calling it, so content can't help yet. Fix discovery first: (1) rewrite the description: with concrete trigger phrases a user would actually type; (2) add "USE WHEN…" bullets; (3) shorten the name/scope so it competes for the model's attention. Re-measure before editing the body. ([both-frontier-miss], [frontier-regressed], [not-invoked])
eval-performance	0/5	—	-0.24	2	3	0	1	100%	+52,485	+2.8	+1.6	FIX-REGRESSION · 🔴 Regresses ~60% of trials (worse than baseline), losing on Opus, GPT, Sonnet. The skill is over-applying. Fixes: (1) add explicit stop-conditions ("do NOT act when…"); (2) narrow the trigger to the exact scenario it helps; (3) demote prescriptive edits to suggestions the agent can decline. Prioritise the frontier miss (Opus, GPT). ([both-frontier-miss], [frontier-regressed], [thin-N])

⚠️ Coverage gap — 4 skills excluded from the cross-family run

The matrix scores a skill only if it has a tests/dotnet-msbuild/<skill>/eval.vally.yaml. The following dotnet-msbuild skills ship a SKILL.md but were never measured — so they carry zero evidence of impact (not counted in the pass rate or scorecard above):

• Has a base eval.yaml but no eval.vally.yaml — add a cross-family Vally config to fold them into the matrix: extension-points, item-management, property-patterns, target-authoring

Until these have an eval.vally.yaml, we cannot tell whether they help, hurt, or are inert. Recommended: add the cross-family eval configs and re-run so they appear in the next scorecard.

---

_{Generated from the cross-family Call-to-Action report (CALL-TO-ACTION.md §4–§5; companion IMPACT-ANALYSIS.md). Regenerate the underlying tables with node deep-metrics.mjs "$env:TEMP\cf-ci" agg-ci → node gen-cta-tables.mjs agg-ci. Numbers are directional where avgN is low; treat single-trial cells as hypotheses to confirm with more runs.}

[github-actions]

🎯 Issue Triage

Summary: This is a cross-family skill evaluation report for the dotnet-msbuild plugin, covering 14 skills across 5 model families (419 scored cells). It identifies actionable items: 2 P0 regressions (msbuild-antipatterns FIX-DISCOVERY, eval-performance FIX-REGRESSION), 1 P1 cost trim (check-bin-obj-clash), 5 exemplars to keep, and 4 unevaluated skills needing eval.vally.yaml configs.

Type: task — tracking/work item capturing evaluation-driven improvement actions

Assigned owners: @ViktorHofer and @JanKrivanek assigned directly. Team @dotnet/skills-msbuild-reviewers (not assignable as a team) should also review.

Confidence: High — issue clearly maps to area-msbuild, type is unambiguous, CODEOWNERS match is exact.

Similar issues: No obvious duplicates found.

Triage details

• Area mapping: Issue title and body are explicitly about dotnet-msbuild plugin skills; CODEOWNERS /plugins/dotnet-msbuild/ maps to @ViktorHofer @JanKrivanek @dotnet/skills-msbuild-reviewers``
• CODEOWNERS path match: /plugins/dotnet-msbuild/ and /tests/dotnet-msbuild/
• Label area-msbuild already present — kept; added task and Triaged
• Confidence rationale: Issue is auto-generated from eval pipeline with precise scope; area, type, and owners are unambiguous
• Suggested next steps: Address P0s first (eval-performance regression, msbuild-antipatterns discovery), then P1 cost trim on check-bin-obj-clash. Add eval.vally.yaml to the 4 unmeasured skills (extension-points, item-management, property-patterns, target-authoring).

Generated by Issue Triage for issue #896 · 45.8 AIC · ⊞ 28.4K · ◷

[AbhitejJohn]

Verifying fixes for this plugin's skills

To re-run the cross-family evaluation and confirm a change moves the pass rate, cherry-pick the dispatch workflow from #920 and dispatch it scoped to this plugin:

git fetch origin abhitejjohn-cross-family-eval-workflow
git cherry-pick 7c97b7346          # single commit in #920
git push origin HEAD

gh workflow run evaluation-run.yml --ref <your-branch> \
  -f plugins="dotnet-msbuild" \
  -f executors="opus,gpt,sonnet46,haiku,mai" -f runs=3

To verify a single skill faster, scope with skills (the skill directory name, e.g. one of the skills listed above):

gh workflow run evaluation-run.yml --ref <your-branch> \
  -f skills="<skill-dir>" -f executors="opus,gpt" -f runs=3

Each skill is run skilled-vs-baseline across the 5-family executor matrix, with the judge never in the executor's family (default judge = latest Opus; GPT when Opus executes). Results assemble into a cross-family pass matrix you can compare against the numbers in this issue. Full inputs/usage: #920.