[skills-eval] dotnet11: 1 skill, 0% pass — 1 P1

[AbhitejJohn]

Context — cross-family skill evaluation: dotnet11

This issue is self-contained: it captures everything a skill author needs to act on the dotnet11 plugin without opening the full report.

What this measures. Every runnable skill in dotnet/skills was run through Vally (0.7) on a cross-family matrix: 5 executor model families — opus-4.8, gpt-5.5, sonnet-4.6, haiku-4.5, mai-flash — each judged by a different family (judge ≠ executor; default judge = latest Opus, or GPT when Opus is the executor). For every skill, a skilled run is compared against a baseline (no-skill) run and scored per executor. This removes single-model and self-judging bias, so a skill that only helps one model family — or only its own family's judge — is visible.

• Data source: cross-family CI grid (run 29228914412 + backfills) — 5 executors × 85 runnable skills, 419 scored cells over 84 skills.
• Row grain: one row per skill, aggregated across its (up to 5) executor cells. avgN is the mean trial count behind the cells (trials 1–17; higher = more statistically trustworthy). thin-N flags directional-only rows.

dotnet11 at a glance (portfolio scorecard)

Plugin	Skills	Cells	Pass	Impact	Tie-trials	Err	avg ΔTok	avg ΔTurns	avg ΔTools	Headline
dotnet11	1	5	0%	0.320	7	0	−3,721	−0.07	−0.20	Inert single skill

Insights. 1 skill(s); mean impact 0.32; 0/1 help ≥1 frontier model. No frontier-validated exemplar.

Address first: system-text-json-net11 — inert.

How to read the table

Each skill is scored skilled vs. baseline on these axes:

Signal	Column	What it means	Good
Breadth	Families ✓ (n/5)	How many of the 5 model families the skill helps (per-family pass count, max 5/5)	4–5 / 5
Where	Passed on	Which families passed. Frontier = latest Opus + latest GPT are bold; Sonnet 4.6 / Haiku 4.5 / MAI Flash are mid/low-weight	frontier ✓
Magnitude	Impact (−1…+1)	How strongly the judge prefers skilled over baseline	≥ 0.4
Decisiveness	Ties▵	Trials where the judge saw no difference → skill is inert	low
Safety	Loss▵	Trials where skilled was WORSE than baseline → skill misfires	~0
Reliability	Err	Trials that errored/crashed in setup or judging	0
Efficiency	ΔTok / ΔTurns / ΔTools	Extra tokens / agent turns / tool calls vs baseline	≤ 0
Confidence	avgN	Mean trials behind the verdict; low N = directional only	≥ 3
Invocation	Call%	Share of skilled trials where the model actually invoked the skill	~100%

Families ✓ is cell-level (max 5/5); Ties▵/Loss▵ are trial-level tallies summed across all families (including the ones where the skill failed). A high Families ✓ next to non-zero Loss▵ is not a contradiction — see Passed on and the Action text for where losses landed.

Action buckets (each skill has one primary action; [flags] note secondary concerns):

Bucket	Priority	Meaning
FIX-RELIABILITY	🔴 P0	Errored trials / no verdict — stabilize the harness before trusting the score
FIX-DISCOVERY	🔴 P0	Model doesn't invoke it (Call% < 50%) — a triggering/description problem
FIX-REGRESSION	🔴 P0	Skilled is worse than baseline on many trials — the skill misfires
ADD-DECISIVENESS	🟠 P1	Called ~100% but ties dominate, ~0 impact — inert; needs sharper behavioral steps
TRIM-COST	🟠 P1	Passes but with heavy token/turn overhead — trim verbosity
EXEMPLAR	🟢 keep	Broad, strong, reliable win — use as a template
EFFICIENT-WIN	🟢 protect	Wins and cuts turns/tools — the ideal shape
KEEP-POLISH	🟢	Solid majority win; minor polish + more trials
STRENGTHEN	🟡 P2	Marginal/mixed lift — sharpen triggers & success criteria

Per-skill actions

Skill	Families ✓	Passed on (frontier bold)	Impact	Ties▵	Loss▵	Err	avgN	Call%	ΔTok	ΔTurns	ΔTools	Action
system-text-json-net11	0/5	—	0.32	7	1	0	3	67%	-3,721	-0.07	-0.2	ADD-DECISIVENESS · 🟠 Called (~67%) but inert — ties are the plurality (7) and the judge sees no behaviour change vs baseline. It reads like reference prose. Fixes: (1) convert "here's what X is" into "when you see A, do B, verify with C" imperative steps; (2) add one concrete worked example; (3) add a verification/checklist step. If still inert after that, the guidance is likely already the model's default → retire. ([both-frontier-miss], [frontier-regressed])

---

_{Generated from the cross-family Call-to-Action report (CALL-TO-ACTION.md §4–§5; companion IMPACT-ANALYSIS.md). Regenerate the underlying tables with node deep-metrics.mjs "$env:TEMP\cf-ci" agg-ci → node gen-cta-tables.mjs agg-ci. Numbers are directional where avgN is low; treat single-trial cells as hypotheses to confirm with more runs.}

[github-actions]

🎯 Issue Triage

Summary: This is an automated skill evaluation report for the dotnet11 plugin. The single skill system-text-json-net11 scored 0% pass across all 5 model families with an impact of 0.32 and 7 tie trials, indicating the skill content is inert (reads as reference prose rather than imperative behavioral guidance).

Type: task — a concrete improvement work item: rewrite the skill as imperative steps, add a worked example, and add a verification checklist; retire it if still inert afterward.

Assigned owners: @JanKrivanek, @ViktorHofer (per CODEOWNERS for /plugins/dotnet11/skills/system-text-json-net11/)

Confidence: High — issue maps cleanly to area-dotnet11, the action bucket (ADD-DECISIVENESS) and specific skill are unambiguous.

Similar issues: No closely related open duplicates found.

Triage details

• Area mapping: system-text-json-net11 lives under /plugins/dotnet11/ → area-dotnet11
• CODEOWNERS path match: /plugins/dotnet11/skills/system-text-json-net11/ → @JanKrivanek @ViktorHofer``
• Confidence rationale: Single skill, clear evaluation data, unambiguous owners and area
• Suggested next steps: Convert descriptive prose to imperative "when A, do B, verify C" steps; add a concrete worked example; add a checklist step. If still inert, consider retiring the skill.

Generated by Issue Triage for issue #902 · 64.6 AIC · ⊞ 26.8K · ◷

[AbhitejJohn]

Verifying fixes for this plugin's skills

To re-run the cross-family evaluation and confirm a change moves the pass rate, cherry-pick the dispatch workflow from #920 and dispatch it scoped to this plugin:

git fetch origin abhitejjohn-cross-family-eval-workflow
git cherry-pick 7c97b7346          # single commit in #920
git push origin HEAD

gh workflow run evaluation-run.yml --ref <your-branch> \
  -f plugins="dotnet11" \
  -f executors="opus,gpt,sonnet46,haiku,mai" -f runs=3

To verify a single skill faster, scope with skills (the skill directory name, e.g. one of the skills listed above):

gh workflow run evaluation-run.yml --ref <your-branch> \
  -f skills="<skill-dir>" -f executors="opus,gpt" -f runs=3

Each skill is run skilled-vs-baseline across the 5-family executor matrix, with the judge never in the executor's family (default judge = latest Opus; GPT when Opus executes). Results assemble into a cross-family pass matrix you can compare against the numbers in this issue. Full inputs/usage: #920.