Trim technology-selection skill cost via gated progressive disclosure#931
Trim technology-selection skill cost via gated progressive disclosure#931luisquintanilla wants to merge 1 commit into
Conversation
Cuts SKILL.md from 355 to 101 lines and moves per-branch depth into references/ (classic-ml, llm, agentic, rag), read only when writing implementation code. Removes the Step 6 `dotnet build -warnaserror` + `dotnet test` mandate that induced build-fix-rebuild loops -- the root cost driver flagged in dotnet#889 (measured: 1.56M tok / 46 turns / 8 builds vs a 220K no-skill baseline on the LLM stimulus). Selected by offline head-to-head Vally runs (runs=3, two model families), not guessed. Versus the current skill: plan-only cost cut ~3-4.5x, eliminates the sk- key leak, best reliability 16/18 on haiku, -37% tokens on the LLM task on GPT, and zero task regression on either family. A cheaper single-file trim was rejected because its judge score dropped to 0/3 and it still leaked sk-. Frontmatter description is unchanged so discovery is held constant (Call% already 100%). Part of dotnet#889 (the dotnet-ai portfolio tracker): this implements the TRIM-COST (P1) "address first" item for technology-selection only. The four mcp-csharp-* skills tracked in dotnet#889 remain separate work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a78b17bd-5e28-4e85-b248-38626ebb5eee
9d4172f to
4516118
Compare
There was a problem hiding this comment.
Pull request overview
This PR trims the plugins/dotnet-ai/skills/technology-selection skill by moving deep per-branch guidance into references and gating those reference reads to implementation-only scenarios, reducing token/turn/tool overhead for plan-only requests.
Changes:
- Rewrites
SKILL.mdinto a shorter decision-core with “plan-only vs implementation” gating guidance. - Adds per-branch reference docs (
classic-ml,llm,agentic,rag) to hold implementation details and minimal code shapes. - Updates validation/anti-pattern checklists to align with the progressive disclosure approach.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| plugins/dotnet-ai/skills/technology-selection/SKILL.md | Replaces verbose workflow with a compact decision tree + gated reference usage. |
| plugins/dotnet-ai/skills/technology-selection/references/classic-ml.md | Adds ML.NET packages/guardrails + minimal example. |
| plugins/dotnet-ai/skills/technology-selection/references/llm.md | Adds MEAI LLM integration packages/guardrails + minimal example. |
| plugins/dotnet-ai/skills/technology-selection/references/agentic.md | Adds Agent Framework packages/guardrails + minimal example. |
| plugins/dotnet-ai/skills/technology-selection/references/rag.md | Adds RAG packages/guardrails + minimal query shape + planning checklist. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| var queryEmbedding = await embeddingGenerator.GenerateAsync(question, ct); | ||
| var hits = await collection.SearchAsync(queryEmbedding, top: 5, new() { }, ct); | ||
| var grounded = hits.Where(h => h.Score >= 0.75); // minimum similarity threshold |
| .AsBuilder() | ||
| .UseFunctionInvocation() | ||
| .Use(inner => new RetryingChatClient(inner, maxRetries: 3)) | ||
| .Build()); |
| <PackageReference Include="Microsoft.Extensions.AI" Version="9.*" /> | ||
| <PackageReference Include="Azure.AI.OpenAI" Version="2.*" /> <!-- or OpenAI / Azure.AI.Inference / OllamaSharp --> | ||
| <PackageReference Include="Microsoft.ML.Tokenizers" Version="2.*" /> <!-- client-side token budgeting --> |
| <PackageReference Include="Microsoft.Extensions.AI" Version="9.*" /> | ||
| <PackageReference Include="Microsoft.Agents.AI" Version="1.*-*" /> <!-- prerelease: dotnet add --prerelease --> | ||
| <PackageReference Include="Azure.AI.OpenAI" Version="2.*" /> <!-- or another MEAI provider --> |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
plugins/dotnet-ai/skills/technology-selection/references/rag.md:32
- The sample mixes a named argument (
top: 5) with positional arguments that follow (new() { }, ct), which is invalid C# syntax. Use either all positional arguments or name the remaining arguments as well so the snippet compiles.
var queryEmbedding = await embeddingGenerator.GenerateAsync(question, ct);
var hits = await collection.SearchAsync(queryEmbedding, top: 5, new() { }, ct);
var grounded = hits.Where(h => h.Score >= 0.75); // minimum similarity threshold
plugins/dotnet-ai/skills/technology-selection/references/llm.md:35
- This reference is scoped to “no tools”, but the minimal shape enables function/tool invocation (
UseFunctionInvocation()). Either remove it here or move this snippet under the agentic reference to keep the guidance consistent.
.AsBuilder()
.UseFunctionInvocation()
.Use(inner => new RetryingChatClient(inner, maxRetries: 3))
plugins/dotnet-ai/skills/technology-selection/references/llm.md:11
- The package list doesn’t include
Azure.Identity, but the minimal example usesDefaultAzureCredentialwhich comes from that package. Adding it avoids a copy/paste compilation failure.
<PackageReference Include="Microsoft.Extensions.AI" Version="9.*" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.*" /> <!-- or OpenAI / Azure.AI.Inference / OllamaSharp -->
<PackageReference Include="Microsoft.ML.Tokenizers" Version="2.*" /> <!-- client-side token budgeting -->
plugins/dotnet-ai/skills/technology-selection/references/agentic.md:11
- The package list doesn’t include
Azure.Identity, but the minimal example usesDefaultAzureCredential. AddingAzure.Identityprevents a copy/paste compilation failure.
<PackageReference Include="Microsoft.Extensions.AI" Version="9.*" />
<PackageReference Include="Microsoft.Agents.AI" Version="1.*-*" /> <!-- prerelease: dotnet add --prerelease -->
<PackageReference Include="Azure.AI.OpenAI" Version="2.*" /> <!-- or another MEAI provider -->
Cross-family Vally evaluation —
|
| Executor | Model | Judge | Result | Win rate (W/T/L) | Mean preference | Trials |
|---|---|---|---|---|---|---|
| opus | claude-opus-4.8 | gpt-5.5 | ❌ inconclusive | 100% (12/0/0) | +65.0% [45.4, 84.6] | 12 (3 unmatched) |
| gpt | gpt-5.5 | claude-opus-4.8 | ✅ credibly better | 94.4% (17/1/0) | +67.8% [50.7, 84.9] | 18 |
| sonnet46 | claude-sonnet-4.6 | claude-opus-4.8 | ❌ inconclusive | 100% (14/0/0) | +78.6% [61.3, 95.8] | 14 (1 unmatched) |
| haiku | claude-haiku-4.5 | claude-opus-4.8 | ✅ credibly better | 77.8% (14/3/1) | +58.9% [35.6, 82.1] | 18 |
| mai | mai-code-1-flash-picker | claude-opus-4.8 | ❌ inconclusive | 66.7% (8/3/1) | +33.3% [7.7, 58.9] | 12 (6 unmatched) |
Cross-family pass rate: 2/5 (40%) at the strict conclusive bar (gpt, haiku).
Interpretation
The trimmed skill is directionally preferred over baseline in all 5 executors — win rate 67–100%, mean preference +33% to +79%. The three non-passes are flagged inconclusive due to unmatched trajectories (3 / 1 / 6 unmatched trials), not losses — the treatment did not lose a single conclusive comparison against baseline anywhere.
Against the 12% dotnet-ai pass rate that motivated #889, this is a strong positive signal: where the comparison is conclusive the trimmed skill is credibly better, and no executor prefers the baseline. The remaining action is reducing unmatched trajectories (opus/sonnet46/mai) so those cells reach conclusiveness rather than any concern about skill quality.
|
@luisquintanilla Thanks for taking this on — the progressive-disclosure restructure is a genuinely nice win. Trimming the top-level skill from ~263 to ~102 lines and moving per-branch detail into references that only load at implementation time directly targets the cost problem this was meant to fix. I ran the change two ways to sanity-check it: a cross-family evaluation (five different models each doing the task, with a separate model scoring them), and several independent AI code reviews plus an automated skill-quality check. Sharing what came back in case it's useful. Evaluation results were encouraging. Across all five models, the trimmed skill was preferred over the current version. Two cleared the strict "credibly better" bar; the other three are positive-but-inconclusive due to unmatched trajectories, not losses — no model preferred the baseline in any conclusive comparison. Full run: https://github.com/dotnet/skills/actions/runs/29985588613
A few things worth a look before merging, grouped by severity. 🔴 Blocking
🟡 Non-blocking
⚪ Minor (from the automated skill-quality check — wording being a touch too absolute)
(Three other compile-level items — the None of this is blocking on capability — the direction is clearly right. The branch-routing gap above is the highest-value fix; the rest is polish. Thanks again for pushing this forward! |
Proposal: trim the
technology-selectionskill (dotnet/skills #889) — measured, not guessedSkill:
plugins/dotnet-ai/skills/technology-selectionIssue: dotnet/skills#889 — flagged TRIM-COST (P1),
"Address first": passes but heavy (+208,946 tok, +3.73 turns, +3.47 tools), passes on GPT only
(1/5 families), Impact 0.57.
Owners: @luisquintanilla, @artl93
TL;DR
We did not guess. We reproduced the eval offline with
@microsoft/vally-cli, calibrated thecurrent skill (V1) to confirm our harness reproduces the report's direction, then ran candidate
rewrites head-to-head at runs = 3 on the full scorecard (deterministic behavior graders, the
LLM judge rubric, and tokens/turns/tools). The winner is V6 — "gated progressive disclosure":
a lean decision core with per-branch detail in
references/, read only when writing implementationcode. Measured against the current skill and three other rewrites, V6:
by not reading references for "show me a plan / do not write code" requests;
sk-key leak and posts the best LLM-guardrail and ML.NET judge scores byre-reading the branch reference exactly where code is written;
versus the current skill on any task.
A cheaper rewrite (V2) that trimmed more tokens but regressed the judge to 0/3 and still leaked
sk-was rejected by the numbers — the "not in a vacuum" guarantee in action.Root cause: the cost is induced behavior, not the 22 KB body
The skill body is ~5.5K tokens. The +208K overhead comes from what the skill makes the agent do.
Its Step 6 "Validate the implementation" mandates, on every task:
and repeats
dotnet build -c Release -warnaserror completes cleanlyin the Validation checklist.-warnaserrorturns any warning into a hard failure, triggering fix→rebuild loops. The skill alsowalks all four technology branches (classic ML / LLM / agentic / RAG) even when only one applies,
and pushes full implement+build+test even on stimuli that explicitly say "show me your plan and
architecture only — do not write any code."
Measured proof (V1, claude-haiku-4.5, LLM-summarization stimulus — one representative run;
runs = 3 median 1,273,780 tok / 44 turns / 63 tool calls):
dotnet build— abuild-fix-rebuild loop that rewrote
Program.cs/.csprojrepeatedly.output-contains: IChatClient→ FAIL) and leakedsk-(graderoutput-not-contains: sk-→FAIL). The cost bought a wrong answer. Across runs = 3, V1's LLM task leaks
sk-in 1/3 runsand the judge passes only 2/3.
This is exactly the report's "passes GPT only, cost not justified," reproduced and explained.
Where the tokens actually go (turn-driven cache reads)
Decomposing the token totals (input+output, from each trial's
session.shutdownusage) shows theoverhead is ~98 % cache-read tokens, not fresh computation:
Cache reads accumulate per turn — every extra agent turn re-reads the growing context. So the
real cost lever is turn count, and the build-fix-rebuild loop is what inflates turns (V1's
LLM run: 46 turns). Removing the mandatory build/test loop cuts turns → cuts cache-read volume →
cuts total tokens, and removes the loop's opportunities to go wrong (the
sk-leak happened deepin that loop). Cost, reliability, and correctness move together. (We report total tokens for
#889-comparability, but track turns + fresh tokens as the load-bearing, lower-variance cost axes.)
Method (measure, don't guess)
eng/vally-adapter/run-vally-evals.sh dotnet-ai technology-selectionovertests/dotnet-ai/technology-selection/eval.vally.yaml(6 stimuli; deterministic gradersoutput-contains/-not-contains/exit-successplus an LLMpromptrubric). baseline =no skill; skilled = the one skill dir under test.
claude-haiku-4.5, judgegpt-5.5.gpt-5.5, judgeclaude-opus-4.6.description, so discovery is heldconstant — Call% is already 100%, not the problem). Each variant = its own skill dir + a literal
skilled.environment.skillspath.in deterministic behavior graders, no Impact/judge drop, no new timeouts. Cost is optimized on a
multi-dimensional objective, never alone.
Candidate ladder
-warnaserrorbuild/test loop)references/{classic-ml,llm,agentic,rag}.md, read for the selected branchAll variants preserve the deterministic grader tokens (
new MLContext(seed:,TrainTestSplit,IChatClient,Temperature, nosk-,MaximumIterations,Microsoft.Agents.AI,Microsoft.Extensions.AI,chunk,embedding) in the branch that emits them, and keep thefrontmatter
descriptionbyte-identical to V1 (discovery held constant).Results — HAIKU (mid) leg, runs = 3
Executor
claude-haiku-4.5, judgegpt-5.5, 3 runs/stimulus. Verdict = all-graders-pass(deterministic and the LLM
promptrubric). Tokens/turns are per-stimulus medians.(Numbers from vally's own
report.md;sk-leak = theoutput-not-containsgrader failing.)Reliability / behavior (pass rate out of 3): V1 from the calibration run; V2/V3/V6
co-measured in one run (baseline v2) so the V6-vs-V2-vs-V3 comparison has no cross-run drift.
sk-· judge 2/3sk-2/3 · judge 0/3† The
ML.NEToutput-containsgrader inspects the literal last assistant message fornew MLContext(seed:/TrainTestSplit; a run that finished with a prose summary fails it eventhough the code was written. All four variants fail it (0/3) — it is a grader-brittleness
constant, not a V6 regression. On the load-bearing signals V6 is best: it completes ML.NET 3/3
and the judge passes 3/3 (every other variant completes ≤1/3, judge ≤1/3).
Cost on the 3 light / plan-only tasks (all variants pass — apples-to-apples), median tokens / turns:
‡ V6 removes V3's reference tax on the two explicit plan-only tasks completely (back to the V2
floor: 28.8K/2 and 29.9K/2 vs V3's 92.5K/6 and 136.7K/9 — a ~3–4.5× reduction). The
Reject LLMtask is borderline (it says "I want to use GPT…", not "plan only"), so V6 sometimeswrites the ML.NET alternative in code (one of three runs spiked to 1.69M); it still passes 3/3.
This is the residual cost variance, called out honestly.
Decision & winner: V6 (gated progressive disclosure)
Applying the decision rule — largest cost cut subject to no regression in behavior graders, judge,
or reliability — V6 wins, and the losers are rejected by the numbers, not by taste:
security guardrail: on the LLM task the judge fails all 3 runs (
prompt0/3, vs V1's 2/3)and it leaks
sk-in 2/3 runs. "Trims tokens but quality suffers" is exactly the tradeoffwe were told to reject.
a 3–4.5× token/turn tax on plan-only tasks by reading a reference it doesn't need — and V2
proves those plan rubrics pass with no reference at all, so the tax is pure waste.
matches V2's cost floor on the plan-only tasks and matches-or-beats V3's safety on
implementation (LLM 0 leaks, best
output-contains2/3, judge 2/3), while posting the bestoverall reliability (16/18) and the best ML.NET outcome (completes 3/3, judge 3/3). No axis
regresses versus V1; safety, quality, and reliability all improve.
Mechanism confirmed: the
sk-leak happens deep in the implementation loop, so the fix is tore-anchor the guardrail exactly there (read
llm.mdwhen writing LLM code) — not to pay for areference on a plan the agent will never code. V6 does precisely that, and the measured numbers
confirm it: safety preserved where code is written, cost cut where it isn't.
Results — FRONTIER (GPT) leg — the "no regression where V1 already passes" check
The report says the skill "passes on GPT only." So the frontier leg is a regression guard:
executor
gpt-5.5, cross-family judgeclaude-opus-4.6, V1 vs V6, runs = 2. On GPT the heavytasks actually complete (both variants 12/12 trials complete, 0 leaks).
Verdicts are identical V1↔V6 on every task (cross-family judge agrees): NL ✅, RAG ✅, Reject ✅,
LLM 🟡 (
output-not-contains2/2 — no leak —prompt2/2), Agentic ❌ and ML.NET ❌ only on thebrittle
output-containsgrader whileexit-success2/2 and the judgeprompt2/2 both pass. Notask V1 passes on GPT regresses under V6.
And V6 still cuts cost on GPT — the trim is not a haiku artifact:
V6's LLM
output-containsalso improves to 5/5 on GPT (V1: 3/5). Net: on the family where theskill already passes, V6 preserves every pass and reduces the LLM task's cost by more than a third.
Validity caveats (stated honestly)
descriptionchangesdon't surface as Call%. We kept
descriptionbyte-identical across variants; any triggeringchange needs a separate discovery probe. (Call% is already 100% — discovery is not the problem.)
relative within-harness comparison (same tasks/judge/harness across variants).
variant on the mid (haiku) tier — haiku is too weak to finish them regardless of skill, so
Agentic is non-discriminating there; the frontier (GPT) leg is where those tasks actually
complete. ML.NET's
output-containsgrader is brittle to prose-summary endings across allvariants (see †).
check. Token totals on heavy tasks are cache-read-dominated and noisy (§ "Where the tokens go"),
so cost claims lean on turns and the low-variance plan-only tasks.
sk-/quality wins are the load-bearing result; they reproduce across two independenthaiku runs (v123 and v236). V2's leak and V3's tax both replicated.
What ships (PR-ready)
plugins/dotnet-ai/skills/technology-selection/:git diff --stat: +85 / −339).references/:classic-ml.md,llm.md,agentic.md,rag.md(182 lines total),read only when implementing the selected branch.
descriptionbyte-identical to the current skill (discovery unchanged; Call% isalready 100%).
markdownlint-cli2config (0 issues).Net effect vs the current skill, by the numbers: removes the
dotnet build -warnaserror/dotnet testloop that was the cost driver; answers plan-only requests from the lean core (~29Ktok / 2 turns, a 3–4.5× cut); re-reads the branch reference while writing code, which stops the
sk-key leak and yields the best LLM/ML.NET judge scores; best overall reliability (16/18) onhaiku and −37% tokens on the LLM task on GPT — with no task regressing on either family.
Reproduce
Experiment files used (variants swap only
environment.skills): mid legclaude-haiku-4.5×gpt-5.5, runs 3 (V2/V3/V6 co-measured); frontier leggpt-5.5×claude-opus-4.6,runs 2 (V1 vs V6). Scorecards were recomputed directly from each trial's
events.jsonland matchedvally's own
report.mdexactly.PR scope
This PR is scoped to one of the five skills tracked in
#889: the TRIM-COST (P1) "address first" item
for
technology-selection. Issue #889 is adotnet-aiportfolio tracker and stays open forthe remaining four
mcp-csharp-*skills (STRENGTHEN / EFFICIENT-WIN) — this PR is a partial step,not the whole issue.
The change touches only the skill body (
SKILL.md355 → 101 lines + newreferences/); thefrontmatter
descriptionis byte-identical, so discovery/triggering is unchanged.