[cherry-pick source, do not merge] Cross-family skill eval — dispatch pre-merge

[AbhitejJohn]

⚠️ Cherry-pick source — NOT intended to merge

This PR exists so anyone can run a cross-family skill evaluation on their own branch before any of this lands on main. It is a single, self-contained commit designed to be cherry-picked. Do not merge it to main as-is (see "Why not merge" below).

What this is

A standalone cross-family evaluation workflow: every runnable skill is run through a matrix of executor models and judged head-to-head (skilled vs. baseline) by vally compare, with the judge never in the same model family as the executor:

• Default judge = claude-opus-4.8
• When Opus is the executor, judge switches to gpt-5.5

Each matrix cell = {executor} x {plugin} (optionally x {shard}). Results upload per cell and assemble into a cross-family pass matrix. Informational only — it never gates PRs.

How to run it (pre-merge, no merge required)

# 1. Get this commit onto your branch
git fetch origin abhitejjohn-cross-family-eval-workflow
git cherry-pick b8cab548b        # the single commit in this PR
git push origin HEAD

# 2. Dispatch against YOUR branch
gh workflow run evaluation-run.yml --ref <your-branch> \
  -f plugins="dotnet-msbuild,dotnet-test" \
  -f executors="opus,gpt,sonnet46,haiku,mai" \
  -f runs=3 -f workers=5 -f shards=1 -f timeout_minutes=300

GitHub runs this workflow version + inputs against your branch's code. Leave plugins empty to auto-discover every plugin with non-agent evals.

Run a single skill (or a few) instead of whole plugins

Pass skills (comma-separated skill directory names). It matches on the skill folder name — e.g. plugins/dotnet-msbuild/skills/msbuild-server → msbuild-server — and works with or without plugins:

# One skill, across whichever plugin contains it, on two executors:
gh workflow run evaluation-run.yml --ref <your-branch> \
  -f skills="msbuild-server" \
  -f executors="opus,gpt" -f runs=3

# A few named skills (plugin optional — narrows the search if given):
gh workflow run evaluation-run.yml --ref <your-branch> \
  -f skills="msbuild-server,incremental-build,check-bin-obj-clash" \
  -f executors="opus,gpt,sonnet46,haiku,mai" -f runs=3

• skills composes with plugins: give both to scope to a skill within a plugin; give only skills to let the workflow find it in any discovered plugin.
• A mistyped skill that matches nothing fails fast in prepare with ::error::Requested skill(s) not found … (typo protection) — no wasted matrix cells.
• Leave skills empty to evaluate every skill in the selected plugins (default).

Why it lives under evaluation-run.yml

workflow_dispatch only registers workflows whose file exists on the default branch. evaluation-run.yml (the reusable Vally eval harness added by #877) already exists on main and is dispatchable — putting the matrix under that filename makes it runnable via --ref <branch> with no merge. A brand-new filename would 404 on dispatch until merged.

Note: the earlier host vally-evaluation.yml was removed from main by #877; evaluation-run.yml is its successor and the current dispatch host.

Inputs

Input	Default	Meaning
plugins	(empty = all)	Comma-separated plugin names
skills	(empty = all in plugins)	Comma-separated skill directory names (e.g. msbuild-server). Works with or without plugins
executors	opus,gpt,sonnet46,haiku,mai	5-family published matrix (default). Allowlist also accepts gpt53codex (opt-in)
runs	3	Trials (n) per eval
workers	5	Parallelism within a cell
shards	1	Split a plugin's skills across N parallel cells
timeout_minutes	300	Per-cell job timeout (hard cap 360)

Compatible with #877 (merged)

#877 (Vally harness migration) is merged to main. This workflow auto-derives the eval-spec filename from dotnet-skills.experiment.yaml's evals: glob, so discovery works whether the repo globs eval.yaml (current main) or eval.vally.yaml (older branches) — no edit needed. The model:/runs: overrides are patched via anchored sed (leaves judge_model: untouched; the judge is applied by adapt.mjs --judge-model).

Why NOT merge

This file replaces main's reusable evaluation-run.yml and drops its workflow_call trigger (the path invoked by evaluation.yml's discover / vally-evaluate job). Merging it would break the PR-eval path. This branch carries the dispatch-only matrix under that filename purely so it is dispatchable pre-merge. Keep it as a cherry-pick / dispatch source only.

Validation

• actionlint (pinned v1.7.7) exits 0.
• Skill-filter logic verified locally against the real tests/ tree: single-skill (with/without plugin), multi-skill, no-filter (full plugin = 18 skills), and typo-rejection (::error:: + non-zero exit) all behave correctly.
• Rebased onto latest main (78054c116, post-Migrate LLM evals to the Vally harness #877) and smoke-tested run 29874307929 dispatched via --ref on this branch: prepare ✅, cell gpt--dotnet-nuget ✅ (full vally eval + adapt.mjs judge), summary ✅ — proving pre-merge cherry-pick-and-run works end to end against current main.

[Copilot]

Pull request overview

Adds a standalone workflow_dispatch-driven “cross-family” Vally evaluation workflow intended to be cherry-picked onto feature branches to run executor×plugin (×shard) comparisons with an enforced judge-family separation.

Changes:

• Replaces the existing reusable vally-evaluation.yml shape with a dispatch-only workflow that builds an executor×plugin×shard matrix and runs vally experiment run + adapt.mjs per cell.
• Adds sharding support by round-robining per-plugin eval specs into per-cell --eval-filter lists.
• Adds a summary job that downloads all cell artifacts and generates a cross-family pass matrix + JSON summary artifact.

Show a summary per file

File	Description
.github/workflows/vally-evaluation.yml	Reworks the workflow into a dispatchable cross-family evaluation matrix with per-cell execution and a results summary job.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 1/1 changed files
• Comments generated: 3