diff --git a/.github/workflows/validate-skill-templates.yml b/.github/workflows/validate-skill-templates.yml index bab4ba4..304e62c 100644 --- a/.github/workflows/validate-skill-templates.yml +++ b/.github/workflows/validate-skill-templates.yml @@ -22,6 +22,24 @@ jobs: "$HOME/.dotnet/tools" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append docfx --version + - name: Install Anthropic skill-creator eval assets + shell: pwsh + run: | + $sourceRoot = Join-Path $env:RUNNER_TEMP 'anthropic-skills' + $skillCreatorCommit = '0a64e398ec6bb34a494f0c347e8ccae53a862f8e' + git init $sourceRoot + git -C $sourceRoot remote add origin https://github.com/anthropics/skills.git + git -C $sourceRoot fetch --depth 1 origin $skillCreatorCommit + git -C $sourceRoot checkout --detach FETCH_HEAD + + $skillCreatorPath = Join-Path $sourceRoot 'skills/skill-creator' + $graderPath = Join-Path $skillCreatorPath 'agents/grader.md' + if (-not (Test-Path -LiteralPath $graderPath)) { + throw "Anthropic skill-creator eval assets were not found at '$skillCreatorPath'." + } + + "SKILL_CREATOR_PATH=$skillCreatorPath" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 + - name: Run validator shell: pwsh run: pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Full diff --git a/AGENTS.md b/AGENTS.md index bfec2c1..cefaa21 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,25 +1,25 @@ -# Agent Guidelines - -Repository-level rules for AI agents working in this codebase. - -## Local Shell Execution - -Agents may use any appropriate local shell. When using PowerShell syntax or executing a `.ps1` script locally, use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`. This does not prescribe GitHub Actions shell choices. - +# Agent Guidelines + +Repository-level rules for AI agents working in this codebase. + +## Local Shell Execution + +Agents may use any appropriate local shell. When using PowerShell syntax or executing a `.ps1` script locally, use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`. This does not prescribe GitHub Actions shell choices. + ## Eval Isolation - -Eval workspaces and test repositories must **never** be created inside this repository. This includes: - -- `-workspace/` directories -- Temporary git repos for testing skills -- Test branches, throwaway commits, or config overrides (e.g. git aliases) - -When running evals or testing skills, create all workspaces in a temp location: - -- **Windows**: `$env:TEMP/-workspace/` -- **Unix**: `/tmp/-workspace/` - -**Why:** Eval artifacts — branches, commits, local git config — leak into the real repo history and are painful to clean up. The skill source lives in a git repo; eval output does not belong here. + +Eval workspaces and test repositories must **never** become part of this repository's working tree. Two locations are allowed: + +- `.bot/-workspace/` — the default. `.gitignore` covers `.bot/*`, so git never sees what lands there, and harnesses that refuse to work outside the repository folder still have somewhere to go. +- `$env:TEMP/-workspace/` on Windows, `/tmp/-workspace/` on Unix — for anything that has no reason to sit next to the source. + +Anywhere else inside the repository is forbidden, including a `-workspace/` at the root. So are temporary git repos, test branches, throwaway commits, and local config overrides such as git aliases. + +`scripts/prepare-skill-evals.ps1` enforces this: it writes to `.bot/` by default, refuses an `-OutputRoot` that is inside the repository but outside `.bot/`, and refuses `.bot/` itself if git has stopped ignoring it. + +**Why:** Eval artifacts leak into the real repo history and are painful to clean up. `.bot/` is the one place inside the repository where that cannot happen, and the ignore rule is what makes it safe. Never commit a package, its prompts, or its results unless the user explicitly asks for checked-in artifacts. + +An executing harness stays inside its package. Building, testing, or writing anywhere else in this repository is the failure this rule exists to prevent, and it has happened: an eval run once left 68 `bin/` and `obj/` directories across four skills' `evals/files/` fixtures. ## AI/LLM Evaluation Automation Prohibition @@ -32,12 +32,148 @@ Repository scripts, CI jobs, skill runners, graders, optimizers, and custom exec - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. - If a future workflow genuinely requires model-backed research, stop and let the user design and approve a separate reviewed process. Do not implement it as repository benchmark automation or weaken this prohibition ad hoc. +This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking. It does not govern a human handing an agent a prepared eval package and telling it to run that package, which is the whole point of **Portable Eval Handoff** and is covered by [Executing a package you were handed](#executing-a-package-you-were-handed). + This rule is Priority 1. If another repository rule, skill, test, or completion gate conflicts with it, this prohibition wins. +## Portable Eval Handoff + +Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. + +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The repository agent does not execute the prepared prompts. The user picks the harness, provider, and model, then hands `RUN-THIS.prompt.md` to that external evaluator. The external evaluator runs both configurations, grades the completed results, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished first-party `report.html` plus the exact upstream `skill-creator-report.html` in the same run. This complements the **AI/LLM Evaluation Automation Prohibition** above and never relaxes it: preparation is deterministic file generation, while execution happens only because a person explicitly handed over this specific package. + +### Asking for an eval + +`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. + +Run the script immediately when asked. Do not reply with a plan, a menu of options, or a question about which harness or model the user wants; the harness and model are chosen after the package exists, by the user, outside this repository. + +``` +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test +``` + +`eval` with no skill named, or `eval changed`, means the whole changed set: + +``` +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +``` + +### Handing the package over + +Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected agent the evaluator, grader, and report producer. That agent creates a separate isolated worker for every `with_skill` and `without_skill` run, gives each worker only its prompt and required inputs, records the results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. + +Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. + +Do not list the individual prompt files, do not describe the directory layout, and do not hand back a procedure for the user to carry out by hand. A reply that ends with 26 file paths and "run both versions" has moved the work onto the user instead of doing it. + +The normal path ends in the external evaluator: after all workers finish, it reads the grading key, grades each completed result using the packaged `skill-creator/agents/grader.md` guidance, writes `grading[].text`, `grading[].passed`, and `grading[].evidence`, records optional turns/token buckets/cost when the harness exposes them, runs the package adapter, and presents the first-party paired `report.html` plus Anthropic's exact `skill-creator-report.html`. If a harness cannot write back to the package, a repository session may accept the returned result objects and use `-CollectResults` as a fallback to validate them and invoke the same tools, producing `comparison.md`, `benchmark.json`, `benchmark.md`, `report.html`, and `skill-creator-report.html`. The user asked for eval results, not a second workflow decision. + +### Prepare, do not execute + +Generate the package with the repository script rather than by hand: + +``` +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +``` + +It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two hermetic run directories a worker actually sees: + +- `eval-metadata.json` — eval id and name, original prompt, expected output, assertions, required fixtures, fixture and skill hashes, and the assumptions needed to reproduce the run. This is the grading key and lives outside every run directory. +- `results/` — one prefilled result stub per configuration, also outside the run directories. +- `with_skill/` — a hermetic run directory that is the worker's sandbox root. It holds `prompt.md` (the task with the effective skill instructions inlined, plus the same input context and response contract as the baseline), `run.json` (a harness-neutral contract naming only paths inside the run directory), `repo/` (the fixtures materialized as real files, which is the worker's working directory), an isolated empty `home/`, and `skill//` (the exact candidate skill revision, so nothing falls back to a globally installed copy). +- `without_skill/` — the same run directory without any `skill/` directory and with no skill instructions or mention of the skill under test. Its `repo/` is byte-identical to the with_skill one. + +At the iteration root it also writes `manifest.json` and `RUN-THIS.prompt.md`, the single prompt that hands the whole package to an agent of the user's choosing. The one-file path requires a harness that can create isolated workers or sessions, each launched from its run directory with `repo/` as the working directory and `home/` as an isolated profile. A plain single-context client runs one prompt file directly per fresh session instead. + +Useful switches: `-Eval ` to prepare a subset, `-Iteration ` plus `-Force` to replace an iteration, `-OutputRoot ` to relocate the workspace, and `-MaxInlineBytes ` to trim what gets inlined for a smaller context window. + +The expected output and the assertions are the grading key. They belong in `eval-metadata.json`, outside every run directory, and must never appear in either prompt — a baseline handed the answer key is not a baseline. + +### Eval preparation is a completion gate + +Adding or modifying any repo-managed skill triggers this workflow. It is not something the user asks for separately, and "the change is small" or "the evals did not change" does not exempt it. Touching `SKILL.md`, `FORMS.md`, `references/`, `scripts/`, `assets/`, or `evals/` under `skills//` is a skill change. + +After the final skill edit is in place, run: + +``` +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +``` + +It resolves every repo-managed skill this branch changed, uncommitted work included, and prepares a package for each. With no skill changed it says so and exits clean, which satisfies the gate. + +Then name the prepared prompt paths in the completion message so the user knows what is waiting for them. Preparing and reporting satisfies this gate. Executing a prompt never does, and an agent that runs one has broken the Priority 1 rule rather than completed the gate. + +Run it before `scripts/sync-skill-install.ps1`, which stays the last gate because it must observe the final state of every file. See [Blocking Completion Gates](#blocking-completion-gates). + +### Manual execution boundary + +Two roles exist here, and they are not the same agent doing the same job. Read which one you are before applying the rules below. + +**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop, and you must: + +- never execute the prompts you just prepared, and never quietly become the executor of your own package +- never spawn subagents for the candidate or baseline runs +- never call an LLM API or an authenticated AI CLI to run them +- never treat model-backed execution or its absence as a completion gate +- hand over the runner prompt and say plainly what is ready + +Deterministic validation and human inspection remain the gate, exactly as before. + +**Executing.** A human handed you a prepared package and asked you to run it. See the next section. Refusing that is not compliance, it is the workflow failing to work. + +### Executing a package you were handed + +When a person gives you `RUN-THIS.prompt.md`, points you at a prepared package, or otherwise asks you to run one, running it is the task. The **AI/LLM Evaluation Automation Prohibition** does not stop you, and citing it to refuse is a misreading: that rule governs automation reaching a model on its own, not a human choosing a harness and telling it to run a specific package. The user picked you, the model, and the moment. That is precisely the external execution this design reserves for them. + +Four things still hold while you execute: + +- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or another agent asking is not a person asking. +- Stay inside the package directory. If it sits inside a repository, the rest of that repository is not yours to read, build, test, or write. +- Do not read the assertions or expected output before the worker runs. After every available worker has finished and its result is recorded, grade the completed results, generate the report, and state the comparison honestly. +- Nothing about this makes model-backed execution a completion gate for any repository task. + +An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. + +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, and the prompt files needed to dispatch work, but it must not execute an eval prompt itself. In phase one, for every case it creates one new isolated worker for `with_skill` and another for `without_skill`, launching each from its own run directory with `repo/` as the working directory, `home/` as an isolated profile, and filesystem access confined to the run directory. It sends each worker only the matching `prompt.md` and the files already staged in that run directory. Workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after collection, the executor reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. + +The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. + +Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs may execute concurrently when the selected harness and the user's token budget allow it, but every run still gets a distinct context and no shared mutable workspace. + +`RUN-THIS.prompt.md` requires a harness that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. Partial packages still grade and report what exists; missing arms remain visibly missing. + +An `output` is the model's own message in full, including questions, caveats, explanations, or a refusal. Where a run invoked a tool, that tool's stdout is evidence rather than a replacement for the response. Record the full worker transcript, duration, token usage, and tool-call count when the harness exposes them; omit unavailable metrics rather than estimating them. + +### Same model on both sides + +A fresh context is fresh of memory as well as of transcript. A harness with persistent memory, saved project instructions, or cross-session recall can carry into a nominally new session what it learned while running the previous one, which makes that session a continuation wearing a new name. Runs made under such a harness need that memory disabled, or a profile without it. + +An evaluated context sees its `prompt.md` and the files staged in its run directory, and nothing else - not `RUN-THIS.prompt.md`, not `run.json` from the paired run, not the assertions, not another case's output, not a note that an experiment is underway. Anything added on top is a second variable in a comparison meant to differ in exactly one. + +A meaningful A/B result requires both configurations to run on the same model, the same version, and the same configuration, varying only whether the skill is present. Running the with-skill case on one model and the baseline on another measures the model and the skill together; that is not a skill-effectiveness benchmark and must not be reported as one. When models are deliberately mixed, say so and treat the comparison as directional only. + +### Result handoff + +An externally produced result comes back identified by eval id, configuration (`with_skill` or `without_skill`), model and provider, and the produced output. It may also carry the transcript, duration, total tokens, tool-call count, output files, and notes. The user can hand it over as filled-in `results/*.result.json` files, or state it in chat and let the agent fill them in. + +Which artifact transfer happens depends on where the harness ran, and `RUN-THIS.prompt.md` tells it to close either way. A harness sharing a disk with the package writes the result files, grading, `benchmark.json`, `benchmark.md`, the first-party `report.html`, and the exact upstream `skill-creator-report.html` itself and reports the first-party report path. A harness that does not - a different product, a browser, or a sandbox - ends with one paste-ready block carrying the package path and every completed result object, including grading, plus the reports as file artifacts when supported. A repository session can use `-CollectResults` only as a fallback for transferred results that lack the report artifacts. "Bring the results back" means those artifacts, never a prose recap of how the runs went. + +Validate and compare a collected iteration with: + +``` +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults +``` + +It checks that each result matches its eval and configuration, warns when an arm is missing, unrun, or ran on a different model, and writes `comparison.md`, the paired `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`. The external evaluator may grade in its user-directed phase-two context; repository automation remains deterministic and never invokes a model. Use deterministic checks for mechanical assertions and evidence-backed human or evaluator judgement only where the assertion is genuinely qualitative. + +### Workspace isolation + +Eval packages obey **Eval Isolation**: they default to `.bot/-workspace/`, which git ignores, and `-OutputRoot` may only point there or outside the repository. Do not commit a package, its prompts, or its results unless the user explicitly asks for checked-in artifacts. + ## Per-Skill Evals - -Every repo-managed skill must include its own `evals/evals.json` file at `skills//evals/evals.json`. - + +Every repo-managed skill must include its own `evals/evals.json` file at `skills//evals/evals.json`. + - Treat this as a required artifact for every first-party skill in this repo - Eval entries may include an optional `files` array of skill-relative fixture paths such as `evals/files/example.md` - When `files` is present, keep the paths relative to `skills//` and validate that every fixture exists @@ -46,21 +182,22 @@ Every repo-managed skill must include its own `evals/evals.json` file at `skills - Run only the changed skill's deterministic validator and focused regression scripts during iteration; independent read-only checks may use bounded local parallelism, while shared-file mutations stay sequential - Run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` once before completion for the repository gate - Follow the top-level **AI/LLM Evaluation Automation Prohibition** for every eval. No per-skill or third-party requirement overrides it. +- To compare a skill against a baseline, prepare a package with **Portable Eval Handoff** and hand `RUN-THIS.prompt.md` to the user; the repository agent never runs the prompts, while the user-directed external executor runs, grades, and reports the paired comparison - Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them - -If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. - -## Git Identity - -Never set or override `git user.name`, `git user.email`, or `alias.bot` in the **local** git config of this repository. Always use the global config. Local overrides silently shadow global settings and produce commits with the wrong author. - + +If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. + +## Git Identity + +Never set or override `git user.name`, `git user.email`, or `alias.bot` in the **local** git config of this repository. Always use the global config. Local overrides silently shadow global settings and produce commits with the wrong author. + ## Git Operations Safeguards Agents must never automatically commit code changes or push to remote repositories. Both actions require explicit user approval: - -- **Commits**: Always request confirmation from the user before staging and committing code. Present a clear summary of changes and wait for user approval before executing the commit. -- **Remote Operations**: Do not push, pull, fetch, or interact with `origin` or any remote repository without explicit user instruction. These operations modify repository history and can cause data loss if performed unexpectedly. - + +- **Commits**: Always request confirmation from the user before staging and committing code. Present a clear summary of changes and wait for user approval before executing the commit. +- **Remote Operations**: Do not push, pull, fetch, or interact with `origin` or any remote repository without explicit user instruction. These operations modify repository history and can cause data loss if performed unexpectedly. + **Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Always require the user to explicitly approve these operations. ### Commit Skill Routing @@ -70,336 +207,330 @@ When the user asks to commit or stage changes, write or review a commit message, Bare `yolo` or `auto` outside an explicit commit request does not invoke `git-visual-commits`. Likewise, those modifiers do not invoke `git-keep-a-changelog` unless the user explicitly requests a changelog or release-note output. Users can force deterministic CLI selection with `/git-visual-commits` when they do not want to rely on automatic skill selection. ## Skill Creation - -Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. - -`skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. - -## Third-Party Skills - -Never modify skills maintained by others (e.g. `skill-creator` by Anthropic). If a third-party skill needs repo-specific behavior, add the rule here in `AGENTS.md` — not in the skill file itself, and not in a companion overlay around the third-party skill. Upstream updates will overwrite local edits without warning. - -## Local Install Sync - -Repo-managed skills live in four places that must stay in sync: - -- `skills//` — source control (and source of truth for edits) -- `~/.claude/skills//` — local Claude install -- `~/.agents/skills//` — local global agent install -- `~/.gemini/antigravity-cli/skills//` — local Gemini Antigravity install - -Changes often start in `~/.claude/skills//`, then get mirrored to the repo and the other local installs: - -- **Claude local → repo** (persist changes to source control): - ```powershell - Copy-Item "$HOME/.claude/skills//" "skills//" -Force - ``` -- **Claude local → agent installs** (keep `~/.agents` and Gemini current): - ```powershell - Copy-Item "$HOME/.claude/skills//" "$HOME/.agents/skills//" -Force - Copy-Item "$HOME/.claude/skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force - ``` -- **Repo → local installs** (after pulling changes or cloning fresh): - ```powershell - Copy-Item "skills//" "$HOME/.claude/skills//" -Force - Copy-Item "skills//" "$HOME/.agents/skills//" -Force - Copy-Item "skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force - ``` - -If you edit the `~/.agents/skills//` copy first, mirror it back to the repo and to `~/.claude/skills//` and `~/.gemini/antigravity-cli/skills//` using the same pattern. - -When renaming a skill, update **all four** locations — the repo folder, the local Claude install folder, the local global agent install folder, and the local Gemini Antigravity install folder. The folder name and the `name:` field in the SKILL.md frontmatter must match. A mismatch causes the skill to disappear from tooling or show stale instructions. - -A sync mismatch means one side runs a stale version, which leads to confusing eval results and wasted iterations. - -After the source copy passes its deterministic tests, SHA-256 identity across the repo and all three local installs is sufficient installation verification. Do not rerun the same deterministic suites from a hash-identical installed copy; that duplicates time, compute, and token use without adding evidence. Run an installed-copy test only when install-path resolution, loader behavior, permissions, or an actual hash mismatch is the subject of the test. - -After changing any repo-managed skill, sync the touched files across the repo copy, `~/.claude/skills//`, `~/.agents/skills//`, and `~/.gemini/antigravity-cli/skills//` before considering the task done. - -## Skill Directory Structure - -Every skill follows this layout: - -``` -skills// -├── SKILL.md # Required — the skill definition (loaded by Claude) -├── FORMS.md # Optional — structured form fields for parameter collection -├── assets/ # Optional — file templates, fonts, icons used in output -│ └── / # Group by variant when a skill supports multiple (e.g. library/, app/) -├── scripts/ # Optional — executable code (Python, Bash, etc.) -├── references/ # Optional — detailed reference docs the agent consults during generation -└── evals/ # Required for repo-managed skills — per-skill eval prompts and expectations - └── files/ # Optional — input fixtures referenced by evals/evals.json files[] -``` - -- `SKILL.md` is the entry point — it contains the workflow, conventions, and step-by-step instructions -- `assets/` holds file templates, fonts, icons, and other static content used in output (the agent reads and substitutes placeholders) -- `references/` holds detailed specs that `SKILL.md` references but are too long to inline -- `evals/` holds the per-skill `evals.json` definitions used to verify that the skill still works after changes -- `evals/files/` holds optional skill-local fixture inputs referenced by `evals/evals.json` when a benchmark needs attached source material - -## Template Files Are Literal - -Asset files in `assets/` are **not** processed by a templating engine. They contain real file content with placeholder values (e.g. `{ProjectName}`, `{TargetFramework}`) that the agent must read, understand, and substitute during generation. Agents should never copy asset files blindly — always read the content and adapt it to the user's specific parameters. - -## Prefer Dynamic Defaults - -When a skill needs time-sensitive or environment-sensitive values, prefer computing them from a reliable source instead of hardcoding them into prompts, defaults, or examples. - -- Prefer repo state, git metadata, official APIs, or vendor-maintained machine-readable feeds over date-stamped literals -- Use hardcoded fallback examples only when a dynamic source is unavailable or would add unreasonable complexity -- When a dynamic default exists, describe both the source and the fallback behavior in `FORMS.md` / `SKILL.md` -- If a value changes over time (supported frameworks, current versions, generated paths, repo-derived names), assume hardcoding will drift and design for refreshable computation - -## Scaffold Invariants - -For repo-managed .NET scaffolding skills, preserve semantic versioning infrastructure unless you are replacing it end-to-end in the same change. - -- App and library scaffolds rely on `MinVer` for versioning from git tags -- Do not remove `MinVer`, its package version, or its MSBuild hooks from scaffold templates unless a complete replacement workflow is implemented and validated in the same change -- Preserve the user-facing solution/product name in `PascalCase` for generated solution filenames such as `.slnx`; do not silently lowercase the solution filename -- Only derive lowercase values for fields that explicitly require them, such as repo slugs, package feeds, or Docker/image-style identifiers - -## Commit Discipline - -When committing changes to this repo, group by technology and logical purpose — don't mix unrelated changes. For example: - -- Skill instruction changes (`SKILL.md`) get their own commit -- Template files (`.csproj`, `.yml`, `.cs`) get their own commit(s) -- Documentation updates (`README.md`, `CONTRIBUTING.md`) get their own commit - -## Markdown Formatting - -All markdown files in this repository must use natural paragraph flow. Do not artificially break paragraphs at fixed column widths or insert hard line breaks within sentences. Paragraphs should flow as complete thoughts, allowing line wrapping to be determined by the reader's viewport or rendering engine, not by arbitrary character limits. - -**Why:** Natural paragraphs are more readable, easier to edit, and render correctly across all devices and markdown renderers. Artificially clipped paragraphs create maintenance friction and look awkward in source control diffs. - -## README Sync - -After modifying any skill (`SKILL.md`, `FORMS.md`) or repo-level config (`AGENTS.md`), **always update `README.md` before considering the task done**. This is a mandatory gate — not a nice-to-have. The README's "Available Skills" table, install examples, and "Why" sections must reflect the current state of all skills. A new skill without a README entry is incomplete work. - -## Blocking Completion Gates - -When repository guidance, an active skill, or a conversation summary identifies required follow-up work as pending, critical, blocking, or equivalent, treat those items as the active completion checklist for the current task rather than as background context. Do not call `task_complete`, describe the task as complete, or claim verification succeeded until every blocking item has either run successfully or been reported with the exact command, exit code, and remaining blocker. - -Before any completion message, reread the skill instructions and the current conversation summary's pending-task or blocker sections. If either one names a required script, validator, or maintenance step, that step is a hard gate, not optional polish. - -For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. - -## User Input UX - -When a skill collects parameters from the user, define the form in a dedicated `FORMS.md` file (Level 3 resource) rather than inlining field definitions in `SKILL.md`. This separates form structure from workflow logic and gives agents a parseable format to present fields correctly. - -Native input widgets are a **host/runtime feature**, not a guaranteed model capability. Treat them as an enhancement, not a dependency. - -- Skills must remain fully usable whether the host renders native fields or not -- When native fields are unavailable, the agent must follow a deterministic plain-text fallback defined in `FORMS.md` instead of improvising the interaction -- The fallback path must preserve the same field order, defaults, recommended choices, and final confirmation flow as the native-field path -- Do not switch interaction styles mid-collection unless the host explicitly upgrades from plain text to native controls -- Favor consistency and low-friction UX over conversational variety during parameter collection - -`FORMS.md` defines each field with: -- **type** — `text`, `single-choice`, or `multi-choice` -- **prompt** — the question to ask -- **choices** — options for choice types -- **default** — pre-filled value (mark as Recommended) -- **required** — whether the field is mandatory - -Presentation rules (enforced in every `FORMS.md`): -- Ask one field at a time — never bundle multiple questions -- Use selectable choices for `single-choice` and `multi-choice` fields — not free text -- When a default exists, present it first and append "(Recommended)" -- For `text` fields with a computed default, offer the computed value as a selectable choice alongside free text -- After all fields are collected, present a summary and ask for confirmation - -This applies to all skills that collect user input, not just scaffolding skills. - -## Status Update Hygiene - -Interim progress updates should describe user-relevant progress, evidence, blockers, and next steps. Do not narrate runner internals, sandbox mechanics, approved command paths, or retry plumbing unless that detail affects user approval, reproducibility, validation, or the final outcome. - -- Say what changed in the task state, not how the host executed the command -- Mention tool/runtime failures only when they block progress, require approval, or change the planned validation -- Prefer concise phrasing such as "The first read attempt failed before returning file content; I'm retrying and will report only if that changes the result" - -## Anthropic Skill Authoring Reference - -Essential conventions from [The Complete Guide to Building Skills for Claude](https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf) (Anthropic, Jan 2026). All skills in this repo must follow these rules. - -### File Structure - -``` -skill-name/ -├── SKILL.md # Required — exact spelling, case-sensitive -├── scripts/ # Optional — executable code (Python, Bash, etc.) -├── references/ # Optional — documentation loaded as needed -└── assets/ # Optional — templates, fonts, icons used in output -``` - -- **No `README.md`** inside the skill folder — all documentation goes in `SKILL.md` or `references/` -- Folder name must be **kebab-case** (no spaces, no underscores, no capitals) -- Folder name must match the `name:` field in YAML frontmatter - -### Progressive Disclosure (Three Levels) - -| Level | When loaded | Token cost | Content | -|-------|------------|------------|---------| -| **Level 1: Metadata** | Always (at startup) | ~100 tokens | `name` and `description` from YAML frontmatter | -| **Level 2: Instructions** | When skill is triggered | Under 5k tokens | SKILL.md body — workflows, steps, guidance | -| **Level 3: Resources** | As needed | Effectively unlimited | Linked files: scripts, references, assets, FORMS.md | - -Keep SKILL.md under **500 lines / 5,000 words**. Move detailed content to `references/`. Keep references **one level deep** from SKILL.md — nested references cause partial reads. - -### YAML Frontmatter - -Required fields: - -```yaml ---- -name: kebab-case-name # max 64 chars, lowercase + numbers + hyphens only -description: > # max 1024 chars, must include WHAT + WHEN + triggers - What it does. Use when user asks to [specific phrases]. ---- -``` - -Optional fields: - -```yaml -license: MIT # for open-source skills -compatibility: > # max 500 chars — environment requirements - Requires network access and Python 3.10+ -metadata: # custom key-value pairs - author: Company Name - version: 1.0.0 - mcp-server: server-name -``` - -**Forbidden**: XML angle brackets (`< >`), names containing "claude" or "anthropic" (reserved). - -### Description Field — The Most Important Part - -Structure: `[What it does] + [When to use it] + [Key capabilities]` - -```yaml -# ✅ Good — specific, actionable, includes triggers -description: > - Manages Linear project workflows including sprint planning, - task creation, and status tracking. Use when user mentions - "sprint", "Linear tasks", "project planning", or asks to - "create tickets". - -# ❌ Bad — too vague, no triggers -description: Helps with projects. -``` - -- Include trigger phrases users would actually say -- Mention file types if relevant -- Add negative triggers to prevent over-triggering: `Do NOT use for simple data exploration` - -### Writing Instructions - -- Be **specific and actionable** — `Run scripts/validate.py --input {filename}` not `Validate the data` -- Include **error handling** — common errors, causes, and solutions -- Use **feedback loops** — run validator → fix errors → repeat -- Put **critical instructions at the top** — use `## Critical` or `## Important` headers -- For critical validations, **use scripts over language instructions** — code is deterministic -- Prefer **dynamic defaults over hardcoded values** when the source data is available from the repo, environment, or an official machine-readable feed - -### Skill Categories - -| Category | Purpose | Example | -|----------|---------|---------| -| **Document & Asset Creation** | Consistent, high-quality output (docs, code, designs) | `frontend-design`, `docx`, `xlsx` | -| **Workflow Automation** | Multi-step processes with validation gates | `skill-creator`, scaffolding skills | -| **MCP Enhancement** | Workflow guidance layered on top of MCP tool access | `sentry-code-review` | - -### Common Patterns - -1. **Sequential workflow** — explicit step ordering with dependencies and rollback -2. **Multi-MCP coordination** — phase separation, data passing between services -3. **Iterative refinement** — draft → validate → fix → repeat until quality threshold -4. **Context-aware selection** — decision trees for choosing the right tool/approach -5. **Domain-specific intelligence** — compliance checks, governance, audit trails - -### Testing Checklist - -Before shipping a skill, verify: - -- [ ] Triggers on obvious tasks -- [ ] Triggers on paraphrased requests -- [ ] Does **not** trigger on unrelated topics -- [ ] Functional tests pass (correct outputs, error handling, edge cases) -- [ ] Performance improves over baseline (fewer messages, fewer errors, fewer tokens) - -Debug triggering: ask Claude `"When would you use the [skill name] skill?"` — it will quote the description back. - -### Troubleshooting Quick Reference - -| Symptom | Likely cause | Fix | -|---------|-------------|-----| -| Skill won't upload | `SKILL.md` misspelled or YAML invalid | Exact case `SKILL.md`, check `---` delimiters | -| Skill never triggers | Description too vague | Add trigger phrases, mention file types | -| Skill triggers too often | Description too broad | Add negative triggers, narrow scope | -| Instructions not followed | Too verbose or ambiguous | Shorten, use bullets, move detail to `references/` | -| Slow / degraded responses | Too much content loaded | Keep SKILL.md under 5k words, use progressive disclosure | - -## Karpathy Rules - -Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed. - -### 1. Think Before Coding - -**Don't assume. Don't hide confusion. Surface tradeoffs.** - -Before implementing: -- State your assumptions explicitly. If uncertain, ask. -- If multiple interpretations exist, present them - don't pick silently. -- If a simpler approach exists, say so. Push back when warranted. -- If something is unclear, stop. Name what's confusing. Ask. -- When the root cause is uncertain, do not present hypotheses as facts. State the uncertainty explicitly and ask whether to investigate before applying a fix. - -### 2. Simplicity First - -**Minimum code that solves the problem. Nothing speculative.** - -- No features beyond what was asked. -- No abstractions for single-use code. -- No "flexibility" or "configurability" that wasn't requested. -- No error handling for impossible scenarios. -- If you write 200 lines and it could be 50, rewrite it. - -Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. - -### 3. Surgical Changes - -**Touch only what you must. Clean up only your own mess.** - -When editing existing code: -- Don't "improve" adjacent code, comments, or formatting. -- Don't refactor things that aren't broken. -- Match existing style, even if you'd do it differently. -- If you notice unrelated dead code, mention it - don't delete it. - -When your changes create orphans: -- Remove imports/variables/functions that YOUR changes made unused. -- Don't remove pre-existing dead code unless asked. - -The test: Every changed line should trace directly to the user's request. - -### 4. Goal-Driven Execution - -**Define success criteria. Loop until verified.** - -Transform tasks into verifiable goals: -- "Add validation" → "Write tests for invalid inputs, then make them pass" -- "Fix the bug" → "Write a test that reproduces it, then make it pass" -- "Refactor X" → "Ensure tests pass before and after" - -For multi-step tasks, state a brief plan: -``` -1. [Step] → verify: [check] -2. [Step] → verify: [check] -3. [Step] → verify: [check] -``` - -Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. + +Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. + +Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. + +`skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. + +## Third-Party Skills + +Never modify skills maintained by others (e.g. `skill-creator` by Anthropic). If a third-party skill needs repo-specific behavior, add the rule here in `AGENTS.md` — not in the skill file itself, and not in a companion overlay around the third-party skill. Upstream updates will overwrite local edits without warning. + +## Local Install Sync + +Repo-managed skills live in four places that must stay in sync: + +- `skills//` — source control (and source of truth for edits) +- `~/.claude/skills//` — local Claude install +- `~/.agents/skills//` — local global agent install +- `~/.gemini/antigravity-cli/skills//` — local Gemini Antigravity install + +Sync the **whole skill tree** with the repository as the source of truth, and prove it with hashes: + +``` +pwsh -NoProfile -File ./scripts/sync-skill-install.ps1 -Skill +``` + +The script copies every file under `skills//` into all three installs, then compares SHA-256 across all four locations and exits non-zero on any difference. `-VerifyOnly` checks without copying, `-Prune` deletes install files that no longer exist in the repository, and omitting `-Skill` sweeps every repo-managed skill. Generated build output (`bin/`, `obj/`) is excluded because it is regenerated per location and never matches; a file that exists only in an install is drift too, because a rename or deletion otherwise leaves the old one loading forever. + +**Run it as the last action before the completion message, after the final edit is in place.** Do not copy a remembered list of touched files: that list goes stale the moment you edit one more file, and a sync performed earlier in the session says nothing about what changed after it. Never report "synced" or "hash-identical" from memory, from an earlier turn, or from a partial per-file copy — the claim must be backed by this command's output in the same response that makes it. This is a [blocking completion gate](#blocking-completion-gates). + +If a change starts in `~/.claude/skills//` or another install, mirror the edited file back into `skills//` first, then run the script so the repository stays authoritative. + +When renaming a skill, update **all four** locations — the repo folder, the local Claude install folder, the local global agent install folder, and the local Gemini Antigravity install folder. The folder name and the `name:` field in the SKILL.md frontmatter must match. A mismatch causes the skill to disappear from tooling or show stale instructions. + +A sync mismatch means one side runs a stale version, which leads to confusing eval results and wasted iterations. + +After the source copy passes its deterministic tests, SHA-256 identity across the repo and all three local installs is sufficient installation verification. Do not rerun the same deterministic suites from a hash-identical installed copy; that duplicates time, compute, and token use without adding evidence. Run an installed-copy test only when install-path resolution, loader behavior, permissions, or an actual hash mismatch is the subject of the test. + +## Skill Directory Structure + +Every skill follows this layout: + +``` +skills// +├── SKILL.md # Required — the skill definition (loaded by Claude) +├── FORMS.md # Optional — structured form fields for parameter collection +├── assets/ # Optional — file templates, fonts, icons used in output +│ └── / # Group by variant when a skill supports multiple (e.g. library/, app/) +├── scripts/ # Optional — executable code (Python, Bash, etc.) +├── references/ # Optional — detailed reference docs the agent consults during generation +└── evals/ # Required for repo-managed skills — per-skill eval prompts and expectations + └── files/ # Optional — input fixtures referenced by evals/evals.json files[] +``` + +- `SKILL.md` is the entry point — it contains the workflow, conventions, and step-by-step instructions +- `assets/` holds file templates, fonts, icons, and other static content used in output (the agent reads and substitutes placeholders) +- `references/` holds detailed specs that `SKILL.md` references but are too long to inline +- `evals/` holds the per-skill `evals.json` definitions used to verify that the skill still works after changes +- `evals/files/` holds optional skill-local fixture inputs referenced by `evals/evals.json` when a benchmark needs attached source material + +## Template Files Are Literal + +Asset files in `assets/` are **not** processed by a templating engine. They contain real file content with placeholder values (e.g. `{ProjectName}`, `{TargetFramework}`) that the agent must read, understand, and substitute during generation. Agents should never copy asset files blindly — always read the content and adapt it to the user's specific parameters. + +## Prefer Dynamic Defaults + +When a skill needs time-sensitive or environment-sensitive values, prefer computing them from a reliable source instead of hardcoding them into prompts, defaults, or examples. + +- Prefer repo state, git metadata, official APIs, or vendor-maintained machine-readable feeds over date-stamped literals +- Use hardcoded fallback examples only when a dynamic source is unavailable or would add unreasonable complexity +- When a dynamic default exists, describe both the source and the fallback behavior in `FORMS.md` / `SKILL.md` +- If a value changes over time (supported frameworks, current versions, generated paths, repo-derived names), assume hardcoding will drift and design for refreshable computation + +## Scaffold Invariants + +For repo-managed .NET scaffolding skills, preserve semantic versioning infrastructure unless you are replacing it end-to-end in the same change. + +- App and library scaffolds rely on `MinVer` for versioning from git tags +- Do not remove `MinVer`, its package version, or its MSBuild hooks from scaffold templates unless a complete replacement workflow is implemented and validated in the same change +- Preserve the user-facing solution/product name in `PascalCase` for generated solution filenames such as `.slnx`; do not silently lowercase the solution filename +- Only derive lowercase values for fields that explicitly require them, such as repo slugs, package feeds, or Docker/image-style identifiers + +## Commit Discipline + +When committing changes to this repo, group by technology and logical purpose — don't mix unrelated changes. For example: + +- Skill instruction changes (`SKILL.md`) get their own commit +- Template files (`.csproj`, `.yml`, `.cs`) get their own commit(s) +- Documentation updates (`README.md`, `CONTRIBUTING.md`) get their own commit + +## Markdown Formatting + +All markdown files in this repository must use natural paragraph flow. Do not artificially break paragraphs at fixed column widths or insert hard line breaks within sentences. Paragraphs should flow as complete thoughts, allowing line wrapping to be determined by the reader's viewport or rendering engine, not by arbitrary character limits. + +**Why:** Natural paragraphs are more readable, easier to edit, and render correctly across all devices and markdown renderers. Artificially clipped paragraphs create maintenance friction and look awkward in source control diffs. + +## README Sync + +After modifying any skill (`SKILL.md`, `FORMS.md`) or repo-level config (`AGENTS.md`), **always update `README.md` before considering the task done**. This is a mandatory gate — not a nice-to-have. The README's "Available Skills" table, install examples, and "Why" sections must reflect the current state of all skills. A new skill without a README entry is incomplete work. + +## Blocking Completion Gates + +When repository guidance, an active skill, or a conversation summary identifies required follow-up work as pending, critical, blocking, or equivalent, treat those items as the active completion checklist for the current task rather than as background context. Do not call `task_complete`, describe the task as complete, or claim verification succeeded until every blocking item has either run successfully or been reported with the exact command, exit code, and remaining blocker. + +Before any completion message, reread the skill instructions and the current conversation summary's pending-task or blocker sections. If either one names a required script, validator, or maintenance step, that step is a hard gate, not optional polish. + +For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. + +Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). + +## User Input UX + +When a skill collects parameters from the user, define the form in a dedicated `FORMS.md` file (Level 3 resource) rather than inlining field definitions in `SKILL.md`. This separates form structure from workflow logic and gives agents a parseable format to present fields correctly. + +Native input widgets are a **host/runtime feature**, not a guaranteed model capability. Treat them as an enhancement, not a dependency. + +- Skills must remain fully usable whether the host renders native fields or not +- When native fields are unavailable, the agent must follow a deterministic plain-text fallback defined in `FORMS.md` instead of improvising the interaction +- The fallback path must preserve the same field order, defaults, recommended choices, and final confirmation flow as the native-field path +- Do not switch interaction styles mid-collection unless the host explicitly upgrades from plain text to native controls +- Favor consistency and low-friction UX over conversational variety during parameter collection + +`FORMS.md` defines each field with: +- **type** — `text`, `single-choice`, or `multi-choice` +- **prompt** — the question to ask +- **choices** — options for choice types +- **default** — pre-filled value (mark as Recommended) +- **required** — whether the field is mandatory + +Presentation rules (enforced in every `FORMS.md`): +- Ask one field at a time — never bundle multiple questions +- Use selectable choices for `single-choice` and `multi-choice` fields — not free text +- When a default exists, present it first and append "(Recommended)" +- For `text` fields with a computed default, offer the computed value as a selectable choice alongside free text +- After all fields are collected, present a summary and ask for confirmation + +This applies to all skills that collect user input, not just scaffolding skills. + +## Status Update Hygiene + +Interim progress updates should describe user-relevant progress, evidence, blockers, and next steps. Do not narrate runner internals, sandbox mechanics, approved command paths, or retry plumbing unless that detail affects user approval, reproducibility, validation, or the final outcome. + +- Say what changed in the task state, not how the host executed the command +- Mention tool/runtime failures only when they block progress, require approval, or change the planned validation +- Prefer concise phrasing such as "The first read attempt failed before returning file content; I'm retrying and will report only if that changes the result" + +## Anthropic Skill Authoring Reference + +Essential conventions from [The Complete Guide to Building Skills for Claude](https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf) (Anthropic, Jan 2026). All skills in this repo must follow these rules. + +### File Structure + +``` +skill-name/ +├── SKILL.md # Required — exact spelling, case-sensitive +├── scripts/ # Optional — executable code (Python, Bash, etc.) +├── references/ # Optional — documentation loaded as needed +└── assets/ # Optional — templates, fonts, icons used in output +``` + +- **No `README.md`** inside the skill folder — all documentation goes in `SKILL.md` or `references/` +- Folder name must be **kebab-case** (no spaces, no underscores, no capitals) +- Folder name must match the `name:` field in YAML frontmatter + +### Progressive Disclosure (Three Levels) + +| Level | When loaded | Token cost | Content | +|-------|------------|------------|---------| +| **Level 1: Metadata** | Always (at startup) | ~100 tokens | `name` and `description` from YAML frontmatter | +| **Level 2: Instructions** | When skill is triggered | Under 5k tokens | SKILL.md body — workflows, steps, guidance | +| **Level 3: Resources** | As needed | Effectively unlimited | Linked files: scripts, references, assets, FORMS.md | + +Keep SKILL.md under **500 lines / 5,000 words**. Move detailed content to `references/`. Keep references **one level deep** from SKILL.md — nested references cause partial reads. + +### YAML Frontmatter + +Required fields: + +```yaml +--- +name: kebab-case-name # max 64 chars, lowercase + numbers + hyphens only +description: > # max 1024 chars, must include WHAT + WHEN + triggers + What it does. Use when user asks to [specific phrases]. +--- +``` + +Optional fields: + +```yaml +license: MIT # for open-source skills +compatibility: > # max 500 chars — environment requirements + Requires network access and Python 3.10+ +metadata: # custom key-value pairs + author: Company Name + version: 1.0.0 + mcp-server: server-name +``` + +**Forbidden**: XML angle brackets (`< >`), names containing "claude" or "anthropic" (reserved). + +### Description Field — The Most Important Part + +Structure: `[What it does] + [When to use it] + [Key capabilities]` + +```yaml +# ✅ Good — specific, actionable, includes triggers +description: > + Manages Linear project workflows including sprint planning, + task creation, and status tracking. Use when user mentions + "sprint", "Linear tasks", "project planning", or asks to + "create tickets". + +# ❌ Bad — too vague, no triggers +description: Helps with projects. +``` + +- Include trigger phrases users would actually say +- Mention file types if relevant +- Add negative triggers to prevent over-triggering: `Do NOT use for simple data exploration` + +### Writing Instructions + +- Be **specific and actionable** — `Run scripts/validate.py --input {filename}` not `Validate the data` +- Include **error handling** — common errors, causes, and solutions +- Use **feedback loops** — run validator → fix errors → repeat +- Put **critical instructions at the top** — use `## Critical` or `## Important` headers +- For critical validations, **use scripts over language instructions** — code is deterministic +- Prefer **dynamic defaults over hardcoded values** when the source data is available from the repo, environment, or an official machine-readable feed + +### Skill Categories + +| Category | Purpose | Example | +|----------|---------|---------| +| **Document & Asset Creation** | Consistent, high-quality output (docs, code, designs) | `frontend-design`, `docx`, `xlsx` | +| **Workflow Automation** | Multi-step processes with validation gates | `skill-creator`, scaffolding skills | +| **MCP Enhancement** | Workflow guidance layered on top of MCP tool access | `sentry-code-review` | + +### Common Patterns + +1. **Sequential workflow** — explicit step ordering with dependencies and rollback +2. **Multi-MCP coordination** — phase separation, data passing between services +3. **Iterative refinement** — draft → validate → fix → repeat until quality threshold +4. **Context-aware selection** — decision trees for choosing the right tool/approach +5. **Domain-specific intelligence** — compliance checks, governance, audit trails + +### Testing Checklist + +Before shipping a skill, verify: + +- [ ] Triggers on obvious tasks +- [ ] Triggers on paraphrased requests +- [ ] Does **not** trigger on unrelated topics +- [ ] Functional tests pass (correct outputs, error handling, edge cases) +- [ ] Performance improves over baseline (fewer messages, fewer errors, fewer tokens) + +Debug triggering: ask Claude `"When would you use the [skill name] skill?"` — it will quote the description back. + +### Troubleshooting Quick Reference + +| Symptom | Likely cause | Fix | +|---------|-------------|-----| +| Skill won't upload | `SKILL.md` misspelled or YAML invalid | Exact case `SKILL.md`, check `---` delimiters | +| Skill never triggers | Description too vague | Add trigger phrases, mention file types | +| Skill triggers too often | Description too broad | Add negative triggers, narrow scope | +| Instructions not followed | Too verbose or ambiguous | Shorten, use bullets, move detail to `references/` | +| Slow / degraded responses | Too much content loaded | Keep SKILL.md under 5k words, use progressive disclosure | + +## Karpathy Rules + +Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed. + +### 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before implementing: +- State your assumptions explicitly. If uncertain, ask. +- If multiple interpretations exist, present them - don't pick silently. +- If a simpler approach exists, say so. Push back when warranted. +- If something is unclear, stop. Name what's confusing. Ask. +- When the root cause is uncertain, do not present hypotheses as facts. State the uncertainty explicitly and ask whether to investigate before applying a fix. + +### 2. Simplicity First + +**Minimum code that solves the problem. Nothing speculative.** + +- No features beyond what was asked. +- No abstractions for single-use code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, rewrite it. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. + +### 3. Surgical Changes + +**Touch only what you must. Clean up only your own mess.** + +When editing existing code: +- Don't "improve" adjacent code, comments, or formatting. +- Don't refactor things that aren't broken. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code, mention it - don't delete it. + +When your changes create orphans: +- Remove imports/variables/functions that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +The test: Every changed line should trace directly to the user's request. + +### 4. Goal-Driven Execution + +**Define success criteria. Loop until verified.** + +Transform tasks into verifiable goals: +- "Add validation" → "Write tests for invalid inputs, then make them pass" +- "Fix the bug" → "Write a test that reproduces it, then make it pass" +- "Refactor X" → "Ensure tests pass before and after" + +For multi-step tasks, state a brief plan: +``` +1. [Step] → verify: [check] +2. [Step] → verify: [check] +3. [Step] → verify: [check] +``` + +Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. diff --git a/CHANGELOG.md b/CHANGELOG.md index f0c5931..bb96609 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,21 +4,25 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.9.0] - 2026-08-14 +## [0.9.0] - 2026-08-20 -This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets` — and replaces the repository's model-backed eval benchmark workflow with a deterministic, local-only validation path. `dotnet-test` bootstraps and modernizes xUnit test projects against Codebelt conventions with role-aware fixtures; `dotnet-remote-testing` runs .NET tests inside official Microsoft SDK containers using either an existing `testenvironments.json` or zero-config, offline-safe release discovery; `dotnet-segregated-assets` migrates ASP.NET Core applications to an artifact-first topology where `wwwroot` stays the authoring root while deployed static content is served by a separate hardened origin. Alongside those, `git-keep-a-changelog` and `git-nuget-release-notes` gained deterministic release-entity classification so a capability introduced and then refined before its first release stays a single `Added` outcome, and `git-visual-commits` gained an invocation routing lock so an explicit commit request can no longer be diverted into a changelog or release-note skill. No published skill was removed or renamed, so adopting this release is non-breaking for existing installs. +This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets` — replaces the repository's model-backed eval benchmark workflow with deterministic, local-only validation, and finalizes the portable eval handoff. The selected external evaluator now runs the paired workers, grades their results, and invokes Anthropic's skill-creator aggregator and eval viewer without sending the user back for a second collection command. `dotnet-test` bootstraps and modernizes xUnit test projects against Codebelt conventions with role-aware fixtures; `dotnet-remote-testing` runs .NET tests inside official Microsoft SDK containers using either an existing `testenvironments.json` or zero-config, offline-safe release discovery; and `dotnet-segregated-assets` migrates ASP.NET Core applications to an artifact-first topology where `wwwroot` stays the authoring root while deployed static content is served by a separate hardened origin. Alongside those, `git-keep-a-changelog` and `git-nuget-release-notes` gained deterministic release-entity classification, and `git-visual-commits` gained an invocation routing lock. No published skill was removed or renamed, so adopting this release is non-breaking for existing installs. > [!NOTE] -> Contributor workflow changed. Repository scripts, CI jobs, skill runners, graders, optimizers, and executor hooks must never invoke an authenticated AI/LLM CLI or API. The previously mandatory paired `with_skill` / `without_skill` model-backed benchmark is no longer a completion gate; deterministic local validators and human inspection of the eval specifications take its place. +> Contributor workflow changed. Repository scripts, CI jobs, skill runners, graders, optimizers, and executor hooks must never invoke an authenticated AI/LLM CLI or API. The previously mandatory paired `with_skill` / `without_skill` model-backed benchmark is no longer a completion gate; deterministic local validators and human inspection of the eval specifications take its place. The paired comparison remains available as prepared prompts you execute yourself: `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ` writes both arms to a gitignored `.bot/` workspace, and `-CollectResults` brings the externally produced results back for deterministic and human grading. ### Added - `dotnet-test` skill that bootstraps and refactors xUnit test projects to Codebelt conventions, classifying each selected project as an ordinary unit test, an ASP.NET Core functional test, or a console/worker functional test, then applying the matching focused or shared fixture pattern while preserving test names, lifecycle behavior, package ownership, and target frameworks, -- `dotnet-test` bundled tooling: `inspect-dotnet-tests.ps1` emits machine-readable role, framework, xUnit-generation, inheritance, migration, and blocker evidence before any mutation, and `resolve-test-package-versions.ps1` resolves stable NuGet candidates and proves the combined package set restores across the selected target frameworks, each covered by its own PowerShell regression harness, +- `dotnet-test` bundled tooling: `inspect-dotnet-tests.ps1` emits machine-readable role, framework, xUnit-generation, inheritance, migration, and blocker evidence before any mutation, and `resolve-test-package-versions.ps1` resolves stable NuGet candidates and proves the combined package set restores across the selected target frameworks while anchoring every `xunit*` package to the Codebelt xUnit release — an id that release declares resolves 1:1 to the declared version and every other `xunit*` id stays at or below its major, so a new xUnit generation on NuGet cannot outrun the Codebelt API the skill targets — each covered by its own PowerShell regression harness, +- `dotnet-test` migration gate `verify-dotnet-test-migration.ps1`, which closes a migration with a `PASSED`/`FAILED` verdict rather than a self-assessment: it reruns the inspector under the expected focused or shared postcondition and adds the checks that only exist once the edits do — a type still deriving from `WebApplicationFactory` behind a wrapper or rename, a retained `Microsoft.AspNetCore.Mvc.Testing` reference, `xunit*` pins past the major the restored Codebelt package declares in `project.assets.json`, and files changed under the selected project while the target pattern appears nowhere in it — each violation reported with its file and line, and covered by a regression harness that includes a completed-migration positive control, - `dotnet-test` assets and reference documentation covering unit-test behavior patterns, focused and shared web-application fixtures, application-focused fixtures, bootstrapper hosts for console and worker services in both minimal and Program/Startup form, xUnit v2-to-v3 modernization, and migration-invariant preservation, - `dotnet-remote-testing` skill that runs .NET tests inside Docker using official `mcr.microsoft.com/dotnet/sdk` images, honoring an existing `testenvironments.json` as authoritative when present and otherwise deriving environments from Microsoft's live release index, while reporting WSL and SSH as unsupported instead of silently falling back to the host, - `dotnet-remote-testing` deterministic runner `remote-test.cs` owning configuration discovery, release parsing, digest-pinned image resolution, isolated source staging, NuGet caching, execution, result parsing, distinct failure classification, and cleanup behind a single entry point, with a built-in `--self-test` alongside a PowerShell harness, - Offline-safe release discovery for `dotnet-remote-testing`: successful release metadata is cached outside the repository so later runs work without network access, and the parameter form surfaces the exact runner-computed target as the recommended option, +- Build-tooling preparation in `dotnet-remote-testing`, probing the resolved image for the `git` that MinVer, Nerdbank.GitVersioning, GitInfo, and SourceLink invoke during `dotnet build`, and layering it on through a digest-addressed image cached outside the repository when the image lacks it, so a minimal runner image no longer fails a sound build with `MINVER1007` while the reported image and digest stay the resolved base, +- Git metadata in the staged workspace for `dotnet-remote-testing`, copying the repository's `.git` directory into the disposable staged copy and resolving a linked worktree's `gitdir:` pointer to the real directory, so version stamping, SourceLink, and any repository-root probe that walks up to a `.git` directory behave as they do on the host instead of silently resolving elsewhere and changing what the tests observe, with `--no-git-metadata` as an explicit, reported opt-out for a repository whose history dominates staging cost, +- `dotnet test`-shaped result reporting in `dotnet-remote-testing`, breaking results down per test assembly and target framework and reporting each failure with its fully-qualified name, target framework, elapsed time, assertion message, stack trace, and test-written output, reporting an infrastructure failure with its own phase's log rather than a tail of the whole run, and adding `--show-log` for the complete container log, - `dotnet-segregated-assets` skill that migrates an ASP.NET Core application to serve deployed static content from Codebelt Static Content Provider (`codebeltnet/web-cdn-origin:2.0.0`) while `wwwroot` remains the authoring root, separating app-owned assets from shared CDN assets and preserving Razor Class Library, framework, and generated Static Web Assets, - `dotnet-segregated-assets` deterministic runner `segregate-assets.cs` that inspects static-asset topology, classifies existing segregation state, escalates Blazor, Razor Class Library, scoped-CSS, and frontend-build risk instead of blindly excluding it, resolves Cuemon TagHelper package versions from the NuGet V3 service index at plan time, reports cache-busting interfaces and registrations without rewriting Razor or C# source, and proves the publish invariant through `verify --run-publish` against an isolated temp directory, - Artifact-first container contract for `dotnet-segregated-assets` in which both application Dockerfiles package an already-published `artifacts/publish/` directory rather than compiling source, with the validator rejecting an SDK stage, a `dotnet build` or `dotnet publish` step, an `mcr.microsoft.com` runtime, or a missing artifact copy, @@ -28,6 +32,14 @@ This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-r - `resolve-release-entity.ps1` and its regression harness for `git-keep-a-changelog`, classifying a release entity as `Added`, `Removed`, `Changed`, or `Unchanged` from its existence at the resolved base and at HEAD rather than from intermediate commit verbs, - AI/LLM Evaluation Automation Prohibition as a Priority 1 rule in `AGENTS.md`, forbidding repository scripts, CI jobs, skill runners, graders, optimizers, and executor hooks from invoking an authenticated AI/LLM CLI or API, and declaring that it wins over any conflicting rule, skill, test, or completion gate, - Invocation Routing Lock in `git-visual-commits`, making `git bot commit`, `git commit`, and `git our commit` authoritative selections of that skill so `yolo` is never read as the commit message nor routed to a changelog, release-note, or squash-summary skill, +- `scripts/prepare-skill-evals.ps1`, which turns a skill's `evals/evals.json` into a portable evaluation package under gitignored `.bot/` storage by default: one directory per eval holding a `with-skill.prompt.md` with the effective skill instructions inlined, a `without-skill.prompt.md` carrying the identical task, input files, and response contract with no skill and no mention of which skill is under test, an `eval-metadata.json` with the expected output, assertions, fixtures, and reproduction assumptions, the fixtures themselves, and prefilled result stubs, with `-CollectResults` validating returned results, flagging a missing arm or two arms run on different models, and writing `comparison.md`, +- `-Changed` mode in `scripts/prepare-skill-evals.ps1`, resolving every repo-managed skill a branch touched, uncommitted and untracked work included, and preparing a package for each, so adding or modifying a skill triggers evaluation instead of relying on someone remembering to ask, +- Eval preparation as a blocking completion gate in `AGENTS.md`, ordered after the last skill edit and before `scripts/sync-skill-install.ps1`, satisfied by preparing the packages and reporting the prompt paths and never by executing one, together with the trigger phrases (`eval `, `evaluate `, `prepare evals for `) that route a request straight to the script instead of to a plan or a menu, +- `RUN-THIS.prompt.md` in every prepared package, making the user-selected agent an eval orchestrator that creates one isolated worker per run, exposes only the matching prompt and inputs, never reuses a worker, and records the complete response plus any transcript, duration, token, and tool-call data the harness provides, +- Blind worker prompts that omit eval headers, configuration labels, grading criteria, runner instructions, and sibling results, while keeping the task, inputs, response contract, model, tools, and limits identical across `with_skill` and `without_skill`, +- An explicit executor role in `AGENTS.md`, separating the repository agent that prepares a package from the harness a person selects to execute that package, while keeping the Priority 1 prohibition in force for repository scripts, jobs, hooks, gates, and unrequested agent fan-out, +- `.bot/-workspace/` as the default eval package location, covered by the existing `.bot/*` ignore rule, so harnesses that refuse to work outside the repository folder have a home that git never sees, while `-OutputRoot` still refuses any other in-repository path and refuses `.bot/` itself if git stops ignoring it, +- Portable Eval Handoff in `AGENTS.md`, which keeps Anthropic `skill-creator`'s controlled with-skill versus baseline methodology and replaces only its execution transport: the repository prepares prompts and stops, the user picks the harness, provider, and model, and the agent never executes a generated prompt, never spawns candidate or baseline subagents, never calls a model API, and never treats model-backed execution as a completion gate, - `-MetadataOnly` mode in `scripts/validate-skill-templates.ps1` for a sub-second repository-wide manifest, eval-fixture, and frontmatter check, together with a validation summary that reports the target ref, the mode, and per-check pass/fail detail. ### Changed @@ -38,11 +50,20 @@ This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-r - `git-visual-commits` description rewritten around authoritative command routing, with auto-approval scoped to `yolo` and `auto` appearing inside an explicit commit request, - Repository eval guidance in `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` reframed around deterministic local validation, treating each `evals/evals.json` as a versioned review specification and describing a layered path from `-MetadataOnly` through the changed skill's own validator to the full repository gate, - `scripts/validate-skill-templates.ps1` extended with deterministic skill-content validation, release-entity classifier enforcement, `git-keep-a-changelog` trigger validation, resolver-script presence checks, GitHub Actions opinionation checks that reject multi-vendor CI references, and worktree-aware local shell policy scanning, -- `README.md` install snippets, skill catalog, and capability sections updated for `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets`. +- `README.md` install snippets, skill catalog, and capability sections updated for `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets`, +- `CONTRIBUTING.md` eval instructions replaced the removed `run-skill-benchmark.ps1` flow with the prepare-and-collect workflow, and `README.md` now documents that the generated prompts are self-contained enough to paste into any capable agent environment, +- `scripts/validate-skill-templates.ps1` extended with a deterministic check that a prepared eval package keeps its controlled-experiment shape: both configurations share one task, one input set, and one response contract, the baseline never names the skill under test, neither prompt carries the expected output or the assertions, each result stub declares its own configuration with one grading entry per assertion, and an output root inside the repository is refused before it is created, +- Eval collection now reports available duration, token, tool-call, and transcript status per run instead of accepting those fields without surfacing them in `comparison.md`, +- `RUN-THIS.prompt.md` now explicitly starts the evaluator immediately, keeps workers blind to the grading key, grades after collection, and reports the completed comparison in the same handoff, +- prepared packages now carry a thin `tools/generate-eval-report.ps1` adapter plus the exact Anthropic skill-creator grader, aggregator, and eval-viewer assets; the adapter writes the upstream `report.html`, `benchmark.json`, and `benchmark.md` artifacts from the portable result files. ### Removed -- The mandatory paired `with_skill` / `without_skill` model-backed benchmark completion gate for repo-managed skill work, along with the eval-viewer review artifacts it required, superseded by deterministic local validators and human inspection of the eval specifications. +- The mandatory paired `with_skill` / `without_skill` model-backed benchmark completion gate for repo-managed skill work, along with the eval-viewer review artifacts it required, superseded by deterministic local validators and human inspection of the eval specifications, + +### Fixed + +- Eval documentation and repository guidance no longer describe `-CollectResults` as the normal post-run step or leave grading and HTML review as an unexplained follow-up; the deprecated compatibility shim's historical references remain intact for backward compatibility. ## [0.8.2] - 2026-08-07 @@ -591,7 +612,6 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. -[Unreleased]: https://github.com/codebeltnet/agentic/compare/v0.8.2...HEAD [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 [0.8.1]: https://github.com/codebeltnet/agentic/compare/v0.8.0...v0.8.1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3f985ad..901ed93 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,153 +1,156 @@ -# Contributing - -Thanks for wanting to add or improve a skill. Here's what to know. - -## Skill structure - -Each skill lives in its own folder: - -``` -skills/ - / - SKILL.md # Required — the skill content - FORMS.md # Optional — structured input collection for agents - assets/ # Optional — literal templates and static files - scripts/ # Optional — executable helpers - references/ # Optional — deeper docs loaded on demand - evals/ # Required for repo-managed skills — test prompts for validation - evals.json -``` - -## Local sync - -Repo-managed skills should be mirrored across all three locations: - -- `skills//` in this repo -- `~/.claude/skills//` -- `~/.agents/skills//` - -If you edit a local install copy first, copy the changed files back into the repo and into the other local install so every agent sees the same skill version. - -## SKILL.md format - -Every `SKILL.md` must start with a YAML front matter block: - -```yaml ---- -name: your-skill-name -description: > - One or two sentences describing what this skill does and when the AI - should automatically invoke it. Be specific about trigger phrases and - use cases — this description is what the AI reads to decide whether - to load the skill. ---- -``` - -The rest of the file is free-form Markdown. Include: - -- **When to use** — what scenarios or requests trigger this skill -- **Rules / conventions** — the core content the AI should follow -- **Examples** — good and bad, so the AI can calibrate -- **Prerequisites** — anything the human needs to set up first (tools, config, etc.) - -## Naming - -- Skill folder and `name` field: `kebab-case` -- Be specific — `git-bot-commits` is better than `git` or `commits` -- Avoid version numbers in names; use the description to note maturity - -## Writing good descriptions (the front matter field) - -The `description` is the most important field — it's how the AI decides to load the skill. Include: - -- What the skill enables -- Specific trigger phrases (e.g. "Use when user says 'commit this' or 'stage changes'") -- What it enforces or prevents - -## Adding evals (required for repo-managed skills) - -Evals let you verify the skill works and measure improvement over a baseline. Every repo-managed skill in this repository must include `evals/evals.json`: - -```json -{ - "skill_name": "your-skill-name", - "evals": [ - { - "id": 0, - "prompt": "The user message to test against", - "expected_output": "What a correct response looks like — used for manual or automated grading", - "files": ["evals/files/example.md"] - } - ] -} -``` - -`files` is optional. When present, list one or more fixture files relative to `skills//`. A common pattern is to store those fixtures under `evals/files/` so benchmark runners can copy or attach the same source inputs for both `with_skill` and `without_skill` runs. - -Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and cases where the skill should *not* do something. - -Run evals from a temp workspace, not from this repository: - -```powershell -$workspace = Join-Path $env:TEMP '-workspace' -``` - -When creating or modifying a repo-managed skill, the eval workflow must include a paired comparison: - -- Resolve the installed Anthropic `skill-creator` path first, usually under `~/.agents/skills/skill-creator/` or `~/.claude/skills/skill-creator/`, then run its benchmark scripts from there -- Run each eval as `with_skill` -- Run the baseline as `without_skill` for new skills -- For an existing skill, use either `without_skill` or the previous/original skill version as the baseline, following the `skill-creator` benchmark model -- Aggregate the results into `benchmark.json` -- Launch `eval-viewer/generate_review.py` from that installed `skill-creator` copy so a human can review both `Outputs` and `Benchmark` - -The preferred local entry point is the repo-owned runner: - -```powershell -pwsh -NoProfile -File .\scripts\run-skill-benchmark.ps1 -SkillPath .\skills\ -CompareWithLegacy -``` - -That runner keeps one temp workspace, stages shared fixtures once, shares a benchmark-scoped cache, prewarms expensive resolver work where available, enforces bounded parallelism plus per-run timeouts, and still delegates aggregation and static review generation to the installed Anthropic `skill-creator` copy. - -This repo treats that paired `with_skill` / `without_skill` comparison as part of the required devex for skill work. The benchmark artifacts live in the temp workspace; do not commit them to this repository unless the change explicitly calls for checked-in examples. - -For scaffold/template skills, keep deterministic validators alongside evals. In this repo, `evals/evals.json` is mandatory, and validators like `scripts/validate-skill-templates.ps1` are additional protection. - -## Prefer dynamic defaults - -When a skill needs defaults for versions, paths, repository names, or support windows, prefer deriving them from a reliable source instead of baking in values that will drift. - -- Good sources: git metadata, repo folder names, environment values, official JSON feeds, vendor docs APIs -- Use hardcoded examples as examples only — not as the real defaulting mechanism — when the value can be computed - -## Template validation - -Use the repo validation harness before submitting scaffold or template changes: - -```console -pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -``` - -Run the validator locally first for the fastest feedback loop. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. - -To compare a change against the initial imported version, run the same harness against a git ref: - -```console -pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD -``` - -## Checklist before submitting - -- [ ] `SKILL.md` has valid front matter with `name` and `description` -- [ ] Skill is stack-agnostic (or clearly scoped to a specific tech in the name/description) -- [ ] Examples are generic — no personal emails, usernames, or project-specific identifiers -- [ ] At least one eval in `evals/evals.json` -- [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name -- [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder -- [ ] Skill changes were benchmarked from a temp workspace with both `with_skill` and `without_skill` runs -- [ ] `benchmark.json` and `eval-viewer/generate_review.py` from the installed Anthropic `skill-creator` copy were used so a human could compare `Outputs` and `Benchmark` -- [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior -- [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too -- [ ] Skill evals are intended to run from `$env:TEMP/-workspace/`, not from inside the repo -- [ ] Changed skill files are synced across `skills//`, `~/.claude/skills//`, and `~/.agents/skills//` -- [ ] Skill added to the table in `README.md` +# Contributing + +Thanks for wanting to add or improve a skill. Here's what to know. + +## Skill structure + +Each skill lives in its own folder: + +``` +skills/ + / + SKILL.md # Required — the skill content + FORMS.md # Optional — structured input collection for agents + assets/ # Optional — literal templates and static files + scripts/ # Optional — executable helpers + references/ # Optional — deeper docs loaded on demand + evals/ # Required for repo-managed skills — test prompts for validation + evals.json +``` + +## Local sync + +Repo-managed skills should be mirrored across all three locations: + +- `skills//` in this repo +- `~/.claude/skills//` +- `~/.agents/skills//` + +If you edit a local install copy first, copy the changed files back into the repo and into the other local install so every agent sees the same skill version. + +## SKILL.md format + +Every `SKILL.md` must start with a YAML front matter block: + +```yaml +--- +name: your-skill-name +description: > + One or two sentences describing what this skill does and when the AI + should automatically invoke it. Be specific about trigger phrases and + use cases — this description is what the AI reads to decide whether + to load the skill. +--- +``` + +The rest of the file is free-form Markdown. Include: + +- **When to use** — what scenarios or requests trigger this skill +- **Rules / conventions** — the core content the AI should follow +- **Examples** — good and bad, so the AI can calibrate +- **Prerequisites** — anything the human needs to set up first (tools, config, etc.) + +## Naming + +- Skill folder and `name` field: `kebab-case` +- Be specific — `git-bot-commits` is better than `git` or `commits` +- Avoid version numbers in names; use the description to note maturity + +## Writing good descriptions (the front matter field) + +The `description` is the most important field — it's how the AI decides to load the skill. Include: + +- What the skill enables +- Specific trigger phrases (e.g. "Use when user says 'commit this' or 'stage changes'") +- What it enforces or prevents + +## Adding evals (required for repo-managed skills) + +Evals let you verify the skill works and measure improvement over a baseline. Every repo-managed skill in this repository must include `evals/evals.json`: + +```json +{ + "skill_name": "your-skill-name", + "evals": [ + { + "id": 0, + "prompt": "The user message to test against", + "expected_output": "What a correct response looks like — used for manual or automated grading", + "files": ["evals/files/example.md"] + } + ] +} +``` + +`files` is optional. When present, list one or more fixture files relative to `skills//`. A common pattern is to store those fixtures under `evals/files/` so the eval package can attach the same source inputs to both the `with_skill` and `without_skill` prompt. + +Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and cases where the skill should *not* do something. + +Evals are prepared, not executed, from this repository. Adding or modifying a repo-managed skill requires preparing the packages for every skill the branch touched, which is a completion gate rather than an optional extra: + +```console +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +``` + +Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, which stays last. For a single skill on demand, use: + +```console +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +``` + +The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two hermetic run directories, `with_skill/` and `without_skill/`. A run directory is the worker's sandbox root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the agent of your choice. That agent starts immediately, creates one isolated worker for every run, launches it from its run directory with `repo/` as the working directory and `home/` as an isolated profile, gives each worker only its `prompt.md` and staged files, writes the results back, grades after collection using the packaged grader guidance, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. + +Repository scripts, CI jobs, and the agent that prepares a package never run those prompts. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A user-selected harness handed a specific package is the executor, not the preparer; its current context orchestrates fresh workers while the workers run the prompt files. + +Run both configurations on the same model, same version, and same configuration. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. + +Record each external result in the matching `results/*.result.json`: `model`, `provider`, `harness`, and the complete `output`; include `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, and `tool_calls` when the harness exposes them, and the `isolation` flags the harness confirmed. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. The normal external evaluator writes `grading[].passed` and evidence, then generates the report before handing the package back. If the results were transferred without those report artifacts, validate and compare with: + +```console +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults +``` + +That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external evaluator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. + +The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. + +For scaffold/template skills, keep deterministic validators alongside evals. In this repo, `evals/evals.json` is mandatory, and validators like `scripts/validate-skill-templates.ps1` are additional protection. + +## Prefer dynamic defaults + +When a skill needs defaults for versions, paths, repository names, or support windows, prefer deriving them from a reliable source instead of baking in values that will drift. + +- Good sources: git metadata, repo folder names, environment values, official JSON feeds, vendor docs APIs +- Use hardcoded examples as examples only — not as the real defaulting mechanism — when the value can be computed + +## Template validation + +Use the repo validation harness before submitting scaffold or template changes: + +```console +pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 +``` + +Run the validator locally first for the fastest feedback loop. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. + +To compare a change against the initial imported version, run the same harness against a git ref: + +```console +pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD +``` + +## Checklist before submitting + +- [ ] `SKILL.md` has valid front matter with `name` and `description` +- [ ] Skill is stack-agnostic (or clearly scoped to a specific tech in the name/description) +- [ ] Examples are generic — no personal emails, usernames, or project-specific identifiers +- [ ] At least one eval in `evals/evals.json` +- [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name +- [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder +- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` was run after the last skill edit, and the prepared prompt paths were reported +- [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only when transferred results need the repository-side fallback +- [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior +- [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too +- [ ] Eval packages live in `.bot/-workspace/` or a temp path, never anywhere else in the working tree +- [ ] Changed skill files are synced across `skills//`, `~/.claude/skills//`, and `~/.agents/skills//` +- [ ] Skill added to the table in `README.md` diff --git a/README.md b/README.md index 92d517b..dabf8f4 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,16 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository scripts, CI jobs, runners, graders, optimizers, and custom hooks from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. Model-backed candidate/baseline fan-out is not a completion gate. +Evaluation keeps Anthropic's `skill-creator` workflow and replaces only its execution transport. The repository prepares the paired candidate and baseline inputs as a portable package and stops. The agent chosen by the user later executes the package, grades the completed results with the packaged grader guidance, and invokes the packaged Anthropic aggregator and eval viewer. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: + +```powershell +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +``` + +That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. + +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two hermetic run directories. `with_skill/` is a self-contained sandbox root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the evaluator, grader, and report producer: it creates one fresh isolated worker per run, launches it from the run directory with `repo/` as the working directory and `home/` as the profile, records the complete response plus transcript, duration, token usage, optional turns/token buckets/cost, tool calls, and isolation guarantees, grades after collection with the packaged `skill-creator` guidance, then invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. The package guarantees identical repositories, skill-only-in-with_skill staging, and an isolated home; the harness must supply the runtime sandbox that keeps global skills, global config, and the source repository out of reach. A harness that can create isolated workers handles the complete run from that one file. `-CollectResults ` remains a fallback for transferred results without report artifacts; it validates the arms and invokes the same packaged tools. Packages land in gitignored `.bot/` storage by default and are not committed. + One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. Repo-level agent guidance also keeps progress updates user-facing: agents should report meaningful progress, evidence, blockers, and next steps without narrating sandbox mechanics, approved command paths, or retry plumbing unless those details affect approval, reproducibility, validation, or the final outcome. @@ -24,7 +34,7 @@ DocFX prose, cleanup, unresolved ownership, and skip-marker diagnostics are not The DocFX validator also treats fallback `docfx.json` discovery conservatively: if a repo lacks a live root DocFX workspace, scaffold/template configs under skill assets are ignored unless their metadata globs resolve real projects. That keeps placeholder files like `skills/dotnet-new-lib-slnx/assets/library/.docfx/docfx.json` from masquerading as the active documentation workspace during repo-wide audits. -Local skill synchronization is verified efficiently: run deterministic tests against the repository source, copy touched files to the three local installs, and compare SHA-256 hashes across all four locations. Hash-identical copies are the same executable content, so agents do not repeat the same suites from an installed path unless install-path or loader behavior is specifically under test. +Local skill synchronization is verified efficiently and mechanically: run deterministic tests against the repository source, then run `pwsh -NoProfile -File ./scripts/sync-skill-install.ps1 -Skill ` as the final step, which copies the whole skill tree to the three local installs and compares SHA-256 hashes across all four locations, exiting non-zero on any difference. Syncing a remembered list of touched files is not enough — that list goes stale on the next edit — and a sync claim must be backed by that command's output rather than an earlier run. Hash-identical copies are the same executable content, so agents do not repeat the same suites from an installed path unless install-path or loader behavior is specifically under test. Resumed DocFX audits preserve every tracked and untracked documentation edit, regenerate the assessment work queue and example inventory, and process that queue in batches with a fast rerun after each batch. Encoding checks focus on actual damage: valid BOM-less UTF-8 is accepted, `ENCODING_BOM_MISSING` is not emitted, and audits do not create BOM-only or line-ending-only diffs. @@ -117,7 +127,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith | [git-keep-a-changelog](skills/git-keep-a-changelog/SKILL.md) | Git-aware Keep a Changelog companion selected only for explicit changelog or release-note intent. Bare yolo/auto and commit-execution requests such as `git bot commit yolo` do not activate it; those words modify autonomy only after changelog intent is established. Bundled deterministic resolvers separate branch-unique commit history from merge-base-to-`HEAD` net diffs, exclude the previous-release or comparison boundary, fail on base-history bleed, and classify explicit path-backed release entities as `Added`, `Removed`, `Changed`, or `Unchanged`. The skill establishes each user-facing release entity against the base before section classification. It asks a mandatory `Yes / No / Custom` question before including pending worktree changes in ordinary concrete-release drafts, includes staged, unstaged, and untracked work automatically only in scoped yolo/auto mode, creates missing changelogs, writes SemVer-aware highlights, maintains compare-link footers, preserves natural prose wrapping, and curates surviving outcomes instead of dumping raw commit logs. | | [git-nuget-release-notes](skills/git-nuget-release-notes/SKILL.md) | Git-aware NuGet release-notes companion for .NET repos that keep cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Discovers packable `src/` projects, resolves concrete package version and availability, creates missing files when needed, reduces each package to its surviving base-to-`HEAD` delta before classifying history, and establishes each package capability against the base so pre-release refinements and fixes to a new capability remain one `ADDED` New Feature. It writes per-package `ALM` / `Breaking Changes` / `New Features` / `Improvements` / `Bug Fixes` style notes from final package state plus supporting commit context instead of dumping commit subjects. | | [git-nuget-readme](skills/git-nuget-readme/SKILL.md) | Git-aware NuGet README companion for .NET repos that advertise a package from `src/`. Resolves the real packable project the README should sell, combines git history with actual package metadata, source capabilities, and relevant tests when feasible, preserves honest badge/docs/contributing sections, and writes a forthcoming, adoption-friendly `README.md` with repo-derived branding, clear value, install, framework-support, and quick-start guidance. | -| [git-visual-squash-summary](skills/git-visual-squash-summary/SKILL.md) | Non-mutating grouped-summary companion to `git-visual-commits`. Turns the full current feature branch into a curated set of compact lowercase-start summary lines for PR or squash-and-merge contexts by default, comparing against the repository base branch rather than a same-named tracking remote, including commits from all authors unless explicitly narrowed, reducing the cumulative base-to-`HEAD` delta first so reverted churn disappears, preserving technical identifiers, merging overlap, keeping surviving dependency/version changes separate from build/refactor work when the final diff still shows them, and avoiding changelog-style wording, unsupported claims, yolo prompts, needless commit-range questions, or commit-selection UI for ordinary branch-level squash requests. | +| [git-visual-squash-summary](skills/git-visual-squash-summary/SKILL.md) | Non-mutating grouped-summary companion to `git-visual-commits`. Turns the full current feature branch into a curated set of compact lowercase-start summary lines for PR or squash-and-merge contexts by default, comparing against the repository base branch rather than a same-named tracking remote, including commits from all authors unless explicitly narrowed, reducing the cumulative base-to-`HEAD` delta first so reverted churn disappears, preserving technical identifiers, merging overlap, keeping surviving dependency/version changes separate from build/refactor work when the final diff still shows them, and avoiding changelog-style wording, unsupported claims, yolo prompts, needless commit-range questions, commit-selection UI for ordinary branch-level squash requests, or an instruction recap that asks permission before starting. | | [skill-creator-agnostic](skills/skill-creator-agnostic/SKILL.md) | **⚠️ Deprecated** — no longer maintained and retained only for backward compatibility until **1.0.0**. Do not use it for new skill-authoring work; use Anthropic `skill-creator` together with this repository's `AGENTS.md`. | | [markdown-illustrator](skills/markdown-illustrator/SKILL.md) | Reads a markdown file and answers directly in chat with one document-wide Visual Brief plus one compiled prompt. Infers a compact visual strategy by default, keeps follow-up questions near zero, and only branches when the user explicitly asks for added specificity. | | [git-repo-digest](skills/git-repo-digest/SKILL.md) | Turns any full repository URL into a deterministic digest workspace using the bundled .NET file-based runner `scripts/digest.cs`. Requires explicit `--repo-url`, resolves omitted output paths to `/.bot/digests` and passes that as `--output-root`, maps multiple positional URLs the same way for slash commands, bare pasted URLs, and natural-language requests by treating the first URL as the digest repo and every later URL as repeated `--external-repo-url`, always writes into `{output-root}/{repo-id}/{yyyyMMdd-HHmmssZ}`, accepts repeated curated public consumer repos, derives `{repo-id}`, fixes `result/`, performs shallow git clones, packs local tracked files with the bundled C# packer using `git ls-files`, separates XML evidence into `source.xml`, `tests.xml`, `projects.xml`, editorial `readmes.xml`, and scenario-only `external-usage.xml`, writes package and conceptual overview prompts under `prompts/`, emits public API summaries, engineering signals, evidence indexes, ordered XML chunks, referenced-package evidence maps for aggregate examples, and manifest-backed frontmatter hints, treats previous digest prose as contamination during fresh generation, then guides the agent to fully read the current phase's required evidence before writing package digests and a concept-led `result/Index.md` with YAML frontmatter containing Product-derived overview title metadata, validated documentation URLs resolved from PackageProjectUrl, documentation-host-filtered exact `.nuget//README.md` documentation links including emoji-prefixed Documentation headings and "More documentation..." blocks, DocFX `metadata[].dest` API paths, and source namespace page candidates from `src//**/*.cs`, target frameworks, package/library counts, external links, package-family links, and context glyphs, and validates authored result examples with `--validate-results` as a deterministic API-shape, Codebelt.Extensions.Xunit shape, PascalCase `MethodName_Scenario_ExpectedBehavior` test-method naming, Basic usage quality, and optimized NuGet-backed executable test gate with bounded parallelism. | @@ -128,9 +138,9 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | -| [dotnet-test](skills/dotnet-test/SKILL.md) | Bootstraps and refactors xUnit projects to Codebelt conventions. It deterministically inspects project roles, target frameworks, xUnit generation, package ownership, inheritance, application entry points—including Bootstrapper `MinimalConsoleProgram`, `MinimalWorkerProgram`, and `MinimalWebProgram` hosts—and every selected `WebApplicationFactory` usage; classifies ordinary unit, ASP.NET Core functional, and console/worker functional tests; modernizes xUnit v2 projects to xUnit v3 plus Microsoft Testing Platform without moving package ownership or changing frameworks; and resolves current stable compatible packages through NuGet-backed isolated compatibility-project restores, including the selected combined package set. Focused web tests use `WebApplicationTestFactory` with an explicit entrypoint-owned `ManagedWebApplicationFixture`, directly or through a narrow `Test`-derived harness; shared web fixtures use `WebApplicationTest` with `ManagedWebApplicationFixture`; focused console/worker tests use `ApplicationTestFactory` with `ManagedApplicationFixture`; and shared non-web fixtures use `ApplicationTest` with `ManagedApplicationFixture`. Deprecated blocking fixtures are migration inputs only and are never emitted because they are scheduled for removal. Functional migrations fail closed unless the chosen Codebelt pattern and managed fixture are present, the legacy or blocking fixture is absent, and test code does not reconstruct the production composition root with its own `WebApplication`, `TestServer`, or `HostBuilder`. Migrations preserve entrypoint-owned startup, host configuration, lazy start, clients, services, configuration, sync/async disposal, isolation, and existing test names, while fresh bootstraps add source-grounded behavior tests. Non-web tests stay in-process and require a resolvable Generic Host; test-only scope reports the exact production adaptation instead of silently rewriting startup or launching a process. | +| [dotnet-test](skills/dotnet-test/SKILL.md) | Moves xUnit projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's ASP.NET-only `WebApplicationFactory`—and the hand-rolled `HostBuilder` that console and worker tests reach for because Microsoft ships no equivalent—with one family of abstractions where the application's own entry point owns startup. Invocation is the request: it inspects and refactors immediately instead of opening with a menu or a questionnaire. It deterministically inspects project roles, target frameworks, xUnit generation, package ownership, inheritance, application entry points—including Bootstrapper `MinimalConsoleProgram`, `MinimalWorkerProgram`, and `MinimalWebProgram` hosts—and every selected `WebApplicationFactory` usage; classifies ordinary unit, ASP.NET Core functional, and console/worker functional tests; modernizes xUnit v2 projects to xUnit v3 plus Microsoft Testing Platform without moving package ownership or changing frameworks; and resolves current stable compatible packages through NuGet-backed isolated compatibility-project restores, including the selected combined package set. Focused web tests use `WebApplicationTestFactory` with an explicit entrypoint-owned `ManagedWebApplicationFixture`, directly or through a narrow `Test`-derived harness; shared web fixtures use `WebApplicationTest` with `ManagedWebApplicationFixture`; focused console/worker tests use `ApplicationTestFactory` with `ManagedApplicationFixture`; and shared non-web fixtures use `ApplicationTest` with `ManagedApplicationFixture`. Deprecated blocking fixtures are migration inputs only and are never emitted because they are scheduled for removal. Functional migrations fail closed unless the chosen Codebelt pattern and managed fixture are present, the legacy or blocking fixture is absent, and test code does not reconstruct the production composition root with its own `WebApplication`, `TestServer`, or `HostBuilder`. Migrations preserve entrypoint-owned startup, host configuration, lazy start, clients, services, configuration, sync/async disposal, isolation, and existing test names, while fresh bootstraps add source-grounded behavior tests. Non-web tests stay in-process and require a resolvable Generic Host; test-only scope reports the exact production adaptation instead of silently rewriting startup or launching a process. | | [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles plus a semantic preflight for truthful workload labels, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | -| [dotnet-remote-testing](skills/dotnet-remote-testing/SKILL.md) | Run .NET tests inside a resolved remote Docker environment and return concise, structured results — Visual Studio's Remote Testing experience (choose an environment → run tests → see results) with the container plumbing hidden behind a deterministic runner (`scripts/remote-test.cs`) the skill orchestrates instead of composing ad-hoc `docker run` commands. It honors Microsoft's existing `testenvironments.json` version-1 contract (`name`, `localRoot`, `dockerImage`, `dockerFile`, with the either/or Docker-source rule), treats that file as authoritative when present, and reports WSL/SSH/unknown types as unsupported rather than converting or silently ignoring them. When no `testenvironments.json` exists it provides a zero-configuration experience built exclusively on official `mcr.microsoft.com/dotnet/sdk` images, discovering the currently supported LTS and STS channels plus the current preview from Microsoft's live `releases-index.json` using `support-phase`/`release-type` (never hardcoded version numbers or even/odd assumptions) and caching that metadata outside the repository for offline reuse. It prefers an exact `latest-sdk` image tag (stripping preview build metadata), validates the tag against Microsoft's registry, and pins each execution to the resolved immutable digest so results are reproducible across environment, image, digest, SDK, and architecture. Execution stages the source into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, mounts a persistent NuGet cache outside the repo, runs restore → build → test with structured TRX collection, classifies failures into distinct kinds (configuration, unsupported environment, Docker unavailable, image resolution, SDK incompatibility, staging, restore, compilation, test-host, test failure, result-processing, cleanup, cancellation, release-metadata) so infrastructure problems are never reported as failing unit tests, and always cleans up transient Docker resources. It never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and never silently falls back to running tests on the host. Docker is the only transport for now, designed so WSL/SSH can be added later without disturbing the deterministic Docker path, which is covered by a comprehensive built-in `--self-test` plus a PowerShell harness. | +| [dotnet-remote-testing](skills/dotnet-remote-testing/SKILL.md) | Run .NET tests inside a resolved remote Docker environment and return concise, structured results — Visual Studio's Remote Testing experience (choose an environment → run tests → see results) with the container plumbing hidden behind a deterministic runner (`scripts/remote-test.cs`) the skill orchestrates instead of composing ad-hoc `docker run` commands. Invoking it is the request: with one applicable Docker environment it runs immediately — no capability menu, no parameter questionnaire, no confirmation — and when several channels are derived, the repository's own highest target framework selects the matching one and the choice is reported. The runner decides when a question is unavoidable, exiting `SelectionRequired` (16) with the exact candidates, and every exit code maps to exactly one next action so behavior is identical across models. It honors Microsoft's existing `testenvironments.json` version-1 contract (`name`, `localRoot`, `dockerImage`, `dockerFile`, with the either/or Docker-source rule), treats that file as authoritative when present, and reports WSL/SSH/unknown types as unsupported rather than converting or silently ignoring them. When no `testenvironments.json` exists it provides a zero-configuration experience built exclusively on official `mcr.microsoft.com/dotnet/sdk` images, discovering the currently supported LTS and STS channels plus the current preview from Microsoft's live `releases-index.json` using `support-phase`/`release-type` (never hardcoded version numbers or even/odd assumptions) and caching that metadata outside the repository for offline reuse. It prefers an exact `latest-sdk` image tag (stripping preview build metadata), validates the tag against Microsoft's registry, and pins each execution to the resolved immutable digest so results are reproducible across environment, image, digest, SDK, and architecture. Execution stages the source into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, mounts a persistent NuGet cache outside the repo, runs restore → build → test with structured TRX collection, classifies failures into distinct kinds (configuration, unsupported environment, Docker unavailable, image resolution, SDK incompatibility, staging, restore, compilation, test-host, test failure, result-processing, cleanup, cancellation, release-metadata) so infrastructure problems are never reported as failing unit tests, and always cleans up transient Docker resources. It never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and never silently falls back to running tests on the host. Docker is the only transport for now, designed so WSL/SSH can be added later without disturbing the deterministic Docker path, which is covered by a comprehensive built-in `--self-test` plus a PowerShell harness. | | [dotnet-segregated-assets](skills/dotnet-segregated-assets/SKILL.md) | Migrate or configure ASP.NET Core static delivery with `codebeltnet/web-cdn-origin:2.0.0` while keeping `wwwroot` as the authoring root. The deterministic runner inspects and verifies topology, publish exclusion, Static Web Assets risks, Cuemon signals, competing `AppAssetOptions`-style abstractions, actual `app-*`/`cdn-*` markup, and scheme-safe local origins; the agent performs semantic edits. For an existing Cuemon package reference, its plan resolves the highest stable version from NuGet.org at execution time, preserves Central Package Management versus inline ownership, excludes prereleases, and fails rather than copying an old fixture or example version. It reuses Cuemon `AppTagHelperOptions`/`CdnTagHelperOptions`, `BaseUrlMode`, and the public `app-link`, `app-script`, `app-img`, `cdn-link`, `cdn-script`, and `cdn-img` helpers when already available, otherwise reuses a suitable project abstraction without adding Cuemon. It keeps App and shared CDN ownership separate, preserves ordinary Project-based Development, adds opt-in segregated Development through a root Docker Compose profile, and makes `compose.assets.yml` directly build artifact-first `LocalDevelopment.Dockerfile` and `Assets.Dockerfile` images. Every generated file comes from a literal template in `assets/` and lands in one fixed location — the three Dockerfiles beside the web `.csproj`, orchestration at the repository root — and `verify --check-local` proves that placement along with the artifact-first contract: no SDK stage or `dotnet publish` inside an application image, a `.dockerignore` that still carries `artifacts/`, `LocalPublishDirectory` behind a guarded post-build target, Compose host ports derived from the ordinary Project profile, and a CI job that produces the artifact those images copy. Production CI publishes the same application artifact for the shell-less runtime `Dockerfile`. The skill excludes app-owned `wwwroot` with targeted MSBuild metadata, preserves `_content`/`_framework` and generated Static Web Assets, and proves publish/local invariants deterministically and idempotently. | | [agent-smith](skills/agent-smith/SKILL.md) | Apply a rigorous, consistent, evidence-driven software-craftsmanship standard across a whole engineering task. Invoke explicitly as `/agent-smith ` or let it auto-trigger for design, architecture, implementation, refactoring, code review, public API review, compatibility and Semantic Versioning analysis, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, repository governance, and engineering assessment. Skill-authoring mode grounds instructions in real execution, requires an explicit bounded-concurrency assessment so independent data retrieval and eval work do not remain sequential by habit, favors reusable C#/.NET scripts and validators against the dynamically resolved latest supported LTS when local constraints do not decide, and follows the Agent Skills guidance for progressive disclosure, description optimization, candidate-versus-baseline evaluation, aggregation, and human review. Its optional .NET EditorConfig conformance mode handles targeted IDE/CA diagnostic remediation and full informational-or-higher `dotnet format` conformance without treating a clean build as proof of policy compliance: user-defined diagnostic IDs remain task-supplied data; target, path, and severity scope remains authoritative; informational workflows explicitly preserve `--severity info` because the formatter defaults to `warn`; targeted IDE and analyzer checks use category-specific formatter subcommands; every formatter invocation is read-only via `--verify-no-changes`; `--no-restore` is never treated as a conformance fallback; fixes are deliberate source edits; repeated multi-target findings are de-duplicated by physical file, diagnostic, and span; and the bundled `repair-roslyn-multiproject-artifacts.ps1` detects conflict artifacts independently of diagnostic ID, preflights directory repairs without partial writes, repairs only proven structural patterns, and refuses unrecognized shapes. Completion requires the same scoped formatter gate plus an artifact scan before affected builds and relevant tests. Technology-neutral work remains unaffected. Performs the requested work (not just a review), loads only relevant `references/`, respects repository conventions, scales process depth without lowering the standard, and reports evidence and risk honestly in concise feedback that may sacrifice grammar but never required evidence. Governing principle: consistency is key. | Invoke explicitly as `/agent-smith ` or let it auto-trigger for design, architecture, implementation, refactoring, code review, public API review, compatibility and Semantic Versioning analysis, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, repository governance, and engineering assessment. Skill-authoring mode grounds instructions in real execution, requires an explicit bounded-concurrency assessment so independent data retrieval and eval work do not remain sequential by habit, favors reusable C#/.NET scripts and validators against the dynamically resolved latest supported LTS when local constraints do not decide, and follows the Agent Skills guidance for progressive disclosure, description optimization, candidate-versus-baseline evaluation, aggregation, and human review. Its optional .NET EditorConfig conformance mode handles targeted IDE/CA diagnostic remediation and full informational-or-higher `dotnet format` conformance without treating a clean build as proof of policy compliance: user-defined diagnostic IDs remain task-supplied data; target, path, and severity scope remains authoritative; informational workflows explicitly preserve `--severity info` because the formatter defaults to `warn`; targeted IDE and analyzer checks use category-specific formatter subcommands; every formatter invocation is read-only via `--verify-no-changes`; `--no-restore` is never treated as a conformance fallback; fixes are deliberate source edits; repeated multi-target findings are de-duplicated by physical file, diagnostic, and span; and the bundled `repair-roslyn-multiproject-artifacts.ps1` detects conflict artifacts independently of diagnostic ID, preflights directory repairs without partial writes, repairs only proven structural patterns, and refuses unrecognized shapes. Completion requires the same scoped formatter gate plus an artifact scan before affected builds and relevant tests. Technology-neutral work remains unaffected. Performs the requested work (not just a review), loads only relevant `references/`, respects repository conventions, scales process depth without lowering the standard, and reports evidence and risk honestly in concise feedback that may sacrifice grammar but never required evidence. Governing principle: consistency is key. | @@ -307,6 +317,7 @@ Sometimes the history is already written and the only thing you need is the fina - **Whole-branch by default** — for squash-and-merge requests, uses the full current feature branch from merge-base to `HEAD` instead of asking which branch commits to include - **All authors included** — branch-level summaries treat branch topology as the scope and include every contributor's commits unless the user explicitly asks for an author-filtered summary - **Bare invocation means summarize now** — calling `git-visual-squash-summary` directly should resolve the current branch scope automatically and return the grouped lines, not a "what do you want me to summarize?" question +- **The summary is the acknowledgment** — the first response is the grouped lines themselves, never an "I understand the instructions" recap followed by "would you like me to generate it now?" - **Base branch, not tracking copy** — a feature branch that is in sync with `origin/` is still summarized against `origin/HEAD`, `origin/main`, `origin/master`, `main`, or `master` before declaring there is nothing to summarize - **No yolo prompt** — the skill is read-only, so it acts directly without asking for auto-approval language from mutating workflows - **No commit-picker UX** — ordinary branch-level squash requests do not become commit-selection questions or widgets; the skill resolves the branch scope and writes the summary @@ -377,7 +388,7 @@ Choosing a NuGet package often happens fast: a developer lands on the README, sc ### Why skill-creator-agnostic is deprecated -`skill-creator-agnostic` is now a legacy compatibility artifact, not an active skill-authoring workflow. It is **⚠️ Deprecated**, no longer maintained, and scheduled for removal in **1.0.0**. +`skill-creator-agnostic` is now a legacy compatibility artifact, not an active skill-authoring workflow. It is **⚠️ Deprecated**, no longer maintained, and scheduled for removal in **1.0.0**. Its references are retained for backward compatibility only; they do not replace Anthropic's supported `skill-creator` workflow. For new skill creation, modification, and benchmarking: @@ -645,8 +656,14 @@ API documentation rots the moment code changes. A new public type ships without Test-project refactoring is deceptively lifecycle-sensitive. A `WebApplicationFactory` wrapper may own temporary directories, defer host startup until the first client, replace services in a specific order, or isolate settings per test. Console and worker tests have a different boundary: they need a resolvable in-process Generic Host, not a child process hidden behind a test helper. -**dotnet-test** begins with machine-readable inspection, then chooses the Codebelt pattern that matches the selected project's role and ownership model. It preserves package ownership and frameworks, migrates xUnit v2 to v3/Microsoft Testing Platform when needed, and makes the chosen focused/shared web or application pattern, entrypoint-owned managed fixture, zero remaining selected `WebApplicationFactory` usages, zero deprecated blocking fixtures, zero replacement composition roots, and restore/build/test explicit gates. +The skill has one job: the test host comes from Codebelt, not from Microsoft, and not from a builder written in the test project. `WebApplicationFactory` covers ASP.NET Core and nothing else, so console and worker tests end up hand-rolling a host that no deployed process ever runs. Codebelt closes both gaps with `WebApplicationTestFactory`, `WebApplicationTest`, `ApplicationTestFactory`, and `ApplicationTest`. + +**dotnet-test** begins with machine-readable inspection, then chooses the Codebelt pattern that matches the selected project's role and ownership model. Because that inspection already answers which project, role, and ownership apply, the skill acts on the evidence instead of asking the developer to retype what the JSON says. It preserves package ownership and frameworks, migrates xUnit v2 to v3/Microsoft Testing Platform when needed, and makes the chosen focused/shared web or application pattern, entrypoint-owned managed fixture, zero remaining selected `WebApplicationFactory` usages, zero deprecated blocking fixtures, zero replacement composition roots, and restore/build/test explicit gates. + +The hardest failure to catch is the migration that only looks like one. Wrapping `WebApplicationFactory` in a private nested class, renaming a constructor to a static `Create`, or bumping package pins all produce a substantial diff while Microsoft's host still starts the application and every test stays green. `verify-dotnet-test-migration.ps1` closes the run with a verdict instead of a self-assessment: it re-checks the expected pattern and adds the post-edit checks for a surviving factory base type, a retained `Microsoft.AspNetCore.Mvc.Testing` reference, `xunit*` pins past the major the restored Codebelt package declares, and files changed under the project while the target pattern appears nowhere in it. +- **Evidence before questions** — the bundled inspector resolves project, role, mode, host ownership, and package owner, so a bare invocation starts working instead of returning a capability menu, +- **Managed-fixture version floor** — inspection flags a Codebelt xUnit package pinned below 11.1.0, where the managed fixtures do not exist yet, before the pattern is written rather than after it fails to compile, - **Three explicit roles** — ordinary unit, ASP.NET Core functional, and console/worker functional tests route to separate references and assets, - **Lifecycle-preserving functional migration** — focused factories or narrow `Test`-derived harnesses and shared managed fixtures retain configuration, lazy start, client/service access, synchronous/asynchronous disposal, and isolation, - **Real entry-point coverage** — focused and shared postconditions reject test-owned `WebApplication`/`TestServer` pipelines that can pass while the production `Program` is broken, @@ -654,7 +671,9 @@ Test-project refactoring is deceptively lifecycle-sensitive. A `WebApplicationFa - **Bootstrapper host fidelity** — Startup-based hosts and `MinimalConsoleProgram`, `MinimalWorkerProgram`, or `MinimalWebProgram` hosts remain in their established family instead of being rewritten for test convenience, - **Dynamic compatibility** — stable package versions come from NuGet and must pass isolated compatibility-project restores, including the selected combined package set and target frameworks, - **Source-grounded bootstrap** — new projects receive at least one behavior test derived from real source instead of a placeholder, -- **Deterministic evidence** — inspection JSON reports roles, frameworks, xUnit generation, package owners, inheritance, migrations, recommendations, and blockers before mutation. +- **Deterministic evidence** — inspection JSON reports roles, frameworks, xUnit generation, package owners, inheritance, migrations, recommendations, and blockers before mutation, +- **A verdict, not a summary** — the migration gate exits `PASSED` or `FAILED` with numbered violations and their file and line, so a wrapped, renamed, or repackaged factory is reported as an unfinished migration rather than a completed one, +- **Anchored xUnit versions** — the gate reads the restored Codebelt package's own declared dependencies from `project.assets.json`, so an after-the-fact bump past that xUnit generation is caught offline instead of at the next compile. ### Why dotnet-benchmark? @@ -689,13 +708,17 @@ Cross-platform .NET developers usually get Linux test feedback the slow way: pus - **Microsoft's contract, not a new one** — honors the existing `testenvironments.json` version-1 schema (`name`, `localRoot`, `dockerImage`, `dockerFile`, either/or Docker source), treats it as authoritative when present, and never modifies it unless asked - **Zero-configuration by default** — with no `testenvironments.json`, it derives environments from Microsoft's live `releases-index.json` (supported LTS/STS channels plus the current preview) using `support-phase`/`release-type`, so no files are added to the repo and `.NET 10`/`.NET 11` are never hardcoded +- **Runs instead of asking** — invoking the skill *is* the request, so a repository with one applicable Docker environment goes straight to a test run with no menu, no questionnaire, and no confirmation; when several environments are derived, the repository's own target frameworks select one and the runner explains the choice +- **Whole matrix in one container** — a Microsoft SDK image ships a single runtime, so a repository targeting `net9.0;net10.0` builds there and then cannot execute the lower TFM; multi-targeted repositories resolve instead to a `codebeltnet/ubuntu-testrunner` combined tag (e.g. `8-9-10-11`) discovered from its live tag feed, running every target framework in one pass. Pointing a multi-targeted repo at a single-SDK image is reported as an SDK incompatibility naming the remedy, and `--framework` narrows the environment choice along with the run +- **One question, only when there is a real one** — the runner decides when a choice remains and exits `SelectionRequired` (16) with the exact candidates, so the single unavoidable question is precise and never expands into an intake form - **Offline-safe discovery and explicit scoping** — successful release metadata is cached outside the repo for offline reuse, and when a project choice is needed the form exposes the exact runner-computed target as the recommended option alongside a custom path -- **Official images, pinned to a digest** — auto-generated environments use only `mcr.microsoft.com/dotnet/sdk`, prefer the exact `latest-sdk` tag (preview build metadata stripped), validate the tag against Microsoft's registry, and resolve an immutable digest so a run is reproducible across environment, image, digest, SDK, and architecture +- **Recommended images, pinned to a digest** — auto-generated environments come from `mcr.microsoft.com/dotnet/sdk` for a single .NET major or `codebeltnet/ubuntu-testrunner` for several, prefer the exact `latest-sdk` tag (preview build metadata stripped), validate the tag against the registry, and resolve an immutable digest so a run is reproducible across environment, image, digest, SDK, and architecture; other images are never substituted on the runner's own initiative - **Tests run in Docker, the host stays clean** — source is staged into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, a persistent NuGet cache lives outside the repo, and it never silently falls back to running tests locally -- **Honest failure classification** — configuration, unsupported environment, Docker-unavailable, image-resolution, SDK-incompatibility, staging, restore, compilation, test-host, test-failure, result-processing, cleanup, cancellation, and release-metadata failures are distinct, so a container problem is never reported as a failing unit test +- **Honest failure classification** — configuration, unsupported environment, Docker-unavailable, image-resolution, SDK-incompatibility, staging, restore, compilation, test-host, test-failure, result-processing, cleanup, cancellation, release-metadata, and selection-required outcomes are distinct exit codes mapped to exactly one next action each, so behavior is the same whichever model is driving and a container problem is never reported as a failing unit test +- **Timing you can trust** — reports test duration and total wall clock separately, so a fast suite behind a slow image pull is never presented as an instant run - **No plumbing added, ever** — never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and always cleans up transient Docker resources — reporting exact identifiers if any remain - **Target-framework aware** — inspects the projects and `global.json`, refuses to pick an SDK that cannot build the requested target framework, and reports incompatibilities instead of editing the repository to force them -- **Deterministic and tested** — the runner ships a comprehensive built-in `--self-test` plus a PowerShell harness covering configuration discovery, release parsing, environment selection, unsupported handling, image resolution, command planning, result parsing, failure classification, cancellation, and cleanup +- **Deterministic and tested** — the runner ships a comprehensive built-in `--self-test` plus a PowerShell harness covering configuration discovery, release parsing, environment selection (including unattended target-framework selection and the cases that must still ask), unsupported handling, image resolution, command planning, result parsing, failure classification, cancellation, and cleanup ### Why dotnet-segregated-assets? `wwwroot` is where every ASP.NET Core developer expects to author static files — editors, hot reload, and the SDK all assume it. But shipping those files inside the deployed web application couples static delivery to business logic, bloats the app artifact, and puts asset caching on the wrong surface. The right shape is architectural: keep authoring in `wwwroot`, but let a separate, hardened static-content host serve the files in production. diff --git a/scripts/eval-report-template.html b/scripts/eval-report-template.html new file mode 100644 index 0000000..e5b6588 --- /dev/null +++ b/scripts/eval-report-template.html @@ -0,0 +1,311 @@ + + + + + + Eval Review + + + +
+

Eval Review:

+
+
+ + +
+
+
+ + + + diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 new file mode 100644 index 0000000..63be59f --- /dev/null +++ b/scripts/generate-eval-report.ps1 @@ -0,0 +1,708 @@ +<# +.SYNOPSIS + Adapts this repository's portable eval results to Anthropic skill-creator's benchmark and viewer tools. + +.DESCRIPTION + This adapter mirrors the recorded portable results into skill-creator's eval workspace contract and invokes the + upstream aggregator and viewer. It also writes a first-party side-by-side report.html with paired outputs, + assertion evidence, optional run telemetry, transcripts, and human feedback. The exact upstream viewer is kept + beside it as skill-creator-report.html for compatibility and comparison. + + No model or grader is started by this script. Grading must already be present in the portable result files or + must have been performed by the user-directed external evaluator before this adapter is called. + +.PARAMETER IterationDirectory + Prepared eval iteration containing manifest.json and the eval-case result files. + +.PARAMETER OutputPath + Optional HTML path. Defaults to report.html at the iteration root. + +.PARAMETER BenchmarkPath + Optional benchmark JSON path. Defaults to benchmark.json at the iteration root. + +.PARAMETER BenchmarkMarkdownPath + Optional benchmark Markdown path. Defaults to benchmark.md at the iteration root. + +.PARAMETER SkillCreatorPath + Optional skill-creator installation or package-local tools/skill-creator path. The package-local path is the + default so a prepared package remains self-contained after preparation. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [string]$IterationDirectory, + + [string]$OutputPath, + + [string]$BenchmarkPath, + + [string]$BenchmarkMarkdownPath, + + [string]$SkillCreatorPath +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Read-JsonFile { + param([string]$Path) + + if (-not (Test-Path -LiteralPath $Path)) { + throw "Missing JSON file '$Path'." + } + + return [System.IO.File]::ReadAllText($Path, $utf8NoBom) | ConvertFrom-Json +} + +function Write-JsonFile { + param( + [string]$Path, + [object]$Value + ) + + $directory = Split-Path -Parent $Path + if (-not [string]::IsNullOrWhiteSpace($directory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 30) + [Environment]::NewLine), $utf8NoBom) +} + +function Write-TextFile { + param( + [string]$Path, + [string]$Content + ) + + $directory = Split-Path -Parent $Path + if (-not [string]::IsNullOrWhiteSpace($directory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + + [System.IO.File]::WriteAllText($Path, (($Content -replace "`r`n", "`n" -replace "`r", "`n") + [Environment]::NewLine), $utf8NoBom) +} + +function Get-Property { + param( + [object]$Object, + [string]$Name, + [object]$Default = $null + ) + + if ($null -ne $Object -and $Object -is [System.Collections.IDictionary] -and $Object.Contains($Name) -and $null -ne $Object[$Name]) { + return $Object[$Name] + } + + if ($null -ne $Object -and $Object.PSObject.Properties.Name -contains $Name -and $null -ne $Object.$Name) { + return $Object.$Name + } + + return $Default +} + +function Get-SafeSegment { + param([string]$Value) + + $safe = $Value -replace '[^A-Za-z0-9._-]+', '-' + $safe = $safe.Trim('-') + if ([string]::IsNullOrWhiteSpace($safe)) { + return 'eval' + } + + return $safe +} + +function Resolve-SkillCreatorPath { + param([string]$RequestedPath) + + $candidates = [System.Collections.Generic.List[string]]::new() + if (-not [string]::IsNullOrWhiteSpace($RequestedPath)) { + $candidates.Add($RequestedPath) + } + + $packagePath = Join-Path $iterationPath 'tools/skill-creator' + $candidates.Add($packagePath) + + if (-not [string]::IsNullOrWhiteSpace($env:SKILL_CREATOR_PATH)) { + $candidates.Add($env:SKILL_CREATOR_PATH) + } + + if (-not [string]::IsNullOrWhiteSpace($env:USERPROFILE)) { + $candidates.Add((Join-Path $env:USERPROFILE '.agents/skills/skill-creator')) + $candidates.Add((Join-Path $env:USERPROFILE '.claude/skills/skill-creator')) + $candidates.Add((Join-Path $env:USERPROFILE '.gemini/antigravity-cli/skills/skill-creator')) + } + + foreach ($candidate in $candidates) { + if ((Test-Path -LiteralPath (Join-Path $candidate 'scripts/aggregate_benchmark.py')) -and + (Test-Path -LiteralPath (Join-Path $candidate 'eval-viewer/generate_review.py')) -and + (Test-Path -LiteralPath (Join-Path $candidate 'eval-viewer/viewer.html'))) { + return (Resolve-Path -LiteralPath $candidate).Path + } + } + + throw "Anthropic skill-creator eval tools were not found. Prepare the package with skill-creator available, or pass -SkillCreatorPath / set SKILL_CREATOR_PATH. Expected scripts/aggregate_benchmark.py and eval-viewer/generate_review.py." +} + +function Resolve-PythonCommand { + $python = Get-Command python -ErrorAction SilentlyContinue + if ($null -ne $python) { + return $python.Source + } + + $pythonLauncher = Get-Command py -ErrorAction SilentlyContinue + if ($null -ne $pythonLauncher) { + return $pythonLauncher.Source + } + + throw 'Python is required to run Anthropic skill-creator aggregation and the eval viewer.' +} + +function Invoke-PythonScript { + param( + [string]$PythonCommand, + [string]$ScriptPath, + [string[]]$Arguments + ) + + $output = & $PythonCommand $ScriptPath @Arguments 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Python script '$ScriptPath' failed with exit code ${LASTEXITCODE}:`n$($output -join [Environment]::NewLine)" + } + + foreach ($line in @($output)) { + Write-Host $line + } +} + +function Get-ResultPath { + param( + [string]$EvalDirectory, + [string]$Configuration + ) + + $fileName = if ($Configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + return Join-Path (Join-Path $EvalDirectory 'results') $fileName +} + +function Copy-RecordedOutputFiles { + param( + [object]$Result, + [string]$RunPackageDirectory, + [string]$EvalDirectory, + [string]$IterationPath, + [string]$OutputDirectory + ) + + $index = 0 + foreach ($recordedPath in @(Get-Property -Object $Result -Name 'output_files' -Default @())) { + if ([string]::IsNullOrWhiteSpace([string]$recordedPath)) { + continue + } + + $relativePath = [string]$recordedPath + if ([System.IO.Path]::IsPathRooted($relativePath)) { + continue + } + + $source = $null + foreach ($candidate in @( + (Join-Path $RunPackageDirectory $relativePath), + (Join-Path $EvalDirectory $relativePath), + (Join-Path $IterationPath $relativePath))) { + if (Test-Path -LiteralPath $candidate -PathType Leaf) { + $source = (Resolve-Path -LiteralPath $candidate).Path + break + } + } + + if ($null -eq $source) { + continue + } + + $leaf = [System.IO.Path]::GetFileName($relativePath) + if ([string]::IsNullOrWhiteSpace($leaf)) { + $leaf = "output-$index.bin" + } + + $destination = Join-Path $OutputDirectory ("{0:D2}-{1}" -f $index, $leaf) + Copy-Item -LiteralPath $source -Destination $destination -Force + $index++ + } +} + +function Get-ReportMimeType { + param([string]$Extension) + + switch ($Extension.ToLowerInvariant()) { + '.png' { return 'image/png' } + '.jpg' { return 'image/jpeg' } + '.jpeg' { return 'image/jpeg' } + '.gif' { return 'image/gif' } + '.svg' { return 'image/svg+xml' } + '.webp' { return 'image/webp' } + default { return 'application/octet-stream' } + } +} + +function Get-ReportOutputFiles { + param( + [object]$Result, + [string]$RunPackageDirectory, + [string]$EvalDirectory, + [string]$IterationPath + ) + + $textExtensions = @('.txt', '.md', '.json', '.csv', '.py', '.js', '.ts', '.tsx', '.jsx', '.yaml', '.yml', '.xml', '.html', '.css', '.sh', '.rb', '.go', '.rs', '.java', '.c', '.cpp', '.h', '.hpp', '.sql', '.r', '.toml') + $imageExtensions = @('.png', '.jpg', '.jpeg', '.gif', '.svg', '.webp') + $files = [System.Collections.Generic.List[object]]::new() + $index = 0 + + foreach ($recordedPath in @(Get-Property -Object $Result -Name 'output_files' -Default @())) { + $relativePath = [string]$recordedPath + if ([string]::IsNullOrWhiteSpace($relativePath) -or [System.IO.Path]::IsPathRooted($relativePath)) { + continue + } + + $source = $null + foreach ($candidate in @( + (Join-Path $RunPackageDirectory $relativePath), + (Join-Path $EvalDirectory $relativePath), + (Join-Path $IterationPath $relativePath))) { + if (Test-Path -LiteralPath $candidate -PathType Leaf) { + $source = (Resolve-Path -LiteralPath $candidate).Path + break + } + } + if ($null -eq $source) { + continue + } + + $extension = [System.IO.Path]::GetExtension($source).ToLowerInvariant() + $leaf = [System.IO.Path]::GetFileName($source) + if ([string]::IsNullOrWhiteSpace($leaf)) { + $leaf = "output-$index.bin" + } + + if ($textExtensions -contains $extension) { + $files.Add([ordered]@{ + name = $leaf + type = 'text' + content = [System.IO.File]::ReadAllText($source, $utf8NoBom) + }) + } elseif ($imageExtensions -contains $extension) { + $bytes = [System.IO.File]::ReadAllBytes($source) + $files.Add([ordered]@{ + name = $leaf + type = 'image' + data_uri = "data:$(Get-ReportMimeType -Extension $extension);base64,$([Convert]::ToBase64String($bytes))" + }) + } else { + $bytes = [System.IO.File]::ReadAllBytes($source) + $files.Add([ordered]@{ + name = $leaf + type = 'binary' + data_uri = "data:application/octet-stream;base64,$([Convert]::ToBase64String($bytes))" + }) + } + $index++ + } + + return @($files) +} + +function Get-ReportGrades { + param( + [object]$Result, + [string[]]$Assertions + ) + + $grading = @(Get-Property -Object $Result -Name 'grading' -Default @()) + $count = [Math]::Max($grading.Count, $Assertions.Count) + $grades = [System.Collections.Generic.List[object]]::new() + for ($index = 0; $index -lt $count; $index++) { + $grade = if ($index -lt $grading.Count) { $grading[$index] } else { $null } + $text = [string](Get-Property -Object $grade -Name 'text' -Default '') + $generic = [string]::IsNullOrWhiteSpace($text) -or $text -match '^(Passed|Failed|Assertion\s+\d+)$' + if ($generic -and $index -lt $Assertions.Count) { + $text = [string]$Assertions[$index] + } + if ([string]::IsNullOrWhiteSpace($text)) { + $text = 'Assertion' + } + + $passed = Get-Property -Object $grade -Name 'passed' -Default $null + $evidence = [string](Get-Property -Object $grade -Name 'evidence' -Default '') + if ($null -eq $grade) { + $evidence = 'Not graded.' + } + $grades.Add([ordered]@{ + text = $text + passed = $passed + evidence = $evidence + }) + } + + return @($grades) +} + +function Get-ReportMetric { + param( + [object]$Result, + [string[]]$Names + ) + + foreach ($name in $Names) { + $value = Get-Property -Object $Result -Name $name -Default $null + if ($null -ne $value -and -not [string]::IsNullOrWhiteSpace([string]$value)) { + return $value + } + } + + return $null +} + +function Get-ReportRun { + param( + [object]$Result, + [string]$Configuration, + [string]$EvalName, + [int]$EvalId, + [string[]]$Assertions, + [string]$RunPackageDirectory, + [string]$EvalDirectory, + [string]$IterationPath + ) + + if ($null -eq $Result) { + return $null + } + + $output = [string](Get-Property -Object $Result -Name 'output' -Default '') + $outputFiles = @(Get-ReportOutputFiles -Result $Result -RunPackageDirectory $RunPackageDirectory -EvalDirectory $EvalDirectory -IterationPath $IterationPath) + $grades = @(Get-ReportGrades -Result $Result -Assertions $Assertions) + $metrics = [ordered]@{ + turns = Get-ReportMetric -Result $Result -Names @('turns', 'turn_count', 'total_turns') + duration_seconds = Get-ReportMetric -Result $Result -Names @('duration_seconds') + base_input_tokens = Get-ReportMetric -Result $Result -Names @('base_input_tokens', 'input_tokens') + output_tokens = Get-ReportMetric -Result $Result -Names @('output_tokens', 'total_output_tokens') + cache_read_tokens = Get-ReportMetric -Result $Result -Names @('cache_read_tokens') + cache_write_tokens = Get-ReportMetric -Result $Result -Names @('cache_write_tokens') + cache_write_1h_tokens = Get-ReportMetric -Result $Result -Names @('cache_write_1h_tokens') + estimated_cost_usd = Get-ReportMetric -Result $Result -Names @('estimated_cost_usd', 'cost_usd') + total_tokens = Get-ReportMetric -Result $Result -Names @('total_tokens') + tool_calls = Get-ReportMetric -Result $Result -Names @('tool_calls') + } + + return [ordered]@{ + configuration = $Configuration + feedback_key = "eval-$EvalId-$Configuration" + model = [string](Get-Property -Object $Result -Name 'model' -Default '') + provider = [string](Get-Property -Object $Result -Name 'provider' -Default '') + harness = [string](Get-Property -Object $Result -Name 'harness' -Default '') + executed_utc = [string](Get-Property -Object $Result -Name 'executed_utc' -Default '') + output = $output + output_files = @($outputFiles) + transcript = [string](Get-Property -Object $Result -Name 'transcript' -Default '') + shell_commands = @(Get-Property -Object $Result -Name 'shell_commands' -Default @()) + files_read = @(Get-Property -Object $Result -Name 'files_read' -Default @()) + files_written = @(Get-Property -Object $Result -Name 'files_written' -Default @()) + stdout = [string](Get-Property -Object $Result -Name 'stdout' -Default '') + stderr = [string](Get-Property -Object $Result -Name 'stderr' -Default '') + exit_status = Get-Property -Object $Result -Name 'exit_status' -Default $null + metrics = $metrics + isolation = Get-Property -Object $Result -Name 'isolation' -Default $null + grades = @($grades) + notes = Get-Property -Object $Result -Name 'notes' -Default '' + } +} + +function Get-ReportSkillStats { + param( + [object]$Manifest, + [string]$IterationPath + ) + + $skillRoot = $null + foreach ($entry in @($Manifest.evals)) { + $candidate = Join-Path (Join-Path (Join-Path $IterationPath ([string]$entry.directory)) 'with_skill') ("skill/$($Manifest.skill_name)") + if (Test-Path -LiteralPath $candidate -PathType Container) { + $skillRoot = $candidate + break + } + } + + $files = if ($null -ne $skillRoot) { @(Get-ChildItem -LiteralPath $skillRoot -Recurse -File -Force) } else { @() } + $totalBytes = if ($files.Count -gt 0) { [int64](($files | Measure-Object -Property Length -Sum).Sum) } else { 0 } + $inlinedBytes = Get-Property -Object $Manifest.skill_instructions -Name 'inlined_resource_bytes' -Default $null + return [ordered]@{ + file_count = $files.Count + total_bytes = $totalBytes + inlined_bytes = $inlinedBytes + token_count = Get-Property -Object $Manifest -Name 'skill_token_count' -Default $null + hash = [string](Get-Property -Object $Manifest -Name 'skill_hash' -Default '') + } +} + +function Write-FirstPartyReport { + param( + [object]$Manifest, + [string]$IterationPath, + [string]$OutputPath, + [object]$Benchmark + ) + + $evals = [System.Collections.Generic.List[object]]::new() + $allModels = [System.Collections.Generic.List[string]]::new() + $allProviders = [System.Collections.Generic.List[string]]::new() + $completedRuns = 0 + foreach ($entry in @($Manifest.evals)) { + $evalDirectory = Join-Path $IterationPath ([string]$entry.directory) + $metadata = Read-JsonFile -Path (Join-Path $evalDirectory 'eval-metadata.json') + $runMap = [ordered]@{} + $assertions = @($metadata.assertions | ForEach-Object { [string]$_ }) + foreach ($configuration in @('with_skill', 'without_skill')) { + $resultPath = Get-ResultPath -EvalDirectory $evalDirectory -Configuration $configuration + $result = if (Test-Path -LiteralPath $resultPath) { Read-JsonFile -Path $resultPath } else { $null } + if ($null -ne $result) { + $run = Get-ReportRun -Result $result -Configuration $configuration -EvalName ([string]$entry.eval_name) -EvalId ([int]$metadata.eval_id) -Assertions $assertions -RunPackageDirectory (Join-Path $evalDirectory $configuration) -EvalDirectory $evalDirectory -IterationPath $IterationPath + $runMap[$configuration] = $run + if (-not [string]::IsNullOrWhiteSpace([string]$run.output) -or @($run.output_files).Count -gt 0) { $completedRuns++ } + if (-not [string]::IsNullOrWhiteSpace([string]$run.model) -and -not $allModels.Contains([string]$run.model)) { + $allModels.Add([string]$run.model) + } + if (-not [string]::IsNullOrWhiteSpace([string]$run.provider) -and -not $allProviders.Contains([string]$run.provider)) { + $allProviders.Add([string]$run.provider) + } + } else { + $runMap[$configuration] = $null + } + } + $evals.Add([ordered]@{ + id = [int]$metadata.eval_id + name = [string]$metadata.eval_name + prompt = [string]$metadata.prompt + expected_output = [string](Get-Property -Object $metadata -Name 'expected_output' -Default '') + assertions = @($assertions) + runs = $runMap + }) + } + + $metadata = [ordered]@{ + model = if ($allModels.Count -gt 0) { $allModels -join ', ' } else { $null } + provider = if ($allProviders.Count -gt 0) { $allProviders -join ', ' } else { $null } + completed_runs = $completedRuns + expected_runs = @($Manifest.evals).Count * 2 + generated_utc = [string](Get-Property -Object $Manifest -Name 'generated_utc' -Default '') + } + $reportData = [ordered]@{ + skill_name = [string]$Manifest.skill_name + iteration = [int]$Manifest.iteration + metadata = $metadata + skill = Get-ReportSkillStats -Manifest $Manifest -IterationPath $IterationPath + evals = @($evals) + benchmark = $Benchmark + } + + $templatePath = Join-Path (Split-Path -Parent $PSCommandPath) 'eval-report-template.html' + if (-not (Test-Path -LiteralPath $templatePath)) { + throw "Missing first-party eval report template '$templatePath'." + } + $template = [System.IO.File]::ReadAllText($templatePath, $utf8NoBom) + $dataJson = ($reportData | ConvertTo-Json -Depth 100 -Compress) + $dataJson = $dataJson -replace '(?i)', [string]$benchmark.metadata.executor_model) +Write-TextFile -Path $benchmarkMarkdownOutputPath -Content $benchmarkMarkdown + +$htmlOutputPath = if ([string]::IsNullOrWhiteSpace($OutputPath)) { Join-Path $iterationPath 'report.html' } else { $OutputPath } +$upstreamHtmlOutputPath = Join-Path $iterationPath 'skill-creator-report.html' +$viewerArguments = @( + $workspacePath, + '--skill-name', [string]$manifest.skill_name, + '--benchmark', $benchmarkOutputPath, + '--static', $upstreamHtmlOutputPath +) +Invoke-PythonScript -PythonCommand $pythonCommand -ScriptPath $viewerPath -Arguments $viewerArguments + +Write-FirstPartyReport -Manifest $manifest -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark + +Write-Host "Anthropic skill-creator tools: $skillCreatorPathResolved" +Write-Host "Anthropic viewer template: $viewerTemplatePath" +Write-Host "Wrote $benchmarkOutputPath" +Write-Host "Wrote $benchmarkMarkdownOutputPath" +Write-Host "Wrote $upstreamHtmlOutputPath" +Write-Host "Wrote $htmlOutputPath" diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 new file mode 100644 index 0000000..c73787a --- /dev/null +++ b/scripts/prepare-skill-evals.ps1 @@ -0,0 +1,1871 @@ +<# +.SYNOPSIS + Prepares portable with-skill and baseline eval prompts for a repo-managed skill, and collects externally produced results. + +.DESCRIPTION + This script computes and prints. It never executes a prompt, never spawns an agent, and never calls a model. + It turns skills//evals/evals.json into a paste-ready evaluation package that a human can run in whatever + harness, provider, and model they choose, then validates the results that come back. + + Prepare mode writes one directory per eval. The grading key and result stubs stay at the eval-case level, outside + the two hermetic run directories a worker actually sees: + eval-metadata.json id, name, prompt, expected output, assertions, fixtures, hashes, assumptions + results/ one prefilled result stub per configuration + with_skill/ a hermetic run: prompt.md, run.json, repo/ (materialized fixtures), home/, skill// + without_skill/ the same run without any skill/ directory and no skill instructions + + Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an isolated profile, and skill/ + (with_skill only) holds the candidate skill revision. run.json is a harness-neutral contract naming only paths inside + the run directory. Preparation validates the isolation invariants and fails early if a package would let a baseline + reach the skill, let a worker reach the source repository, or stage mismatched fixtures. + + Collect mode reads a prepared package plus whatever result files were filled in, validates them, and invokes the + packaged Anthropic skill-creator aggregator and static eval viewer after writing a deterministic comparison. The + selected external evaluator owns model-backed execution and grading; nothing in this repository launches a model. + +.PARAMETER Skill + Name of the repo-managed skill under skills/. + +.PARAMETER Eval + Optional eval ids to include. Defaults to every eval in the skill's evals.json. + +.PARAMETER Iteration + Iteration number to write. Defaults to the next unused iteration in the workspace. + +.PARAMETER OutputRoot + Workspace root. Defaults to .bot/-workspace. A path inside this repository must stay under .bot/. + +.PARAMETER MaxInlineBytes + Budget for inlining referenced skill resources beyond SKILL.md, which is always inlined. Anything over budget is + bundled under skill/ and listed in the prompt instead. + +.PARAMETER Force + Overwrite an existing iteration directory. + +.PARAMETER Changed + Prepares a package for every repo-managed skill this branch changed, including uncommitted work. This is the form + the eval completion gate uses after adding or modifying a skill. + +.PARAMETER Base + Base ref for -Changed. Defaults to origin/main, then main, then the working tree alone. + +.PARAMETER CollectResults + Path to a prepared iteration directory. Validates the result files in it, invokes the packaged Anthropic + skill-creator aggregator and static viewer, and writes comparison.md, benchmark.json, benchmark.md, the first-party + side-by-side report.html, and the exact upstream skill-creator-report.html compatibility artifact. + This is the fallback for results that were not finalized by the external evaluator. + +.EXAMPLE + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-strong-name-signing + +.EXAMPLE + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed + +.EXAMPLE + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults $env:TEMP/dotnet-strong-name-signing-workspace/iteration-1 +#> +[CmdletBinding(DefaultParameterSetName = 'Prepare')] +param( + [Parameter(ParameterSetName = 'Prepare', Mandatory = $true, Position = 0)] + [string]$Skill, + + [Parameter(ParameterSetName = 'Prepare')] + [int[]]$Eval, + + [Parameter(ParameterSetName = 'Prepare')] + [int]$Iteration, + + [Parameter(ParameterSetName = 'Changed', Mandatory = $true)] + [switch]$Changed, + + [Parameter(ParameterSetName = 'Changed')] + [string]$Base, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$OutputRoot, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [int]$MaxInlineBytes = 120000, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [switch]$Force, + + [Parameter(ParameterSetName = 'Collect', Mandatory = $true)] + [string]$CollectResults +) + +$ErrorActionPreference = 'Stop' + +Set-StrictMode -Version Latest + +# Captured at script scope: $PSBoundParameters inside a function describes that function, not this script. +$scriptBoundParameters = $PSBoundParameters + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +[Console]::OutputEncoding = $utf8NoBom +$OutputEncoding = $utf8NoBom + +$packageSchema = 'codebeltnet/agentic/eval-package/2' +$metadataSchema = 'codebeltnet/agentic/eval-metadata/2' +$resultSchema = 'codebeltnet/agentic/eval-result/2' +$runSchema = 'codebeltnet/agentic/eval-run/1' +$maxFixtureInlineBytes = 32768 + +# A materialized run is hermetic: the harness treats the run directory as the worker's sandbox root, mounts repo/ as +# the working directory and home/ as the isolated user profile, and exposes skill/ only for a with_skill run. Nothing +# else in the package - the grading key, the paired run, other evals, or results - lives inside a run directory, so a +# worker confined to its run directory cannot reach any of it. +$runDirectoryNames = [ordered]@{ + Working = 'repo' + Home = 'home' + Skill = 'skill' + Prompt = 'prompt.md' + Run = 'run.json' +} +$reportToolRelativePath = 'tools/generate-eval-report.ps1' +$skillCreatorToolRelativePath = 'tools/skill-creator' +$skillCreatorEvalFiles = @( + 'LICENSE.txt', + 'agents/grader.md', + 'agents/comparator.md', + 'agents/analyzer.md', + 'references/schemas.md', + 'scripts/aggregate_benchmark.py', + 'eval-viewer/generate_review.py', + 'eval-viewer/viewer.html' +) + +# These directory names are generated build state or harness bookkeeping. They must never be staged into a run's +# repository, and their presence (other than an intentional .git) means a fixture leaked build output. +$forbiddenFixtureSegments = @('bin', 'obj', '.vs', '.bot', '__pycache__', 'BenchmarkDotNet.Artifacts', 'TestResults') + +$responseContract = @' +# Response contract + +Respond in a single message. + +- Do the work in that message. If the task produces or changes files, include every file path with its final content in fenced code blocks, and also write them to disk when the environment allows it. +- If you would normally pause and ask before acting, say what you would ask and why, then stop there. That pause is a valid response. +- State any assumption you had to make instead of waiting for an answer. +'@ + +$withSkillPreamble = @' +# Operating instructions + +The instructions below are a skill: reusable operating instructions that a capable agent loads before doing this kind of work. Follow them for the task in this message. They are reproduced here in full, so you do not need to load anything else. +'@ + +$withoutSkillPreamble = @' +# Operating instructions + +You have no special instructions for this task beyond your normal capabilities. Solve the task in this message the way you normally would. +'@ + +# Identical for both configurations, and placed before the task so it never disturbs the task-and-inputs invariant that +# a with_skill and a without_skill prompt share. It reinforces, in prose, the boundary the harness enforces for real: +# operate on the staged files, not on anything discovered elsewhere on the machine. +$workingEnvironmentSection = @' +# Working environment + +Your working directory is a repository that was staged for this task. Treat it as the project root. The files under it are real, complete, and the only source of truth. Read and edit those files directly. + +Do not look for the project anywhere else on the machine, and do not reconstruct it from the text of this prompt. This is a disposable copy prepared for a single run. Your home and configuration directories are isolated to this run as well, so anything you install, configure, or discover stays local to it. Work only inside your run package; nothing outside it is part of this task. +'@ + +function Get-RepoRoot { + return (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +} + +function Write-Utf8File { + param( + [string]$Path, + [string]$Content + ) + + $directory = Split-Path -Parent $Path + if (-not (Test-Path -LiteralPath $directory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + + # Package files are portable artifacts that get pasted between machines and harnesses, so they are written with + # LF endings regardless of the platform that generated them. + $normalized = $Content -replace "`r`n", "`n" -replace "`r", "`n" + [System.IO.File]::WriteAllText($Path, $normalized, [System.Text.UTF8Encoding]::new($false)) +} + +function ConvertTo-JsonFile { + param( + [string]$Path, + [object]$Value + ) + + Write-Utf8File -Path $Path -Content (($Value | ConvertTo-Json -Depth 12) + [Environment]::NewLine) +} + +function Test-IsBinaryFile { + param([string]$Path) + + $stream = [System.IO.File]::OpenRead($Path) + try { + $buffer = [byte[]]::new(8000) + $read = $stream.Read($buffer, 0, $buffer.Length) + for ($index = 0; $index -lt $read; $index++) { + if ($buffer[$index] -eq 0) { + return $true + } + } + } finally { + $stream.Dispose() + } + + return $false +} + +function Get-Fence { + param([string]$Content) + + $longest = 0 + foreach ($match in [regex]::Matches($Content, '`+')) { + if ($match.Length -gt $longest) { + $longest = $match.Length + } + } + + return ('`' * [Math]::Max(3, $longest + 1)) +} + +function Get-FenceLanguage { + param([string]$Path) + + switch ([System.IO.Path]::GetExtension($Path).ToLowerInvariant()) { + '.cs' { 'csharp' } + '.csproj' { 'xml' } + '.props' { 'xml' } + '.targets' { 'xml' } + '.slnx' { 'xml' } + '.xml' { 'xml' } + '.cshtml' { 'html' } + '.razor' { 'html' } + '.html' { 'html' } + '.css' { 'css' } + '.js' { 'javascript' } + '.json' { 'json' } + '.md' { 'markdown' } + '.ps1' { 'powershell' } + '.psm1' { 'powershell' } + '.py' { 'python' } + '.sh' { 'bash' } + '.yml' { 'yaml' } + '.yaml' { 'yaml' } + default { 'text' } + } +} + +function Get-RelativePath { + param( + [string]$BasePath, + [string]$FullPath + ) + + $baseFull = ([System.IO.Path]::GetFullPath($BasePath)).TrimEnd('\', '/') + $targetFull = [System.IO.Path]::GetFullPath($FullPath) + if (-not $targetFull.StartsWith($baseFull, [System.StringComparison]::OrdinalIgnoreCase)) { + throw "Path '$FullPath' is not under '$BasePath'." + } + + return ($targetFull.Substring($baseFull.Length).TrimStart('\', '/') -replace '\\', '/') +} + +function Test-IsInsidePath { + param( + [string]$BasePath, + [string]$CandidatePath + ) + + $baseFull = ([System.IO.Path]::GetFullPath($BasePath)).TrimEnd('\', '/') + $candidateFull = ([System.IO.Path]::GetFullPath($CandidatePath)).TrimEnd('\', '/') + + return $candidateFull -eq $baseFull -or $candidateFull.StartsWith($baseFull + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::OrdinalIgnoreCase) +} + +function Assert-WorkspaceLocation { + param( + [string]$RepoRoot, + [string]$WorkspaceRoot + ) + + if (-not (Test-IsInsidePath -BasePath $RepoRoot -CandidatePath $WorkspaceRoot)) { + return + } + + # Inside the repository only under .bot/, which this repository ignores. Some harnesses refuse to work outside + # the repository folder at all, and .bot/ gives them a home that git never sees. + $botRoot = Join-Path $RepoRoot '.bot' + if (-not (Test-IsInsidePath -BasePath $botRoot -CandidatePath $WorkspaceRoot)) { + throw "Eval packages inside this repository must live under .bot/. '$WorkspaceRoot' does not, so it would become part of the working tree. Use .bot/-workspace or a path outside the repository." + } + + # A .bot/ that stopped being ignored would quietly turn eval output into stageable files. + [void](git -C $RepoRoot check-ignore -q -- $WorkspaceRoot 2>$null) + if ($LASTEXITCODE -eq 1) { + throw "'$WorkspaceRoot' is inside the repository but git does not ignore it. Restore the .bot/ ignore rule before writing eval packages there." + } +} + +function Get-EvalName { + param([object]$EvalEntry) + + if ($EvalEntry.PSObject.Properties.Name -contains 'name' -and -not [string]::IsNullOrWhiteSpace([string]$EvalEntry.name)) { + $source = [string]$EvalEntry.name + } else { + $words = @(([string]$EvalEntry.prompt) -split '\s+' | Where-Object { $_ -ne '' } | Select-Object -First 8) + $source = [string]::Join(' ', $words) + } + + $slug = ($source.ToLowerInvariant() -replace '[^a-z0-9]+', '-').Trim('-') + if ($slug.Length -gt 48) { + $slug = $slug.Substring(0, 48).Trim('-') + } + if ([string]::IsNullOrWhiteSpace($slug)) { + $slug = 'eval' + } + + return ('eval-{0:d2}-{1}' -f [int]$EvalEntry.id, $slug) +} + +function Get-SkillFileInventory { + param( + [string]$SkillDirectory, + [string]$SkillBody, + [int]$Budget + ) + + $files = Get-ChildItem -LiteralPath $SkillDirectory -Recurse -File -Force | + ForEach-Object { Get-RelativePath -BasePath $SkillDirectory -FullPath $_.FullName } | + Where-Object { + $_ -ne 'SKILL.md' -and + -not $_.StartsWith('evals/') -and + $_ -notmatch '(^|/)(bin|obj)/' -and + $_ -notmatch '(^|/)__pycache__/' + } | + Sort-Object + + $inventory = foreach ($relativePath in $files) { + $fullPath = Join-Path $SkillDirectory $relativePath + $index = $SkillBody.IndexOf($relativePath, [System.StringComparison]::Ordinal) + [pscustomobject]@{ + Path = $relativePath + FullPath = $fullPath + Bytes = (Get-Item -LiteralPath $fullPath).Length + Referenced = $index -ge 0 + Order = if ($index -ge 0) { $index } else { [int]::MaxValue } + } + } + + $inventory = @($inventory | Sort-Object Order, Path) + + $inlined = [System.Collections.Generic.List[object]]::new() + $bundled = [System.Collections.Generic.List[object]]::new() + $used = 0 + + foreach ($item in $inventory) { + $extension = [System.IO.Path]::GetExtension($item.Path).ToLowerInvariant() + $isInlineCandidate = $item.Referenced -and @('.md', '.txt') -contains $extension -and -not (Test-IsBinaryFile -Path $item.FullPath) + + if ($isInlineCandidate -and ($used + $item.Bytes) -le $Budget) { + $used += $item.Bytes + $inlined.Add($item) + } else { + $bundled.Add($item) + } + } + + return [pscustomobject]@{ + Inlined = @($inlined) + Bundled = @($bundled) + InlinedBytes = $used + } +} + +function New-SkillInstructionSection { + param( + [string]$SkillName, + [string]$SkillBody, + [object]$Inventory + ) + + $builder = [System.Text.StringBuilder]::new() + [void]$builder.AppendLine($withSkillPreamble) + [void]$builder.AppendLine() + [void]$builder.AppendLine("## Skill: $SkillName") + [void]$builder.AppendLine() + [void]$builder.AppendLine($SkillBody.Trim()) + [void]$builder.AppendLine() + + foreach ($item in $Inventory.Inlined) { + $resourceText = ([System.IO.File]::ReadAllText($item.FullPath, [System.Text.UTF8Encoding]::new($false))).TrimEnd() + $fence = Get-Fence -Content $resourceText + [void]$builder.AppendLine("## Skill resource: $($item.Path)") + [void]$builder.AppendLine() + [void]$builder.AppendLine($fence + 'markdown') + [void]$builder.AppendLine($resourceText) + [void]$builder.AppendLine($fence) + [void]$builder.AppendLine() + } + + if ($Inventory.Bundled.Count -gt 0) { + [void]$builder.AppendLine('## Skill resources that are not inlined') + [void]$builder.AppendLine() + [void]$builder.AppendLine("These files belong to the skill but are too large, not text, or not referenced from its main instructions. The complete skill tree, including these, is staged in your run package under ``skill/$SkillName/`` (a sibling of your working directory). Read them from there when your environment allows it; otherwise work from the instructions above and say what you could not reach.") + [void]$builder.AppendLine() + foreach ($item in $Inventory.Bundled) { + [void]$builder.AppendLine("- ``skill/$SkillName/$($item.Path)`` ($($item.Bytes) bytes)") + } + [void]$builder.AppendLine() + } + + return $builder.ToString().TrimEnd() +} + +function New-InputFilesSection { + param([object[]]$Fixtures) + + if (@($Fixtures).Count -eq 0) { + return $null + } + + $builder = [System.Text.StringBuilder]::new() + [void]$builder.AppendLine('# Input files') + [void]$builder.AppendLine() + [void]$builder.AppendLine('These files already exist as real files in your working directory, at the paths shown. They are the input for the task. Read and edit them there; the copies below are only for reference.') + [void]$builder.AppendLine() + + foreach ($fixture in $Fixtures) { + [void]$builder.AppendLine("## ``$($fixture.RepoRelative)``") + [void]$builder.AppendLine() + if ($fixture.Inlined) { + $fixtureText = $fixture.Content.TrimEnd() + $fence = Get-Fence -Content $fixtureText + [void]$builder.AppendLine($fence + $fixture.Language) + [void]$builder.AppendLine($fixtureText) + [void]$builder.AppendLine($fence) + } else { + [void]$builder.AppendLine("Present in your working directory at ``$($fixture.RepoRelative)`` ($($fixture.Bytes) bytes, $($fixture.SkipReason)); read it there.") + } + [void]$builder.AppendLine() + } + + return $builder.ToString().TrimEnd() +} + +function New-PromptDocument { + param( + [object]$EvalEntry, + [string]$InstructionSection, + [string]$InputFilesSection + ) + + $sections = [System.Collections.Generic.List[string]]::new() + $sections.Add($InstructionSection) + $sections.Add($workingEnvironmentSection) + $sections.Add("# Task`n`n$(([string]$EvalEntry.prompt).Trim())") + if (-not [string]::IsNullOrWhiteSpace($InputFilesSection)) { + $sections.Add($InputFilesSection) + } + $sections.Add($responseContract) + + return ([string]::Join("`n`n", $sections)).TrimEnd() + [Environment]::NewLine +} + +function New-ResultStub { + param( + [string]$SkillName, + [int]$IterationNumber, + [object]$EvalEntry, + [string]$EvalName, + [string]$Configuration, + [string[]]$Assertions + ) + + $grading = foreach ($assertion in $Assertions) { + [ordered]@{ + text = $assertion + passed = $null + evidence = '' + } + } + + return [ordered]@{ + schema = $resultSchema + skill_name = $SkillName + iteration = $IterationNumber + eval_id = [int]$EvalEntry.id + eval_name = $EvalName + configuration = $Configuration + model = '' + provider = '' + harness = '' + executed_utc = '' + output = '' + output_files = @() + transcript = '' + shell_commands = @() + files_read = @() + files_written = @() + stdout = '' + stderr = '' + exit_status = $null + duration_seconds = $null + total_tokens = $null + tool_calls = $null + turns = $null + base_input_tokens = $null + output_tokens = $null + cache_read_tokens = $null + cache_write_tokens = $null + cache_write_1h_tokens = $null + estimated_cost_usd = $null + model_effort = '' + isolation = [ordered]@{ + fresh_context = $null + isolated_home = $null + isolated_cwd = $null + filesystem_sandbox = $null + candidate_skill_exposed = $null + transcript_captured = $null + } + grading = @($grading) + notes = '' + } +} + +function Get-JsonProperty { + param( + [object]$Object, + [string]$Name, + [object]$Default = $null + ) + + if ($null -ne $Object -and $Object.PSObject.Properties.Name -contains $Name -and $null -ne $Object.$Name) { + return $Object.$Name + } + + return $Default +} + +# Render a run's self-reported isolation guarantees as a compact Y/N/? line. A missing object reads as "not reported", +# which tells the grader the harness did not confirm any boundary and that process-dependent assertions are suspect. +function Format-IsolationReport { + param([object]$Isolation) + + if ($null -eq $Isolation) { + return 'not reported' + } + + $flags = [ordered]@{ + fresh = 'fresh_context' + home = 'isolated_home' + cwd = 'isolated_cwd' + fs = 'filesystem_sandbox' + skill = 'candidate_skill_exposed' + tx = 'transcript_captured' + } + $parts = foreach ($key in $flags.Keys) { + $value = Get-JsonProperty -Object $Isolation -Name $flags[$key] + $mark = if ($null -eq $value) { '?' } elseif ([bool]$value) { 'Y' } else { 'N' } + "$key=$mark" + } + + return ($parts -join ' ') +} + +function Get-Assertions { + param([object]$EvalEntry) + + if ($EvalEntry.PSObject.Properties.Name -contains 'expectations' -and $null -ne $EvalEntry.expectations) { + return @($EvalEntry.expectations | ForEach-Object { [string]$_ }) + } + + return @() +} + +function Get-Sha256Hex { + param([byte[]]$Bytes) + + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + return ([System.BitConverter]::ToString($sha.ComputeHash($Bytes)) -replace '-', '').ToLowerInvariant() + } finally { + $sha.Dispose() + } +} + +function Get-FileSha256 { + param([string]$Path) + + return Get-Sha256Hex -Bytes ([System.IO.File]::ReadAllBytes($Path)) +} + +# A stable fingerprint of a directory tree: every file's forward-slashed relative path and its content hash, sorted, +# hashed again. Two trees with byte-identical files produce the same value regardless of enumeration order or platform. +function Get-TreeHash { + param( + [string]$Root, + [string[]]$ExcludeSegments = @() + ) + + if (-not (Test-Path -LiteralPath $Root)) { + return $null + } + + $entries = [System.Collections.Generic.List[string]]::new() + foreach ($file in (Get-ChildItem -LiteralPath $Root -Recurse -File -Force | Sort-Object FullName)) { + $relative = Get-RelativePath -BasePath $Root -FullPath $file.FullName + $segments = $relative.Split('/') + if ($ExcludeSegments.Count -gt 0 -and (@($segments | Where-Object { $ExcludeSegments -contains $_ }).Count -gt 0)) { + continue + } + $entries.Add("$relative`:$(Get-FileSha256 -Path $file.FullName)") + } + + $joined = [string]::Join("`n", @($entries | Sort-Object)) + return Get-Sha256Hex -Bytes ([System.Text.Encoding]::UTF8.GetBytes($joined)) +} + +function Get-EvalWorkspaceOption { + param([object]$EvalEntry) + + $wantsGit = $false + if ($EvalEntry.PSObject.Properties.Name -contains 'workspace' -and $null -ne $EvalEntry.workspace) { + $workspace = $EvalEntry.workspace + if ($workspace.PSObject.Properties.Name -contains 'git' -and $null -ne $workspace.git) { + $wantsGit = [bool]$workspace.git + } + } + + return [pscustomobject]@{ + Git = $wantsGit + } +} + +# The fixtures for one eval share a scenario directory under evals/files/ (for example evals/files/zero-config/...). That +# scenario directory is the repository root the worker should see, so it is stripped when a fixture is materialized: +# evals/files/zero-config/src/App.cs becomes repo/src/App.cs. Flat fixtures placed directly under evals/files/ (a single +# document, say) keep their own name at the repository root. +function Resolve-FixtureLayout { + param([string[]]$FixturePaths) + + $normalized = @($FixturePaths | ForEach-Object { ([string]$_).Trim() -replace '\\', '/' } | Where-Object { $_ -ne '' }) + $underFiles = @(foreach ($path in $normalized) { + if ($path -notmatch '^evals/files/.+') { + throw "Fixture '$path' must live under evals/files/." + } + $path.Substring('evals/files/'.Length) + }) + + $firstSegments = @($underFiles | ForEach-Object { ($_ -split '/')[0] } | Sort-Object -Unique) + $scenario = $null + if ($firstSegments.Count -eq 1) { + $candidate = $firstSegments[0] + # A shared first segment is the scenario root only when it is a directory, meaning at least one fixture has a + # path below it. A lone file such as report.md keeps its name at the repository root instead. + if (@($underFiles | Where-Object { $_ -like "$candidate/*" }).Count -gt 0) { + $scenario = $candidate + } + } + + $map = [System.Collections.Generic.List[object]]::new() + for ($index = 0; $index -lt $normalized.Count; $index++) { + $evalRelative = $underFiles[$index] + $repoRelative = if ($null -ne $scenario -and $evalRelative -like "$scenario/*") { + $evalRelative.Substring($scenario.Length + 1) + } else { + $evalRelative + } + if ([string]::IsNullOrWhiteSpace($repoRelative)) { + throw "Fixture '$($normalized[$index])' resolves to an empty repository path." + } + $map.Add([pscustomobject]@{ + EvalPath = $normalized[$index] + RepoRelative = $repoRelative + }) + } + + return [pscustomobject]@{ + Scenario = $scenario + Files = @($map) + } +} + +function Assert-RepoRelativeIsSafe { + param([string]$RepoRelative) + + $segments = $RepoRelative.Split('/') + if ($segments -contains '..') { + throw "Fixture path '$RepoRelative' escapes the repository root." + } + foreach ($segment in $segments) { + if ($forbiddenFixtureSegments -contains $segment) { + throw "Fixture path '$RepoRelative' includes generated build state ('$segment'); eval fixtures must not carry $($forbiddenFixtureSegments -join ', ')." + } + } +} + +# Copy an eval's fixtures into a run's repo/ as real files, preserving structure and returning the repo-relative paths so +# the manifest and run.json can describe exactly what the worker received. +function Copy-FixtureRepo { + param( + [string]$SkillDirectory, + [object]$Layout, + [string]$RepoDirectory, + [int]$EvalId + ) + + New-Item -ItemType Directory -Path $RepoDirectory -Force | Out-Null + foreach ($file in $Layout.Files) { + Assert-RepoRelativeIsSafe -RepoRelative $file.RepoRelative + $sourcePath = Join-Path $SkillDirectory ($file.EvalPath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $sourcePath)) { + throw "Missing fixture 'skills/*/$($file.EvalPath)' referenced by eval $EvalId." + } + $destination = Join-Path $RepoDirectory ($file.RepoRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destinationDirectory = Split-Path -Parent $destination + if (-not (Test-Path -LiteralPath $destinationDirectory)) { + New-Item -ItemType Directory -Path $destinationDirectory -Force | Out-Null + } + Copy-Item -LiteralPath $sourcePath -Destination $destination -Force + } +} + +# Stage a real, disposable git repository so tools that probe for a repository root or derive a version from git history +# (MinVer, Nerdbank.GitVersioning, SourceLink) behave exactly as they do on a developer's machine. Fixed identity and +# timestamps keep the two paired runs byte-identical; nothing is written to the caller's global or local git config. +function Initialize-GitWorkspace { + param([string]$RepoDirectory) + + $identity = @( + '-c', 'user.name=Eval Harness', + '-c', 'user.email=eval-harness@localhost', + '-c', 'commit.gpgsign=false', + '-c', 'core.autocrlf=false' + ) + $env:GIT_AUTHOR_DATE = '2020-01-01T00:00:00Z' + $env:GIT_COMMITTER_DATE = '2020-01-01T00:00:00Z' + try { + & git @identity init -b main --quiet -- $RepoDirectory 2>$null | Out-Null + if ($LASTEXITCODE -ne 0) { + & git @identity init --quiet -- $RepoDirectory 2>$null | Out-Null + } + if ($LASTEXITCODE -ne 0) { + throw "git init failed while staging a workspace at '$RepoDirectory'." + } + & git @identity -C $RepoDirectory add -A 2>$null | Out-Null + & git @identity -C $RepoDirectory commit -m 'Staged eval workspace' --quiet 2>$null | Out-Null + if ($LASTEXITCODE -ne 0) { + throw "git commit failed while staging a workspace at '$RepoDirectory'." + } + & git @identity -C $RepoDirectory tag 'v1.0.0' 2>$null | Out-Null + } finally { + Remove-Item Env:GIT_AUTHOR_DATE -ErrorAction SilentlyContinue + Remove-Item Env:GIT_COMMITTER_DATE -ErrorAction SilentlyContinue + } +} + +# Stage the exact candidate skill revision the worker is meant to evaluate. The whole tree ships (minus evals and build +# output) so the SKILL.md and everything it references - scripts, references, assets - are present without any fallback +# to a globally installed copy. +function Copy-SkillTree { + param( + [string]$SkillDirectory, + [string]$DestinationSkillRoot + ) + + New-Item -ItemType Directory -Path $DestinationSkillRoot -Force | Out-Null + $files = Get-ChildItem -LiteralPath $SkillDirectory -Recurse -File -Force | + ForEach-Object { Get-RelativePath -BasePath $SkillDirectory -FullPath $_.FullName } | + Where-Object { + -not $_.StartsWith('evals/') -and + $_ -notmatch '(^|/)(bin|obj)/' -and + $_ -notmatch '(^|/)__pycache__/' + } | + Sort-Object + + foreach ($relative in $files) { + $sourcePath = Join-Path $SkillDirectory ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destination = Join-Path $DestinationSkillRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destinationDirectory = Split-Path -Parent $destination + if (-not (Test-Path -LiteralPath $destinationDirectory)) { + New-Item -ItemType Directory -Path $destinationDirectory -Force | Out-Null + } + Copy-Item -LiteralPath $sourcePath -Destination $destination -Force + } + + return @($files) +} + +# Carry the report adapter and the exact upstream skill-creator grading, aggregation, and viewer assets with the +# package. The adapter never renders HTML itself: it stages our portable result schema into skill-creator's workspace +# contract, then invokes aggregate_benchmark.py and eval-viewer's generate_review.py --static. +function Copy-ReportTool { + param( + [string]$RepoRoot, + [string]$IterationDirectory + ) + + $source = Join-Path (Join-Path $RepoRoot 'scripts') 'generate-eval-report.ps1' + if (-not (Test-Path -LiteralPath $source)) { + throw "Missing report tool '$source'." + } + + $destination = Join-Path $IterationDirectory $reportToolRelativePath + $destinationDirectory = Split-Path -Parent $destination + New-Item -ItemType Directory -Path $destinationDirectory -Force | Out-Null + Copy-Item -LiteralPath $source -Destination $destination -Force + + $template = Join-Path (Split-Path -Parent $source) 'eval-report-template.html' + if (-not (Test-Path -LiteralPath $template)) { + throw "Missing first-party eval report template '$template'." + } + Copy-Item -LiteralPath $template -Destination (Join-Path $destinationDirectory 'eval-report-template.html') -Force + return $destination +} + +function Resolve-SkillCreatorSourcePath { + param([string]$RequestedPath) + + $candidates = [System.Collections.Generic.List[string]]::new() + if (-not [string]::IsNullOrWhiteSpace($RequestedPath)) { + $candidates.Add($RequestedPath) + } + + if (-not [string]::IsNullOrWhiteSpace($env:SKILL_CREATOR_PATH)) { + $candidates.Add($env:SKILL_CREATOR_PATH) + } + + if (-not [string]::IsNullOrWhiteSpace($env:USERPROFILE)) { + $candidates.Add((Join-Path $env:USERPROFILE '.agents/skills/skill-creator')) + $candidates.Add((Join-Path $env:USERPROFILE '.claude/skills/skill-creator')) + $candidates.Add((Join-Path $env:USERPROFILE '.gemini/antigravity-cli/skills/skill-creator')) + } + + foreach ($candidate in $candidates) { + $missing = @($skillCreatorEvalFiles | Where-Object { -not (Test-Path -LiteralPath (Join-Path $candidate $_)) }) + if ($missing.Count -eq 0) { + return (Resolve-Path -LiteralPath $candidate).Path + } + } + + throw "Anthropic skill-creator eval assets were not found. Install skill-creator or set SKILL_CREATOR_PATH before preparing an eval package. Required files: $($skillCreatorEvalFiles -join ', ')." +} + +function Copy-SkillCreatorEvalTools { + param( + [string]$IterationDirectory, + [string]$SourcePath + ) + + $destinationRoot = Join-Path $IterationDirectory $skillCreatorToolRelativePath + foreach ($relative in $skillCreatorEvalFiles) { + $source = Join-Path $SourcePath ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destination = Join-Path $destinationRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destinationDirectory = Split-Path -Parent $destination + New-Item -ItemType Directory -Path $destinationDirectory -Force | Out-Null + Copy-Item -LiteralPath $source -Destination $destination -Force + } + + return $destinationRoot +} + +# The candidate skill's fingerprint, computed from the source over exactly the files Copy-SkillTree stages. The staged +# copy in each with_skill run must reproduce this value, which is how preparation proves the worker received the +# revision under development rather than a globally installed one. +function Get-CandidateSkillHash { + param([string]$SkillDirectory) + + $files = Get-ChildItem -LiteralPath $SkillDirectory -Recurse -File -Force | + ForEach-Object { Get-RelativePath -BasePath $SkillDirectory -FullPath $_.FullName } | + Where-Object { + -not $_.StartsWith('evals/') -and + $_ -notmatch '(^|/)(bin|obj)/' -and + $_ -notmatch '(^|/)__pycache__/' + } | + Sort-Object + + $entries = foreach ($relative in $files) { + $fullPath = Join-Path $SkillDirectory ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + "$relative`:$(Get-FileSha256 -Path $fullPath)" + } + + $joined = [string]::Join("`n", @($entries | Sort-Object)) + return Get-Sha256Hex -Bytes ([System.Text.Encoding]::UTF8.GetBytes($joined)) +} + +function New-RunManifest { + param( + [string]$SkillName, + [int]$IterationNumber, + [object]$EvalEntry, + [string]$EvalName, + [string]$Configuration, + [string[]]$RepoFiles, + [string]$FixtureHash, + [string]$SkillHash, + [bool]$GitWorkspace + ) + + $skillDirectory = if ($Configuration -eq 'with_skill') { "$($runDirectoryNames.Skill)/$SkillName" } else { $null } + + return [ordered]@{ + schema = $runSchema + evalId = [int]$EvalEntry.id + evalName = $EvalName + skillName = if ($Configuration -eq 'with_skill') { $SkillName } else { $null } + iteration = $IterationNumber + mode = $Configuration + promptFile = $runDirectoryNames.Prompt + workingDirectory = $runDirectoryNames.Working + homeDirectory = $runDirectoryNames.Home + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $GitWorkspace + inputFiles = @($RepoFiles) + fixtureHash = $FixtureHash + skillHash = if ($Configuration -eq 'with_skill') { $SkillHash } else { $null } + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = $runDirectoryNames.Working + homeDirectory = $runDirectoryNames.Home + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } +} + +# Fail package generation the moment a run violates an isolation invariant, so a contaminated package never reaches a +# harness. These checks operate on the materialized run directories, not on prose. +function Assert-RunIsolation { + param( + [string]$EvalName, + [string]$SkillName, + [string]$EvalCaseDirectory, + [string]$SkillHash, + [bool]$GitWorkspace + ) + + $withSkillDir = Join-Path $EvalCaseDirectory 'with_skill' + $withoutSkillDir = Join-Path $EvalCaseDirectory 'without_skill' + + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDir = Join-Path $EvalCaseDirectory $configuration + $repoDir = Join-Path $runDir $runDirectoryNames.Working + $homeDir = Join-Path $runDir $runDirectoryNames.Home + $promptPath = Join-Path $runDir $runDirectoryNames.Prompt + $runJsonPath = Join-Path $runDir $runDirectoryNames.Run + + # 1. Every run has its own materialized repository, and 3. it holds no build state. + if (-not (Test-Path -LiteralPath $repoDir)) { + throw "$EvalName/$configuration is missing its materialized repo/." + } + foreach ($directory in (Get-ChildItem -LiteralPath $repoDir -Recurse -Directory -Force -ErrorAction SilentlyContinue)) { + if ($forbiddenFixtureSegments -contains $directory.Name) { + throw "$EvalName/$configuration staged generated build state under repo/ ('$($directory.Name)')." + } + if ($directory.Name -eq '.git' -and -not $GitWorkspace) { + throw "$EvalName/$configuration staged an unexpected .git directory." + } + } + if ($GitWorkspace -and -not (Test-Path -LiteralPath (Join-Path $repoDir '.git'))) { + throw "$EvalName/$configuration declared a git workspace but no .git was staged." + } + + # 5. Prompt path and manifest resolve only to staged resources, and 10. fresh context is declared. + if (-not (Test-Path -LiteralPath $promptPath)) { + throw "$EvalName/$configuration is missing prompt.md." + } + if (-not (Test-Path -LiteralPath $runJsonPath)) { + throw "$EvalName/$configuration is missing run.json." + } + if (-not (Test-Path -LiteralPath $homeDir)) { + throw "$EvalName/$configuration is missing its isolated home/." + } + + $runJsonText = [System.IO.File]::ReadAllText($runJsonPath, $utf8NoBom) + $runManifest = $runJsonText | ConvertFrom-Json + if (-not [bool]$runManifest.freshContextRequired) { + throw "$EvalName/$configuration run.json must require fresh context." + } + if (-not [bool]$runManifest.filesystemIsolationRequired -or -not [bool]$runManifest.isolatedHomeRequired) { + throw "$EvalName/$configuration run.json must require filesystem and home isolation." + } + + # 6. No run manifest references the source repository, and 7. none references a global skill install. + foreach ($needle in @('skills/', '.agents', '.claude', '.codex', '.gemini', ':\', ':/')) { + if ($configuration -eq 'with_skill' -and $needle -eq 'skills/') { + # with_skill legitimately names skill/; only reject an out-of-package skills/ reference. + continue + } + if ($runJsonText.IndexOf($needle, [System.StringComparison]::OrdinalIgnoreCase) -ge 0) { + throw "$EvalName/$configuration run.json references '$needle', which points outside the run package." + } + } + } + + # 2. with_skill contains the candidate skill; 4. required skill files are staged and match the source revision. + $stagedSkillRoot = Join-Path (Join-Path $withSkillDir $runDirectoryNames.Skill) $SkillName + if (-not (Test-Path -LiteralPath (Join-Path $stagedSkillRoot 'SKILL.md'))) { + throw "$EvalName/with_skill is missing the candidate skill (skill/$SkillName/SKILL.md)." + } + $stagedSkillHash = Get-TreeHash -Root $stagedSkillRoot + if ($stagedSkillHash -ne $SkillHash) { + throw "$EvalName/with_skill staged a candidate skill that does not match the source revision." + } + + # 3 (baseline). without_skill contains no copy of the candidate skill by any name. + $baselineSkillDir = Join-Path $withoutSkillDir $runDirectoryNames.Skill + if (Test-Path -LiteralPath $baselineSkillDir) { + throw "$EvalName/without_skill must not contain a skill/ directory." + } + if (Test-Path -LiteralPath (Join-Path $withoutSkillDir 'SKILL.md')) { + throw "$EvalName/without_skill must not contain a SKILL.md." + } + + # 9. The with_skill and without_skill repositories are otherwise identical. + $withHash = Get-TreeHash -Root (Join-Path $withSkillDir $runDirectoryNames.Working) -ExcludeSegments @('.git') + $withoutHash = Get-TreeHash -Root (Join-Path $withoutSkillDir $runDirectoryNames.Working) -ExcludeSegments @('.git') + if ($withHash -ne $withoutHash) { + throw "$EvalName repositories differ between with_skill and without_skill; the only difference must be the skill." + } +} + +function Invoke-PrepareMode { + param([string]$SkillName = $Skill) + + $Skill = $SkillName + $repoRoot = Get-RepoRoot + $skillDirectory = Join-Path (Join-Path $repoRoot 'skills') $Skill + if (-not (Test-Path -LiteralPath $skillDirectory)) { + throw "Unknown repo-managed skill '$Skill'. Expected skills/$Skill/ under $repoRoot." + } + + $skillMarkdownPath = Join-Path $skillDirectory 'SKILL.md' + if (-not (Test-Path -LiteralPath $skillMarkdownPath)) { + throw "Missing skills/$Skill/SKILL.md." + } + + $evalsPath = Join-Path (Join-Path $skillDirectory 'evals') 'evals.json' + if (-not (Test-Path -LiteralPath $evalsPath)) { + throw "Missing skills/$Skill/evals/evals.json." + } + + $evalsDocument = [System.IO.File]::ReadAllText($evalsPath, $utf8NoBom) | ConvertFrom-Json + $selectedEvals = @($evalsDocument.evals) + if ($scriptBoundParameters.ContainsKey('Eval')) { + $selectedEvals = @($selectedEvals | Where-Object { $Eval -contains [int]$_.id }) + $missing = @($Eval | Where-Object { $id = $_; -not (@($evalsDocument.evals) | Where-Object { [int]$_.id -eq $id }) }) + if ($missing.Count -gt 0) { + throw "Unknown eval id(s) for '$Skill': $($missing -join ', ')." + } + } + if ($selectedEvals.Count -eq 0) { + throw "No evals selected for '$Skill'." + } + + $workspaceRoot = if ([string]::IsNullOrWhiteSpace($OutputRoot)) { + Join-Path (Join-Path $repoRoot '.bot') "$Skill-workspace" + } else { + $OutputRoot + } + $workspaceRoot = [System.IO.Path]::GetFullPath($workspaceRoot, (Get-Location).Path) + Assert-WorkspaceLocation -RepoRoot $repoRoot -WorkspaceRoot $workspaceRoot + if (-not (Test-Path -LiteralPath $workspaceRoot)) { + New-Item -ItemType Directory -Path $workspaceRoot -Force | Out-Null + } + + $iterationNumber = if ($scriptBoundParameters.ContainsKey('Iteration')) { + $Iteration + } else { + $existing = @(Get-ChildItem -LiteralPath $workspaceRoot -Directory -Filter 'iteration-*' -ErrorAction SilentlyContinue | + ForEach-Object { if ($_.Name -match '^iteration-(\d+)$') { [int]$Matches[1] } }) + if ($existing.Count -eq 0) { 1 } else { (($existing | Measure-Object -Maximum).Maximum + 1) } + } + if ($iterationNumber -lt 1) { + throw "Iteration must be 1 or greater; got $iterationNumber." + } + + $iterationDirectory = Join-Path $workspaceRoot "iteration-$iterationNumber" + if (Test-Path -LiteralPath $iterationDirectory) { + if (-not $Force) { + throw "'$iterationDirectory' already exists. Pass -Force to replace it, or -Iteration to write a new one." + } + Remove-Item -LiteralPath $iterationDirectory -Recurse -Force + } + New-Item -ItemType Directory -Path $iterationDirectory -Force | Out-Null + [void](Copy-ReportTool -RepoRoot $repoRoot -IterationDirectory $iterationDirectory) + $skillCreatorSourcePath = Resolve-SkillCreatorSourcePath -RequestedPath $null + [void](Copy-SkillCreatorEvalTools -IterationDirectory $iterationDirectory -SourcePath $skillCreatorSourcePath) + + $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) + $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } + + $inventory = Get-SkillFileInventory -SkillDirectory $skillDirectory -SkillBody $skillBody -Budget $MaxInlineBytes + $skillHash = Get-CandidateSkillHash -SkillDirectory $skillDirectory + + $generatedUtc = [DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ') + $withSkillInstructions = New-SkillInstructionSection -SkillName $Skill -SkillBody $skillBody -Inventory $inventory + $manifestEvals = [System.Collections.Generic.List[object]]::new() + + foreach ($evalEntry in $selectedEvals) { + $evalName = Get-EvalName -EvalEntry $evalEntry + $evalDirectory = Join-Path $iterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + + $workspaceOption = Get-EvalWorkspaceOption -EvalEntry $evalEntry + + $fixturePaths = @() + if ($evalEntry.PSObject.Properties.Name -contains 'files' -and $null -ne $evalEntry.files) { + $fixturePaths = @($evalEntry.files) + } + + $fixtures = [System.Collections.Generic.List[object]]::new() + $layout = $null + if (@($fixturePaths).Count -gt 0) { + $layout = Resolve-FixtureLayout -FixturePaths $fixturePaths + foreach ($file in $layout.Files) { + $sourcePath = Join-Path $skillDirectory ($file.EvalPath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $sourcePath)) { + throw "Missing fixture 'skills/$Skill/$($file.EvalPath)' referenced by eval $($evalEntry.id)." + } + $bytes = (Get-Item -LiteralPath $sourcePath).Length + $isBinary = Test-IsBinaryFile -Path $sourcePath + $skipReason = if ($isBinary) { 'not text' } elseif ($bytes -gt $maxFixtureInlineBytes) { "over the $maxFixtureInlineBytes-byte inline cap" } else { $null } + $fixtures.Add([pscustomobject]@{ + EvalPath = $file.EvalPath + RepoRelative = $file.RepoRelative + Bytes = $bytes + Inlined = $null -eq $skipReason + SkipReason = $skipReason + Language = Get-FenceLanguage -Path $sourcePath + Content = if ($null -eq $skipReason) { [System.IO.File]::ReadAllText($sourcePath, $utf8NoBom) } else { '' } + }) + } + } + + $repoFiles = @($fixtures | ForEach-Object { $_.RepoRelative } | Sort-Object) + + # Materialize both runs. Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an + # isolated profile, and skill/ (with_skill only) holds the candidate. The grading key and results live one level + # up, outside every run directory, so a worker confined to its run directory can never reach them. + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDir = Join-Path $evalDirectory $configuration + New-Item -ItemType Directory -Path $runDir -Force | Out-Null + + $repoDir = Join-Path $runDir $runDirectoryNames.Working + if ($null -ne $layout) { + Copy-FixtureRepo -SkillDirectory $skillDirectory -Layout $layout -RepoDirectory $repoDir -EvalId ([int]$evalEntry.id) + } else { + New-Item -ItemType Directory -Path $repoDir -Force | Out-Null + } + if ($workspaceOption.Git) { + Initialize-GitWorkspace -RepoDirectory $repoDir + } + + $homeDir = Join-Path $runDir $runDirectoryNames.Home + New-Item -ItemType Directory -Path $homeDir -Force | Out-Null + Write-Utf8File -Path (Join-Path $homeDir 'README.txt') -Content "This is an isolated, deliberately empty home directory for one eval run. A harness sets HOME - and the platform-equivalent profile and config roots - here so the worker cannot see the machine's global agent configuration, skills, plugins, MCP servers, or memories.`n" + + if ($configuration -eq 'with_skill') { + $stagedSkillRoot = Join-Path (Join-Path $runDir $runDirectoryNames.Skill) $Skill + [void](Copy-SkillTree -SkillDirectory $skillDirectory -DestinationSkillRoot $stagedSkillRoot) + } + } + + # Identical between runs by construction; validated below. The .git directory is excluded because two git init + # runs would otherwise differ, while the tracked fixture content is the same. + $fixtureHash = Get-TreeHash -Root (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Working) -ExcludeSegments @('.git') + + $inputFilesSection = New-InputFilesSection -Fixtures @($fixtures) + $assertions = Get-Assertions -EvalEntry $evalEntry + + $withSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withSkillInstructions -InputFilesSection $inputFilesSection + $withoutSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withoutSkillPreamble -InputFilesSection $inputFilesSection + + Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Prompt) -Content $withSkillPrompt + Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Prompt) -Content $withoutSkillPrompt + + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $skillHash -GitWorkspace $workspaceOption.Git) + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $null -GitWorkspace $workspaceOption.Git) + + $assumptions = [System.Collections.Generic.List[string]]::new() + $assumptions.Add('Run with_skill and without_skill on the same model, same version, and same configuration. Different models measure the model, not the skill.') + $assumptions.Add('Each run is hermetic: launch a fresh worker with its run directory as the sandbox root, its repo/ as the working directory, and its home/ as the isolated profile.') + $assumptions.Add("Both runs share an identical materialized repository. Only the with_skill run exposes the candidate skill under skill/$Skill/.") + $notInlinedFixtures = @($fixtures | Where-Object { -not $_.Inlined }) + if ($notInlinedFixtures.Count -gt 0) { + $assumptions.Add("$($notInlinedFixtures.Count) input file(s) are large or binary; they are materialized in repo/ but not inlined in the prompt.") + } + if ($workspaceOption.Git) { + $assumptions.Add('This eval stages a real .git in repo/ so repository-root detection and version-deriving tools behave as on a developer machine.') + } + $assumptions.Add('The expected output and assertions in this file are the grading key. They live outside every run directory and must never reach a worker.') + + $metadata = [ordered]@{ + schema = $metadataSchema + skill_name = $Skill + iteration = $iterationNumber + eval_id = [int]$evalEntry.id + eval_name = $evalName + prompt = [string]$evalEntry.prompt + expected_output = [string]$evalEntry.expected_output + assertions = @($assertions) + fixture_hash = $fixtureHash + skill_hash = $skillHash + git_workspace = $workspaceOption.Git + input_files = @($fixtures | ForEach-Object { + [ordered]@{ + eval_path = $_.EvalPath + repo_path = $_.RepoRelative + bytes = $_.Bytes + inlined = $_.Inlined + } + }) + configurations = [ordered]@{ + with_skill = [ordered]@{ + run_directory = 'with_skill' + prompt_file = "with_skill/$($runDirectoryNames.Prompt)" + run_manifest = "with_skill/$($runDirectoryNames.Run)" + result_file = 'results/with-skill.result.json' + } + without_skill = [ordered]@{ + run_directory = 'without_skill' + prompt_file = "without_skill/$($runDirectoryNames.Prompt)" + run_manifest = "without_skill/$($runDirectoryNames.Run)" + result_file = 'results/without-skill.result.json' + } + } + assumptions = @($assumptions) + } + + ConvertTo-JsonFile -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value $metadata + ConvertTo-JsonFile -Path (Join-Path $evalDirectory 'results/with-skill.result.json') -Value (New-ResultStub -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -Assertions $assertions) + ConvertTo-JsonFile -Path (Join-Path $evalDirectory 'results/without-skill.result.json') -Value (New-ResultStub -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -Assertions $assertions) + + Assert-RunIsolation -EvalName $evalName -SkillName $Skill -EvalCaseDirectory $evalDirectory -SkillHash $skillHash -GitWorkspace $workspaceOption.Git + + $manifestEvals.Add([ordered]@{ + eval_id = [int]$evalEntry.id + eval_name = $evalName + directory = $evalName + metadata = "$evalName/eval-metadata.json" + fixture_hash = $fixtureHash + skill_hash = $skillHash + git_workspace = $workspaceOption.Git + input_files = @($repoFiles) + runs = [ordered]@{ + with_skill = [ordered]@{ + mode = 'with_skill' + directory = "$evalName/with_skill" + run_manifest = "$evalName/with_skill/$($runDirectoryNames.Run)" + prompt = "$evalName/with_skill/$($runDirectoryNames.Prompt)" + working_directory = "$evalName/with_skill/$($runDirectoryNames.Working)" + home_directory = "$evalName/with_skill/$($runDirectoryNames.Home)" + skill_directory = "$evalName/with_skill/$($runDirectoryNames.Skill)/$Skill" + result = "$evalName/results/with-skill.result.json" + } + without_skill = [ordered]@{ + mode = 'without_skill' + directory = "$evalName/without_skill" + run_manifest = "$evalName/without_skill/$($runDirectoryNames.Run)" + prompt = "$evalName/without_skill/$($runDirectoryNames.Prompt)" + working_directory = "$evalName/without_skill/$($runDirectoryNames.Working)" + home_directory = "$evalName/without_skill/$($runDirectoryNames.Home)" + skill_directory = $null + result = "$evalName/results/without-skill.result.json" + } + } + }) + } + + $manifest = [ordered]@{ + schema = $packageSchema + skill_name = $Skill + skill_source = "skills/$Skill" + iteration = $iterationNumber + generated_utc = $generatedUtc + configurations = @('with_skill', 'without_skill') + execution = 'external_handoff' + runner_prompt = 'RUN-THIS.prompt.md' + report = [ordered]@{ + tool = $reportToolRelativePath + template = 'tools/eval-report-template.html' + skill_creator = $skillCreatorToolRelativePath + aggregator = "$skillCreatorToolRelativePath/scripts/aggregate_benchmark.py" + viewer = "$skillCreatorToolRelativePath/eval-viewer/generate_review.py" + viewer_template = "$skillCreatorToolRelativePath/eval-viewer/viewer.html" + html = 'report.html' + upstream_html = 'skill-creator-report.html' + benchmark = 'benchmark.json' + benchmark_markdown = 'benchmark.md' + } + max_inline_bytes = $MaxInlineBytes + skill_hash = $skillHash + isolation = [ordered]@{ + fresh_context_required = $true + isolated_home_required = $true + isolated_cwd_required = $true + filesystem_sandbox_recommended = $true + candidate_skill_exposure = 'run_directory' + transcript_capture_requested = $true + sandbox_root = 'each run directory' + working_directory = $runDirectoryNames.Working + home_directory = $runDirectoryNames.Home + } + harness_contract = @( + 'fresh context', + 'isolated HOME/config', + 'isolated CWD', + 'filesystem sandbox', + 'candidate skill exposure', + 'transcript capture' + ) + skill_instructions = [ordered]@{ + inlined = @('SKILL.md') + @($inventory.Inlined | ForEach-Object { $_.Path }) + inlined_resource_bytes = $inventory.InlinedBytes + staged_full_tree = $true + } + evals = @($manifestEvals) + } + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'manifest.json') -Value $manifest + + Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals)) + $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' + Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals)) + + Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) hermetic run package(s)." + Write-Host "Package: $iterationDirectory" + Write-Host '' + Write-Host 'Every run is a self-contained directory: repo/ is the working tree, home/ is an isolated' + Write-Host 'profile, and skill/ (with_skill only) holds the candidate. A harness runs a worker from that' + Write-Host 'directory alone and never needs the source repository or a globally installed skill.' + Write-Host '' + Write-Host 'Hand this one file to the agent of your choice. It drives the whole package:' + Write-Host " $runnerPath" + Write-Host '' + Write-Host 'Point the harness at that path. Do not reproduce its contents in chat: a pasted copy' + Write-Host 'loses the absolute paths it depends on, and the harness then cannot find the package.' + Write-Host '' + Write-Host 'The runner makes the selected agent the evaluator, grader, and report producer. It must create' + Write-Host 'one isolated fresh worker per run, then grade the collected results and generate both report artifacts.' + Write-Host '' + Write-Host 'This script prepared prompts only. It did not run them, and nothing here will.' + Write-Host 'The selected evaluator should finish the package in one run. If it cannot write back to this package,' + Write-Host 'bring back the result objects and use the repository collector as a fallback:' + Write-Host (" pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `"$iterationDirectory`"") +} + +function New-RunnerPrompt { + param( + [string]$IterationDirectory, + [int]$IterationNumber, + [object[]]$ManifestEvals + ) + + $builder = [System.Text.StringBuilder]::new() + [void]$builder.AppendLine('# Run, grade, and report this evaluation package') + [void]$builder.AppendLine() + [void]$builder.AppendLine('START NOW. You are the evaluator, grader, and report producer for this package. Do not ask me which role to perform, whether to run the workers, or whether to continue to grading. Execute the complete workflow in this one handoff and return the finished report.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('Do not execute evaluation prompts in the current agent context. Create an isolated fresh-context worker for every run, then continue in this coordinator context with grading and report generation after all workers finish.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('Everything you need is in this directory:') + [void]$builder.AppendLine() + [void]$builder.AppendLine(" $IterationDirectory") + [void]$builder.AppendLine() + [void]$builder.AppendLine("This file is `"$(Join-Path $IterationDirectory 'RUN-THIS.prompt.md')`". Use that package path as authoritative. If the package is inaccessible, report the concrete access failure; do not ask the user to choose another workflow.") + [void]$builder.AppendLine() + [void]$builder.AppendLine("It holds $(@($ManifestEvals).Count) eval case(s), each with a `with_skill` and `without_skill` run. Every run is a self-contained directory that holds a `prompt.md`, a `run.json` contract, a `repo/` working tree, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate. The runs differ only in whether that candidate skill is present. You run the experiment, grade the completed results, and generate the report.") + [void]$builder.AppendLine() + [void]$builder.AppendLine('A person selected this harness and handed you this package. That authorizes these runs, grading, aggregation, and report generation. The repository prohibition on automated model evaluation binds repository scripts and the agent that prepared the package; it does not bind the user-directed external executor completing this specific handoff.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Execute now') + [void]$builder.AppendLine() + [void]$builder.AppendLine('This package assumes the harness can give each run a fresh context, an isolated HOME/config root, an isolated working directory, and a filesystem sandbox scoped to the run directory. A harness that cannot provide fresh, independent sessions is incompatible with these evals. Continue without asking for role clarification, and report per run which guarantees you satisfied - fresh context, isolated HOME/config, isolated CWD, filesystem sandbox, candidate skill exposure, and transcript capture.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Orchestration contract') + [void]$builder.AppendLine() + [void]$builder.AppendLine('1. For every eval case, create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`. Never reuse a worker or session between runs, between cases, or between iterations.') + [void]$builder.AppendLine('2. Launch each worker from its own run directory, which is the worker''s sandbox root. Set the working directory to that run''s `repo/`, set HOME and the platform-equivalent profile and config roots to its `home/`, and confine filesystem access to the run directory. Read the run''s `run.json` for the exact contract: `workingDirectory`, `homeDirectory`, `skillDirectory`, and the fresh-context, filesystem, and home isolation flags.') + [void]$builder.AppendLine('3. Give each worker only its `prompt.md` and the files already staged in its run directory. Do not expose this runner, `manifest.json`, any `eval-metadata.json`, `comparison.md`, result files, grading criteria, expectations, the paired run, another case''s output, or any note that an experiment is underway. All of those live outside the run directory, so keeping the worker inside it keeps them hidden.') + [void]$builder.AppendLine('4. The candidate skill is already inlined in the with_skill run''s `prompt.md` and staged under its `skill/` directory. Do not load, summarize, or add it yourself. The without_skill run carries no skill instructions and no `skill/` directory; do not expose the candidate skill to that worker by any route, including a globally installed copy.') + [void]$builder.AppendLine('5. Send each `prompt.md` unchanged as the worker''s first message. The input files are already real files in the worker''s `repo/`; the worker reads and edits them there rather than from attachments.') + [void]$builder.AppendLine('6. Use the same model, version, configuration, tools, and limits for every worker. Disable persistent memory or cross-session recall. Independent runs may execute concurrently when the selected harness and token budget allow it.') + [void]$builder.AppendLine('7. Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count. When the harness exposes them, also record the shell commands, files read and written, stdout and stderr, and exit status, and which isolation guarantees you satisfied. Record refusals, questions, and failures as results. Do not retry to improve an answer.') + [void]$builder.AppendLine('8. Work only inside this package. Do not read or modify the source repository around it. Do not begin grading until every available worker has completed or failed and its result is recorded.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('For each case in `manifest.json`, the `runs.with_skill` and `runs.without_skill` entries give each run''s directory, its `prompt`, its `run_manifest` (`run.json`), and the `result` file to write. Run the two prompts in separate workers, then overwrite the matching result file without reading its existing contents. A partial package is valid: record every completed run, continue to grading/reporting, and mark missing arms honestly instead of asking what to do next.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Result shape') + [void]$builder.AppendLine() + [void]$builder.AppendLine('```json') + [void]$builder.AppendLine('{') + [void]$builder.AppendLine(' "schema": "codebeltnet/agentic/eval-result/2",') + [void]$builder.AppendLine(" `"iteration`": $IterationNumber,") + [void]$builder.AppendLine(' "eval_id": 1,') + [void]$builder.AppendLine(' "eval_name": "the directory name",') + [void]$builder.AppendLine(' "configuration": "with_skill",') + [void]$builder.AppendLine(' "model": "the exact model id you used",') + [void]$builder.AppendLine(' "provider": "who served it",') + [void]$builder.AppendLine(' "harness": "what you are",') + [void]$builder.AppendLine(' "executed_utc": "2026-01-01T00:00:00Z",') + [void]$builder.AppendLine(' "output": "the complete response the run produced",') + [void]$builder.AppendLine(' "output_files": ["paths of any files the run wrote"],') + [void]$builder.AppendLine(' "transcript": "the complete worker transcript when the harness exposes it",') + [void]$builder.AppendLine(' "shell_commands": ["commands the run executed, when exposed"],') + [void]$builder.AppendLine(' "files_read": ["paths the run read, when exposed"],') + [void]$builder.AppendLine(' "files_written": ["paths the run wrote, when exposed"],') + [void]$builder.AppendLine(' "exit_status": 0,') + [void]$builder.AppendLine(' "duration_seconds": 12.5,') + [void]$builder.AppendLine(' "total_tokens": 1234,') + [void]$builder.AppendLine(' "tool_calls": 6,') + [void]$builder.AppendLine(' "turns": 12,') + [void]$builder.AppendLine(' "base_input_tokens": 27,') + [void]$builder.AppendLine(' "output_tokens": 3800,') + [void]$builder.AppendLine(' "cache_read_tokens": 515605,') + [void]$builder.AppendLine(' "cache_write_1h_tokens": 129582,') + [void]$builder.AppendLine(' "estimated_cost_usd": 2.27,') + [void]$builder.AppendLine(' "model_effort": "high",') + [void]$builder.AppendLine(' "isolation": {') + [void]$builder.AppendLine(' "fresh_context": true,') + [void]$builder.AppendLine(' "isolated_home": true,') + [void]$builder.AppendLine(' "isolated_cwd": true,') + [void]$builder.AppendLine(' "filesystem_sandbox": true,') + [void]$builder.AppendLine(' "candidate_skill_exposed": true,') + [void]$builder.AppendLine(' "transcript_captured": true') + [void]$builder.AppendLine(' },') + [void]$builder.AppendLine(' "grading": [],') + [void]$builder.AppendLine(' "notes": "anything that would change how this result reads"') + [void]$builder.AppendLine('}') + [void]$builder.AppendLine('```') + [void]$builder.AppendLine() + [void]$builder.AppendLine('`transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls`, the optional efficiency telemetry fields, and every `isolation` flag are optional. Include each when the harness exposes it and omit it otherwise. Never estimate a missing value. For `with_skill`, set `isolation.candidate_skill_exposed` to how the skill actually reached the worker.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('`configuration` is `with_skill` or `without_skill` and must match the prompt you ran. Read `eval_id` and `eval_name` from `manifest.json`; do not send them to the worker. Put the full model response in `output`. If it is very long, write it beside the result file and list that path in `output_files` with a summary in `output`.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('`output` is the model''s message in full, including questions, caveats, explanations, or a refusal. Tool output is evidence from the run, not a replacement for the model response. Put the full worker event history in `transcript` when the harness exposes it.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Grade and report immediately') + [void]$builder.AppendLine() + [void]$builder.AppendLine('After all available workers finish, read each eval''s `eval-metadata.json`. Only now may you read `expected_output` and `assertions`; they are the grading key and were intentionally hidden from the workers.') + [void]$builder.AppendLine('1. Grade every completed result against every assertion. Use deterministic checks for mechanical assertions and concrete output, transcript, and file evidence for process assertions. Use judgement only where the assertion is genuinely qualitative, and say so in the evidence. Never infer a tool or file action from the model''s self-report when process evidence is absent.') + [void]$builder.AppendLine('2. Write grading back into the matching result file using exactly `grading[].text`, `grading[].passed`, and `grading[].evidence`. Use `passed: null` when an assertion cannot be judged from captured evidence. Do not grade a missing run as passed.') + [void]$builder.AppendLine('The package carries Anthropic skill-creator under `tools/skill-creator`. Use `tools/skill-creator/agents/grader.md` for grading guidance, and use `tools/skill-creator/scripts/aggregate_benchmark.py` plus `tools/skill-creator/eval-viewer/generate_review.py` as the source-of-truth aggregation and review tools.') + $reportCommand = 'pwsh -NoProfile -File "' + (Join-Path $IterationDirectory $reportToolRelativePath) + '" -IterationDirectory "' + $IterationDirectory + '"' + [void]$builder.AppendLine(('3. Run the package report adapter now; do not ask the user to run a second command: ' + $reportCommand + '. It stages the recorded results into the upstream skill-creator workspace contract, invokes the exact upstream aggregator and static viewer, then writes the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root.')) + [void]$builder.AppendLine('4. If the harness can open local files, open `report.html` after it is written. Otherwise return its absolute path as the primary artifact. Do not wait for browser feedback before finishing the handoff.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('The report is the completion artifact. Do not stop after worker execution, do not return a prose-only recap, and do not ask whether grading or HTML generation is wanted.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Final handoff') + [void]$builder.AppendLine() + [void]$builder.AppendLine('The finished artifacts are the first-party paired review and the exact upstream skill-creator viewer report, not a request for another command. If you can write to the package machine, leave every result, grading field, `benchmark.json`, `benchmark.md`, `report.html`, and `skill-creator-report.html` in place. Return the absolute first-party report path, the completed/expected run count, any missing arms, the model/provider, and a concise quality summary.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('If you cannot write to the package machine, return one fenced JSON block containing every completed result object, including its `grading` array, plus the generated report as an artifact when the harness supports file handoff. Do not return separate blocks or a human summary in place of the result objects. State any missing arms and the concrete artifact-transfer limitation.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('If you cannot write to that machine - a different product, a browser, a sandbox that shares no disk with it - the results have to travel as text. End with one fenced block, and say plainly that it is meant to be pasted into the repository session as-is:') + [void]$builder.AppendLine() + [void]$builder.AppendLine('```') + [void]$builder.AppendLine('Eval results, grading, and report artifact.') + [void]$builder.AppendLine("Package: $IterationDirectory") + [void]$builder.AppendLine('Model: via , harness ') + [void]$builder.AppendLine() + [void]$builder.AppendLine('') + [void]$builder.AppendLine() + [void]$builder.AppendLine('Still unfilled: ') + [void]$builder.AppendLine('```') + [void]$builder.AppendLine() + [void]$builder.AppendLine('One block covering everything you ran, not one per case, and the outputs go in it verbatim - a summary written for a human to skim cannot be graded against assertions.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('The repository collector is only a fallback when result files were transferred without the report artifacts: `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `. It validates the returned files and invokes the same packaged skill-creator aggregator and viewer; it is not the normal next step after this prompt.') + + return $builder.ToString() +} + +function New-PackageReadme { + param( + [string]$SkillName, + [int]$IterationNumber, + [string]$IterationDirectory, + [object[]]$ManifestEvals + ) + + $builder = [System.Text.StringBuilder]::new() + [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") + [void]$builder.AppendLine() + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. You choose the harness, provider, and model; the selected external evaluator runs both configurations, grades them, and generates the report.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## What is here') + [void]$builder.AppendLine() + foreach ($entry in $ManifestEvals) { + [void]$builder.AppendLine("- ``$($entry.eval_name)/`` - eval $($entry.eval_id)") + } + [void]$builder.AppendLine() + [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two hermetic run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them.') + [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Isolation model') + [void]$builder.AppendLine() + [void]$builder.AppendLine('The package guarantees what a generator can: identical materialized repositories for both runs, the candidate skill staged only under `with_skill/skill/`, an empty isolated `home/` per run, and a `run.json` that names only paths inside the run directory. Fixture and skill hashes are recorded so you can prove what each worker received.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('The harness must supply the rest at runtime: a fresh context per run, the run directory as the working and config root (working directory `repo/`, HOME `home/`), and a filesystem sandbox that keeps the worker inside its run directory so global skills, global config, the source repository, the paired run, and the grading key stay out of reach. Prompt wording alone does not enforce this; the sandbox does.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## How to run') + [void]$builder.AppendLine() + [void]$builder.AppendLine('1. Pick one model and configuration. Use the same one for every run in this iteration.') + [void]$builder.AppendLine('2. For each eval, launch a fresh worker for `with_skill/` with its run directory as the sandbox root, `repo/` as the working directory, and `home/` as HOME. Send `prompt.md` as the first message. Read `run.json` for the contract.') + [void]$builder.AppendLine('3. Launch a second fresh worker for `without_skill/` the same way. Never reuse a worker between runs.') + [void]$builder.AppendLine('4. Save each response into the matching file under the eval''s `results/` directory, grade every completed result using the packaged `agents/grader.md` guidance, and run `tools/generate-eval-report.ps1`. The resulting `report.html` is the first-party side-by-side review; `skill-creator-report.html` is the exact upstream viewer.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('`RUN-THIS.prompt.md` turns a harness that can create isolated workers or sessions into the evaluator, grader, and report producer. It reads the package, creates one new worker per run from its run directory, keeps runner instructions and grading data out of every worker, records results, grades after collection, and invokes Anthropic skill-creator''s aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. `-CollectResults` accepts a partial iteration and reports unfilled runs as missing.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('A with_skill run on one model compared against a baseline on another measures both the model and the skill. That is not a skill-effectiveness result, so do not report it as one. If you do mix models, say so explicitly and treat the comparison as directional only.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Report artifacts') + [void]$builder.AppendLine() + [void]$builder.AppendLine('Fill in each `results/*.result.json`:') + [void]$builder.AppendLine() + [void]$builder.AppendLine('- `model`, `provider`, `harness` - what actually ran it, as specifically as you know') + [void]$builder.AppendLine('- `executed_utc` - when') + [void]$builder.AppendLine('- `output` - the produced output, or a summary plus paths in `output_files`') + [void]$builder.AppendLine('- `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls` - include the values the harness exposes; omit unavailable values rather than estimating them') + [void]$builder.AppendLine('- Optional efficiency telemetry: `turns`, `base_input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens` or `cache_write_1h_tokens`, `estimated_cost_usd`, and `model_effort`. These are shown when recorded and never inferred from totals.') + [void]$builder.AppendLine('- `isolation` - the guarantees the harness satisfied for this run; process-dependent assertions can only be graded from a run that captured the needed evidence') + [void]$builder.AppendLine('- `grading[].passed` - `true`, `false`, or `null` per assertion once the external evaluator or a deterministic script has checked it, with `evidence`') + [void]$builder.AppendLine('- `notes` - anything that would change how the result reads') + [void]$builder.AppendLine() + [void]$builder.AppendLine('The normal handoff finishes with these package-root artifacts:') + [void]$builder.AppendLine() + [void]$builder.AppendLine('```text') + [void]$builder.AppendLine('report.html first-party side-by-side review with paired outputs, grades, evidence, telemetry, and feedback') + [void]$builder.AppendLine('skill-creator-report.html exact Anthropic skill-creator static viewer for compatibility') + [void]$builder.AppendLine('benchmark.json Anthropic skill-creator machine-readable quality and runtime summary') + [void]$builder.AppendLine('benchmark.md Anthropic skill-creator human-readable benchmark summary') + [void]$builder.AppendLine('```') + [void]$builder.AppendLine() + [void]$builder.AppendLine('If the harness cannot write to the package machine, its final handoff should contain one paste-ready JSON array of completed result objects including grading, plus the report as a file artifact when supported. A prose-only recap is not sufficient.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('For transferred results without report artifacts, the repository-side fallback is `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `, which validates the files and invokes the same packaged skill-creator aggregator and viewer.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('This workspace is temporary. Do not commit it to the repository unless someone explicitly asks for a checked-in example.') + + return $builder.ToString() +} + +function Get-BaseRef { + param([string]$RepoRoot) + + if (-not [string]::IsNullOrWhiteSpace($Base)) { + [void](git -C $RepoRoot rev-parse --verify --quiet "$Base^{commit}" 2>$null) + if ($LASTEXITCODE -ne 0) { + throw "Unknown base ref '$Base'." + } + return $Base + } + + foreach ($candidate in @('origin/main', 'main')) { + [void](git -C $RepoRoot rev-parse --verify --quiet "$candidate^{commit}" 2>$null) + if ($LASTEXITCODE -eq 0) { + return $candidate + } + } + + return $null +} + +function Get-ChangedSkillNames { + param( + [string]$RepoRoot, + [string]$BaseRef + ) + + $paths = [System.Collections.Generic.List[string]]::new() + + # Uncommitted work, staged or not, including files git has never seen. + $status = git -C $RepoRoot status --porcelain --untracked-files=all -- 'skills' 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($line in @($status)) { + if ([string]::IsNullOrWhiteSpace($line)) { + continue + } + $entry = $line.Substring(2).Trim() + $arrow = $entry.IndexOf(' -> ', [System.StringComparison]::Ordinal) + if ($arrow -ge 0) { + $paths.Add($entry.Substring(0, $arrow).Trim('"')) + $paths.Add($entry.Substring($arrow + 4).Trim('"')) + } else { + $paths.Add($entry.Trim('"')) + } + } + } + + # Everything this branch changed relative to its base. + if (-not [string]::IsNullOrWhiteSpace($BaseRef)) { + $committed = git -C $RepoRoot diff --name-only "$BaseRef...HEAD" -- 'skills' 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($line in @($committed)) { + if (-not [string]::IsNullOrWhiteSpace($line)) { + $paths.Add($line.Trim('"')) + } + } + } + } + + $names = foreach ($path in $paths) { + $segments = ($path -replace '\\', '/').Split('/') + if ($segments.Length -ge 2 -and $segments[0] -eq 'skills') { + $segments[1] + } + } + + return @($names | + Sort-Object -Unique | + Where-Object { Test-Path -LiteralPath (Join-Path (Join-Path $RepoRoot 'skills') $_) }) +} + +function Invoke-ChangedMode { + $repoRoot = Get-RepoRoot + $baseRef = Get-BaseRef -RepoRoot $repoRoot + # An empty pipeline result unrolls to $null on assignment, so keep the array wrapper here. + $changedSkills = @(Get-ChangedSkillNames -RepoRoot $repoRoot -BaseRef $baseRef) + + $scope = if ([string]::IsNullOrWhiteSpace($baseRef)) { 'the working tree' } else { "$baseRef...HEAD plus the working tree" } + Write-Host "Changed repo-managed skills in $scope" + + if ($changedSkills.Count -eq 0) { + Write-Host ' none' + Write-Host '' + Write-Host 'No skill changed, so there is nothing to evaluate. The eval gate is satisfied.' + return + } + + foreach ($changedSkill in $changedSkills) { + Write-Host " $changedSkill" + } + Write-Host '' + + foreach ($changedSkill in $changedSkills) { + Invoke-PrepareMode -SkillName $changedSkill + Write-Host '' + } + + Write-Host ("Prepared eval packages for {0} changed skill(s). Hand these prompts to the user; do not run them." -f $changedSkills.Count) +} + +function Invoke-CollectMode { + if (-not (Test-Path -LiteralPath $CollectResults)) { + throw "Missing eval package directory '$CollectResults'." + } + + $iterationDirectory = (Resolve-Path -LiteralPath $CollectResults).Path + $manifestPath = Join-Path $iterationDirectory 'manifest.json' + if (-not (Test-Path -LiteralPath $manifestPath)) { + throw "'$iterationDirectory' is not a prepared eval package; manifest.json is missing." + } + + $manifest = [System.IO.File]::ReadAllText($manifestPath, $utf8NoBom) | ConvertFrom-Json + $errors = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $rows = [System.Collections.Generic.List[object]]::new() + + foreach ($entry in @($manifest.evals)) { + $evalDirectory = Join-Path $iterationDirectory $entry.directory + $metadata = [System.IO.File]::ReadAllText((Join-Path $evalDirectory 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json + $observed = @{} + + foreach ($configuration in @('with_skill', 'without_skill')) { + $fileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + $resultPath = Join-Path (Join-Path $evalDirectory 'results') $fileName + if (-not (Test-Path -LiteralPath $resultPath)) { + $warnings.Add("$($entry.eval_name)/$configuration - no result file at results/$fileName.") + continue + } + + try { + $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json + } catch { + $errors.Add("$($entry.eval_name)/$configuration - results/$fileName is not valid JSON: $($_.Exception.Message)") + continue + } + + if ([string]$result.configuration -ne $configuration) { + $errors.Add("$($entry.eval_name)/$configuration - results/$fileName declares configuration '$($result.configuration)'.") + continue + } + if ([int]$result.eval_id -ne [int]$metadata.eval_id) { + $errors.Add("$($entry.eval_name)/$configuration - results/$fileName declares eval_id $($result.eval_id) but the package says $($metadata.eval_id).") + continue + } + + $outputText = [string](Get-JsonProperty -Object $result -Name 'output' -Default '') + $outputFiles = @(Get-JsonProperty -Object $result -Name 'output_files' -Default @()) + $hasOutput = -not [string]::IsNullOrWhiteSpace($outputText) -or $outputFiles.Count -gt 0 + if (-not $hasOutput) { + $warnings.Add("$($entry.eval_name)/$configuration - not run yet (empty output and no output_files).") + continue + } + + $model = [string](Get-JsonProperty -Object $result -Name 'model' -Default '') + if ([string]::IsNullOrWhiteSpace($model)) { + $warnings.Add("$($entry.eval_name)/$configuration - no model recorded, so this arm cannot back a controlled comparison.") + } + + # A transferred or partial result may arrive without grading. Fall back to the assertion count from the + # package so the row still shows how much is left to check. + $grading = @(Get-JsonProperty -Object $result -Name 'grading' -Default @()) + $graded = @($grading | Where-Object { $null -ne (Get-JsonProperty -Object $_ -Name 'passed') }) + $passed = @($graded | Where-Object { [bool]$_.passed }).Count + $total = if ($grading.Count -gt 0) { $grading.Count } else { @($metadata.assertions).Count } + if ($graded.Count -eq 0) { + $warnings.Add("$($entry.eval_name)/$configuration - ran but nothing is graded yet; $total assertion(s) still need a deterministic check or human judgement.") + } + + $transcriptText = [string](Get-JsonProperty -Object $result -Name 'transcript' -Default '') + $shellCommands = @(Get-JsonProperty -Object $result -Name 'shell_commands' -Default @()) + $filesRead = @(Get-JsonProperty -Object $result -Name 'files_read' -Default @()) + $filesWritten = @(Get-JsonProperty -Object $result -Name 'files_written' -Default @()) + $hasProcessEvidence = (-not [string]::IsNullOrWhiteSpace($transcriptText)) -or $shellCommands.Count -gt 0 -or $filesRead.Count -gt 0 -or $filesWritten.Count -gt 0 + if (-not $hasProcessEvidence) { + $warnings.Add("$($entry.eval_name)/$configuration - no transcript or process evidence recorded; assertions about tool, shell, or file behavior are ungradeable for this run and must not be inferred from the model's self-report.") + } + + $isolation = Get-JsonProperty -Object $result -Name 'isolation' -Default $null + $isolationReport = Format-IsolationReport -Isolation $isolation + + $observed[$configuration] = [pscustomobject]@{ + Model = $model + Provider = [string](Get-JsonProperty -Object $result -Name 'provider' -Default '') + Graded = $graded.Count + Passed = $passed + Total = $total + TranscriptRecorded = -not [string]::IsNullOrWhiteSpace($transcriptText) + ProcessEvidence = $hasProcessEvidence + IsolationReport = $isolationReport + DurationSeconds = Get-JsonProperty -Object $result -Name 'duration_seconds' + TotalTokens = Get-JsonProperty -Object $result -Name 'total_tokens' + ToolCalls = Get-JsonProperty -Object $result -Name 'tool_calls' + } + } + + $withSkill = if ($observed.ContainsKey('with_skill')) { $observed['with_skill'] } else { $null } + $withoutSkill = if ($observed.ContainsKey('without_skill')) { $observed['without_skill'] } else { $null } + + if ($null -ne $withSkill -and $null -ne $withoutSkill) { + if ($withSkill.Model -ne $withoutSkill.Model) { + $warnings.Add("$($entry.eval_name) - with_skill ran on '$($withSkill.Model)' and without_skill on '$($withoutSkill.Model)'. That comparison measures the model as well as the skill.") + } + } elseif ($null -ne $withSkill -or $null -ne $withoutSkill) { + $warnings.Add("$($entry.eval_name) - only one configuration has a result, so there is no controlled comparison yet.") + } + + $rows.Add([pscustomobject]@{ + EvalName = $entry.eval_name + WithSkill = $withSkill + WithoutSkill = $withoutSkill + Assertions = @($metadata.assertions).Count + }) + } + + $builder = [System.Text.StringBuilder]::new() + [void]$builder.AppendLine("# Eval comparison: $($manifest.skill_name) (iteration $($manifest.iteration))") + [void]$builder.AppendLine() + [void]$builder.AppendLine('This repository-side comparison validates recorded grading and never invokes a model. Grading may have been performed by the user-directed external evaluator before this report was generated.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('| Eval | Assertions | with_skill model | with_skill graded | without_skill model | without_skill graded |') + [void]$builder.AppendLine('| --- | --- | --- | --- | --- | --- |') + + foreach ($row in $rows) { + $withModel = if ($null -ne $row.WithSkill) { $row.WithSkill.Model } else { '-' } + $withGraded = if ($null -ne $row.WithSkill) { "$($row.WithSkill.Passed)/$($row.WithSkill.Graded) of $($row.WithSkill.Total)" } else { 'not run' } + $withoutModel = if ($null -ne $row.WithoutSkill) { $row.WithoutSkill.Model } else { '-' } + $withoutGraded = if ($null -ne $row.WithoutSkill) { "$($row.WithoutSkill.Passed)/$($row.WithoutSkill.Graded) of $($row.WithoutSkill.Total)" } else { 'not run' } + [void]$builder.AppendLine("| $($row.EvalName) | $($row.Assertions) | $withModel | $withGraded | $withoutModel | $withoutGraded |") + } + + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Run metrics') + [void]$builder.AppendLine() + [void]$builder.AppendLine('| Eval | Configuration | Duration (s) | Tokens | Tool calls | Transcript |') + [void]$builder.AppendLine('| --- | --- | --- | --- | --- | --- |') + foreach ($row in $rows) { + foreach ($configuration in @('with_skill', 'without_skill')) { + $run = if ($configuration -eq 'with_skill') { $row.WithSkill } else { $row.WithoutSkill } + $duration = if ($null -ne $run -and $null -ne $run.DurationSeconds -and -not [string]::IsNullOrWhiteSpace([string]$run.DurationSeconds)) { [string]$run.DurationSeconds } else { '-' } + $tokens = if ($null -ne $run -and $null -ne $run.TotalTokens -and -not [string]::IsNullOrWhiteSpace([string]$run.TotalTokens)) { [string]$run.TotalTokens } else { '-' } + $toolCalls = if ($null -ne $run -and $null -ne $run.ToolCalls -and -not [string]::IsNullOrWhiteSpace([string]$run.ToolCalls)) { [string]$run.ToolCalls } else { '-' } + $transcript = if ($null -ne $run -and $run.TranscriptRecorded) { 'recorded' } else { '-' } + [void]$builder.AppendLine("| $($row.EvalName) | $configuration | $duration | $tokens | $toolCalls | $transcript |") + } + } + + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Isolation reported') + [void]$builder.AppendLine() + [void]$builder.AppendLine('Flags each run''s harness confirmed: fresh context, isolated home, isolated cwd, filesystem sandbox, candidate skill exposure, transcript capture (Y/N, ? unknown). Process-dependent assertions are only gradeable from a run with process evidence.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('| Eval | Configuration | Isolation | Process evidence |') + [void]$builder.AppendLine('| --- | --- | --- | --- |') + foreach ($row in $rows) { + foreach ($configuration in @('with_skill', 'without_skill')) { + $run = if ($configuration -eq 'with_skill') { $row.WithSkill } else { $row.WithoutSkill } + $isolationReport = if ($null -ne $run) { $run.IsolationReport } else { '-' } + $evidence = if ($null -ne $run -and $run.ProcessEvidence) { 'yes' } elseif ($null -ne $run) { 'none' } else { '-' } + [void]$builder.AppendLine("| $($row.EvalName) | $configuration | $isolationReport | $evidence |") + } + } + + if ($warnings.Count -gt 0) { + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Open items') + [void]$builder.AppendLine() + foreach ($warning in $warnings) { + [void]$builder.AppendLine("- $warning") + } + } + + $comparisonPath = Join-Path $iterationDirectory 'comparison.md' + Write-Utf8File -Path $comparisonPath -Content ($builder.ToString().TrimEnd() + [Environment]::NewLine) + + Write-Host $builder.ToString().TrimEnd() + Write-Host '' + Write-Host "Wrote $comparisonPath" + + $reportScript = Join-Path (Join-Path (Get-RepoRoot) 'scripts') 'generate-eval-report.ps1' + $reportOutput = & pwsh -NoProfile -File $reportScript -IterationDirectory $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + $errors.Add("Report generation failed: $($reportOutput -join [Environment]::NewLine)") + } else { + foreach ($line in @($reportOutput)) { + Write-Host $line + } + } + + if ($errors.Count -gt 0) { + Write-Host '' + Write-Host 'Result files rejected:' + foreach ($errorMessage in $errors) { + Write-Host " $errorMessage" + } + exit 1 + } +} + +switch ($PSCmdlet.ParameterSetName) { + 'Collect' { Invoke-CollectMode } + 'Changed' { Invoke-ChangedMode } + default { Invoke-PrepareMode } +} diff --git a/scripts/sync-skill-install.ps1 b/scripts/sync-skill-install.ps1 new file mode 100644 index 0000000..7f495e3 --- /dev/null +++ b/scripts/sync-skill-install.ps1 @@ -0,0 +1,194 @@ +param( + [string[]]$Skill, + [switch]$VerifyOnly, + [switch]$Prune +) + +$ErrorActionPreference = 'Stop' + +Set-StrictMode -Version Latest + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +[Console]::InputEncoding = $utf8NoBom +[Console]::OutputEncoding = $utf8NoBom +$OutputEncoding = $utf8NoBom + +# Build output is regenerated per location, so its hashes never match across installs. Comparing it +# would bury real drift under noise and push agents back to fragile per-file copying. +$excludePattern = '(^|/)(bin|obj)/' + +function Get-RepoRoot { + return (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +} + +# HostRoot is the tool's own directory. Its absence means the tool is not installed on this machine and +# its skill path is skipped; a missing skill directory *under* an installed tool is drift. The caller still +# requires at least one recognized host root so a mandatory sync cannot pass without checking any copy. +function Get-InstallRoot { + param([string]$SkillName) + + $home_ = [Environment]::GetFolderPath('UserProfile') + return @( + [pscustomobject]@{ HostRoot = (Join-Path $home_ '.claude'); Path = (Join-Path $home_ ".claude/skills/$SkillName") }, + [pscustomobject]@{ HostRoot = (Join-Path $home_ '.agents'); Path = (Join-Path $home_ ".agents/skills/$SkillName") }, + [pscustomobject]@{ HostRoot = (Join-Path $home_ '.gemini/antigravity-cli'); Path = (Join-Path $home_ ".gemini/antigravity-cli/skills/$SkillName") } + ) +} + +function Get-RelativeFile { + param([string]$Root) + + if (-not (Test-Path $Root)) { + return @() + } + + $base = (Resolve-Path $Root).Path.TrimEnd('\', '/') + return @(Get-ChildItem -LiteralPath $Root -Recurse -File -Force | + ForEach-Object { $_.FullName.Substring($base.Length + 1).Replace('\', '/') } | + Where-Object { $_ -notmatch $excludePattern }) +} + +function Sync-SkillTree { + param( + [string]$SourceRoot, + [string]$InstallRoot, + [string[]]$RelativeFile + ) + + foreach ($rel in $RelativeFile) { + $destination = Join-Path $InstallRoot $rel + $destinationDir = Split-Path -Parent $destination + if (-not (Test-Path $destinationDir)) { + New-Item -ItemType Directory -Force -Path $destinationDir | Out-Null + } + + Copy-Item -LiteralPath (Join-Path $SourceRoot $rel) -Destination $destination -Force + } +} + +function Test-SkillTree { + param( + [string]$SourceRoot, + [string]$InstallRoot, + [string[]]$RelativeFile + ) + + $drift = @() + + foreach ($rel in $RelativeFile) { + $expected = (Get-FileHash -LiteralPath (Join-Path $SourceRoot $rel) -Algorithm SHA256).Hash + $destination = Join-Path $InstallRoot $rel + $actual = if (Test-Path -LiteralPath $destination) { + (Get-FileHash -LiteralPath $destination -Algorithm SHA256).Hash + } + else { + 'MISSING' + } + + if ($actual -ne $expected) { + $drift += " DRIFT $rel" + } + } + + # A rename or deletion in the repository leaves the old file behind in an install, where a stale + # skill keeps loading it. Extras are drift too, not cosmetic residue. + foreach ($rel in (Get-RelativeFile -Root $InstallRoot)) { + if ($RelativeFile -notcontains $rel) { + if ($Prune) { + Remove-Item -LiteralPath (Join-Path $InstallRoot $rel) -Force + } + else { + $drift += " EXTRA $rel" + } + } + } + + return $drift +} + +$repoRoot = Get-RepoRoot +$skillsRoot = Join-Path $repoRoot 'skills' + +if (-not $Skill -or $Skill.Count -eq 0) { + $Skill = @(Get-ChildItem -LiteralPath $skillsRoot -Directory | ForEach-Object { $_.Name }) +} + +$totalDrift = 0 +$recognizedHostRootCount = 0 +$unrecognizedSkillCount = 0 + +foreach ($name in $Skill) { + # Both roots are built by joining this name onto a trusted prefix, so a separator or `..` in it walks + # the sync out of the skill tree: the source becomes the repo and the install becomes the skills root, + # where -Prune would delete every other installed skill. Skill directories are kebab-case by + # convention, so anything else is malformed input rather than a skill that is merely missing. + if ($name -notmatch '^[a-z0-9]+(-[a-z0-9]+)*$') { + throw "Invalid skill name: '$name'. Expected a kebab-case skill directory name." + } + + $sourceRoot = Join-Path $skillsRoot $name + if (-not (Test-Path -LiteralPath $sourceRoot)) { + Write-Host "[FAIL] $name (not a repo-managed skill)" + $unrecognizedSkillCount += 1 + continue + } + + $relativeFile = Get-RelativeFile -Root $sourceRoot + + foreach ($install in (Get-InstallRoot -SkillName $name)) { + $installRoot = $install.Path + + if (-not (Test-Path -LiteralPath $install.HostRoot)) { + Write-Host "[SKIP] $name -> $installRoot (host tool not installed)" + continue + } + + $recognizedHostRootCount += 1 + + # An installed host with no copy of the skill used to be skipped, which let a run where nothing + # was ever installed still report "verified, 0 drift". A sync creates the install; a verify fails. + if (-not (Test-Path -LiteralPath $installRoot)) { + if ($VerifyOnly) { + Write-Host "[FAIL] $name -> $installRoot (not installed)" + $totalDrift += 1 + continue + } + + New-Item -ItemType Directory -Force -Path $installRoot | Out-Null + } + + if (-not $VerifyOnly) { + Sync-SkillTree -SourceRoot $sourceRoot -InstallRoot $installRoot -RelativeFile $relativeFile + } + + $drift = @(Test-SkillTree -SourceRoot $sourceRoot -InstallRoot $installRoot -RelativeFile $relativeFile) + $totalDrift += $drift.Count + + if ($drift.Count -eq 0) { + Write-Host "[PASS] $name -> $installRoot ($($relativeFile.Count) files identical)" + } + else { + Write-Host "[FAIL] $name -> $installRoot ($($drift.Count) drifted)" + $drift | ForEach-Object { Write-Host $_ } + } + } +} + +Write-Host '' +if ($unrecognizedSkillCount -gt 0) { + Write-Host "Install sync: $unrecognizedSkillCount requested skill entr$(if ($unrecognizedSkillCount -eq 1) { 'y is' } else { 'ies are' }) not repo-managed." + exit 1 +} + +if ($recognizedHostRootCount -eq 0) { + Write-Host 'Install sync: no recognized local host installation roots found.' + exit 1 +} + +if ($totalDrift -eq 0) { + Write-Host "Install sync: verified, 0 drift." + exit 0 +} + +Write-Host "Install sync: $totalDrift drifted entr$(if ($totalDrift -eq 1) { 'y' } else { 'ies' })." +exit 1 diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 395681e..f66755c 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -639,6 +639,15 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills include va [void](Get-FileText -RepoRoot $repoRoot -RelativePath $skillRelativeFixturePath -GitRef $Ref) } } + + if ($eval.PSObject.Properties.Name -contains 'workspace' -and $null -ne $eval.workspace) { + if ($eval.workspace -isnot [System.Management.Automation.PSCustomObject]) { + throw "$relativeEvalPath eval $($eval.id) has a non-object 'workspace'" + } + if ($eval.workspace.PSObject.Properties.Name -contains 'git' -and $eval.workspace.git -isnot [bool]) { + throw "$relativeEvalPath eval $($eval.id) must declare 'workspace.git' as a boolean" + } + } } } } @@ -1226,6 +1235,401 @@ Add-ValidationResult -Results $results -Name 'Repository automation cannot launc Assert-Contains -Name 'README.md' -Content $readme -Needle 'validate-skill-templates.ps1 -MetadataOnly' } +Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Action { + $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref + $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref + $contributing = Get-FileText -RepoRoot $repoRoot -RelativePath 'CONTRIBUTING.md' -GitRef $Ref + $prepare = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/prepare-skill-evals.ps1' -GitRef $Ref + + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## Portable Eval Handoff' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'this repository prepares a portable evaluation package and stops' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'never execute the prompts you just prepared, and never quietly become the executor of your own package' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'never spawn subagents for the candidate or baseline runs' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'never call an LLM API or an authenticated AI CLI' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'the same model, the same version, and the same configuration' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'a baseline handed the answer key is not a baseline' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Handing the package over' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Executing a package you were handed' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'running it is the task' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'citing it to refuse is a misreading' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'An agent that prepared a package in this session does not get to turn around and execute it.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Hand the user that one file by its absolute path' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Its current context may read `RUN-THIS.prompt.md`' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'but it must not execute an eval prompt itself.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Never reuse a worker or session between runs.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'The user asked for eval results, not a second workflow decision.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Asking for an eval' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`eval `, `evaluate `' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Run the script immediately when asked. Do not reply with a plan, a menu of options' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Eval preparation is a completion gate' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Adding or modifying any repo-managed skill triggers this workflow.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Preparing and reporting satisfies this gate. Executing a prompt never does' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`scripts/sync-skill-install.ps1` runs last' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-NotContains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'run-skill-benchmark.ps1' + Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'Eval packages inside this repository must live under .bot/.' + Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'git does not ignore it' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`.bot/-workspace/` — the default.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Anywhere else inside the repository is forbidden' + Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'It did not run them, and nothing here will.' + + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + + $scriptPath = Join-Path $repoRoot 'scripts/prepare-skill-evals.ps1' + $packageRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-package-' + [Guid]::NewGuid().ToString('N')) + $taskMarker = "`n# Task`n" + try { + $prepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 failed: $($prepareOutput -join [Environment]::NewLine)" + } + + $iterationDirectory = Join-Path $packageRoot 'iteration-1' + $manifest = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'manifest.json'), $utf8NoBom) | ConvertFrom-Json + if (@($manifest.evals).Count -lt 1) { + throw 'The prepared package must contain at least one eval case.' + } + if ([string]$manifest.schema -ne 'codebeltnet/agentic/eval-package/2') { + throw "The package manifest must declare schema eval-package/2; got '$($manifest.schema)'." + } + if ([string]$manifest.report.tool -ne 'tools/generate-eval-report.ps1' -or + [string]$manifest.report.template -ne 'tools/eval-report-template.html' -or + [string]$manifest.report.skill_creator -ne 'tools/skill-creator' -or + [string]$manifest.report.aggregator -ne 'tools/skill-creator/scripts/aggregate_benchmark.py' -or + [string]$manifest.report.viewer -ne 'tools/skill-creator/eval-viewer/generate_review.py' -or + [string]$manifest.report.viewer_template -ne 'tools/skill-creator/eval-viewer/viewer.html' -or + [string]$manifest.report.html -ne 'report.html' -or + [string]$manifest.report.upstream_html -ne 'skill-creator-report.html' -or + [string]$manifest.report.benchmark -ne 'benchmark.json' -or + [string]$manifest.report.benchmark_markdown -ne 'benchmark.md') { + throw 'The prepared package manifest must declare the first-party report, upstream skill-creator tools, and output artifacts.' + } + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/generate-eval-report.ps1')) -or + -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/eval-report-template.html')) -or + -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/skill-creator/scripts/aggregate_benchmark.py')) -or + -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/skill-creator/eval-viewer/generate_review.py')) -or + -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/skill-creator/eval-viewer/viewer.html'))) { + throw 'The prepared package must carry tools/generate-eval-report.ps1.' + } + foreach ($isolationField in @('fresh_context_required', 'isolated_home_required', 'isolated_cwd_required')) { + if (-not [bool]$manifest.isolation.$isolationField) { + throw "manifest.isolation.$isolationField must be true so a harness knows the run is hermetic." + } + } + + $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' + if (-not (Test-Path -LiteralPath $runnerPath)) { + throw 'The prepared package must carry RUN-THIS.prompt.md so the handoff is one paste.' + } + $runner = [System.IO.File]::ReadAllText($runnerPath, $utf8NoBom) + foreach ($needle in @( + 'START NOW. You are the evaluator, grader, and report producer', + 'Do not execute evaluation prompts in the current agent context.', + 'create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`', + 'Never reuse a worker or session between runs', + 'Do not expose this runner', + 'The candidate skill is already inlined in the with_skill run', + 'Launch each worker from its own run directory', + 'Use the same model, version, configuration, tools, and limits for every worker.', + 'Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count.', + 'Do not begin grading until every available worker has completed or failed', + '## Grade and report immediately', + 'grading[].text', + 'tools/skill-creator/agents/grader.md', + 'scripts/aggregate_benchmark.py', + 'eval-viewer/generate_review.py', + 'report.html' + )) { + if (-not $runner.Contains($needle)) { + throw "RUN-THIS.prompt.md must state '$needle'." + } + } + foreach ($forbidden in @( + 'this context has read the runner instructions and can no longer produce a clean run', + '## If you can only hold one context', + 'If you truly cannot, this package is not for you' + )) { + if ($runner.Contains($forbidden)) { + throw "RUN-THIS.prompt.md must not contain the refusal path '$forbidden'." + } + } + foreach ($entry in @($manifest.evals)) { + $metadataForLeak = [System.IO.File]::ReadAllText((Join-Path (Join-Path $iterationDirectory $entry.directory) 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json + if ($runner.Contains([string]$metadataForLeak.expected_output)) { + throw 'RUN-THIS.prompt.md must not carry an expected output; the executing harness never sees the grading key.' + } + } + + foreach ($entry in @($manifest.evals)) { + $evalDirectory = Join-Path $iterationDirectory $entry.directory + + # Manifest run wiring resolves to real files inside each run directory. + foreach ($configuration in @('with_skill', 'without_skill')) { + $run = $entry.runs.$configuration + foreach ($pathProperty in @('prompt', 'run_manifest', 'result', 'working_directory', 'home_directory')) { + if ($run.PSObject.Properties.Name -notcontains $pathProperty) { + throw "$($entry.eval_name)/$configuration manifest entry must declare '$pathProperty'." + } + } + foreach ($mustExist in @($run.prompt, $run.run_manifest, $run.working_directory, $run.home_directory)) { + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory $mustExist))) { + throw "$($entry.eval_name)/$configuration manifest path '$mustExist' does not exist." + } + } + } + + $withRunDir = Join-Path $evalDirectory 'with_skill' + $withoutRunDir = Join-Path $evalDirectory 'without_skill' + + # 2. with_skill stages the candidate skill; 3. without_skill carries no copy of it. + if (-not (Test-Path -LiteralPath (Join-Path $withRunDir 'skill/dotnet-strong-name-signing/SKILL.md'))) { + throw "$($entry.eval_name)/with_skill must stage the candidate skill under skill/dotnet-strong-name-signing/." + } + if (Test-Path -LiteralPath (Join-Path $withoutRunDir 'skill')) { + throw "$($entry.eval_name)/without_skill must not contain a skill/ directory." + } + if (Test-Path -LiteralPath (Join-Path $withoutRunDir 'SKILL.md')) { + throw "$($entry.eval_name)/without_skill must not contain a SKILL.md." + } + + # 1. Each run is a self-contained sandbox root with its own repo/, home/, prompt.md, run.json. + foreach ($runDir in @($withRunDir, $withoutRunDir)) { + foreach ($required in @('repo', 'home', 'prompt.md', 'run.json')) { + if (-not (Test-Path -LiteralPath (Join-Path $runDir $required))) { + throw "$($entry.eval_name) run directory '$runDir' is missing '$required'." + } + } + } + + # 10. run.json requires fresh context and isolation; 6/7. it references nothing outside the run package. + $withRunJson = [System.IO.File]::ReadAllText((Join-Path $withRunDir 'run.json'), $utf8NoBom) + $withoutRunJson = [System.IO.File]::ReadAllText((Join-Path $withoutRunDir 'run.json'), $utf8NoBom) + $withRun = $withRunJson | ConvertFrom-Json + $withoutRun = $withoutRunJson | ConvertFrom-Json + foreach ($run in @($withRun, $withoutRun)) { + if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { + throw "$($entry.eval_name) run.json must require fresh context, filesystem, and home isolation." + } + if ([string]$run.workingDirectory -ne 'repo' -or [string]$run.homeDirectory -ne 'home') { + throw "$($entry.eval_name) run.json must set workingDirectory=repo and homeDirectory=home." + } + } + if ([string]$withRun.skillDirectory -ne 'skill/dotnet-strong-name-signing') { + throw "$($entry.eval_name)/with_skill run.json must point skillDirectory at the staged candidate skill." + } + if ($null -ne $withoutRun.skillDirectory -or $null -ne $withoutRun.skillName) { + throw "$($entry.eval_name)/without_skill run.json must not name a skill or skill directory." + } + foreach ($runJson in @($withRunJson, $withoutRunJson)) { + foreach ($leak in @('skills/dotnet-strong-name-signing', '.agents', '.claude', '.codex')) { + if ($runJson.Contains($leak)) { + throw "$($entry.eval_name) run.json references '$leak', which points outside the run package." + } + } + } + if ($withoutRunJson.Contains([string]$manifest.skill_name)) { + throw "$($entry.eval_name)/without_skill run.json must not name the skill under test." + } + + # 9. The with_skill and without_skill repositories are identical (proven by fixture hash). + if ([string]$withRun.fixtureHash -ne [string]$withoutRun.fixtureHash) { + throw "$($entry.eval_name) with_skill and without_skill fixture hashes differ; the repositories must match." + } + + $withSkill = [System.IO.File]::ReadAllText((Join-Path $withRunDir 'prompt.md'), $utf8NoBom) + $withoutSkill = [System.IO.File]::ReadAllText((Join-Path $withoutRunDir 'prompt.md'), $utf8NoBom) + $metadata = [System.IO.File]::ReadAllText((Join-Path $evalDirectory 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json + + if (-not $withSkill.Contains([string]$metadata.prompt) -or -not $withoutSkill.Contains([string]$metadata.prompt)) { + throw "$($entry.eval_name) must put the same task prompt in both configurations." + } + if (-not $withSkill.Contains('# Working environment') -or -not $withoutSkill.Contains('# Working environment')) { + throw "$($entry.eval_name) must give both configurations the working-environment boundary." + } + if ($withSkill.Contains('codebeltnet/agentic portable eval prompt') -or $withoutSkill.Contains('codebeltnet/agentic portable eval prompt')) { + throw "$($entry.eval_name) worker prompts must not announce that they are part of an evaluation." + } + if (-not $withSkill.Contains('# Response contract') -or -not $withoutSkill.Contains('# Response contract')) { + throw "$($entry.eval_name) must give both configurations a response contract." + } + + $withSkillTaskIndex = $withSkill.IndexOf($taskMarker, [System.StringComparison]::Ordinal) + $withoutSkillTaskIndex = $withoutSkill.IndexOf($taskMarker, [System.StringComparison]::Ordinal) + if ($withSkillTaskIndex -lt 0 -or $withoutSkillTaskIndex -lt 0) { + throw "$($entry.eval_name) must open its task with a '# Task' heading in both configurations." + } + if ($withSkill.Substring($withSkillTaskIndex) -ne $withoutSkill.Substring($withoutSkillTaskIndex)) { + throw "$($entry.eval_name) must vary only the operating-instructions section; the task, inputs, or response contract differ." + } + + if ($withoutSkill.Contains([string]$manifest.skill_name)) { + throw "$($entry.eval_name) baseline prompt must not name the skill under test." + } + foreach ($prompt in @($withSkill, $withoutSkill)) { + if ($prompt.Contains([string]$metadata.expected_output)) { + throw "$($entry.eval_name) prompts must not carry the expected output; that is the grading key." + } + foreach ($assertion in @($metadata.assertions)) { + if ($prompt.Contains([string]$assertion)) { + throw "$($entry.eval_name) prompts must not carry assertion '$assertion'; that is the grading key." + } + } + } + + foreach ($configuration in @('with_skill', 'without_skill')) { + $resultFile = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + $stubPath = Join-Path (Join-Path $evalDirectory 'results') $resultFile + $stub = [System.IO.File]::ReadAllText($stubPath, $utf8NoBom) | ConvertFrom-Json + if ([string]$stub.configuration -ne $configuration) { + throw "$($entry.eval_name) result stub $resultFile must declare configuration '$configuration'." + } + if (@($stub.grading).Count -ne @($metadata.assertions).Count) { + throw "$($entry.eval_name) result stub $resultFile must carry one grading entry per assertion." + } + foreach ($propertyName in @('transcript', 'shell_commands', 'files_read', 'files_written', 'exit_status', 'duration_seconds', 'total_tokens', 'tool_calls', 'turns', 'base_input_tokens', 'output_tokens', 'cache_read_tokens', 'cache_write_tokens', 'cache_write_1h_tokens', 'estimated_cost_usd', 'model_effort', 'isolation')) { + if ($stub.PSObject.Properties.Name -notcontains $propertyName) { + throw "$($entry.eval_name) result stub $resultFile must expose optional field '$propertyName'." + } + } + foreach ($isolationField in @('fresh_context', 'isolated_home', 'isolated_cwd', 'filesystem_sandbox', 'candidate_skill_exposed', 'transcript_captured')) { + if ($stub.isolation.PSObject.Properties.Name -notcontains $isolationField) { + throw "$($entry.eval_name) result stub $resultFile must expose isolation flag '$isolationField'." + } + } + } + } + + $collectOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -CollectResults failed on an unrun package: $($collectOutput -join [Environment]::NewLine)" + } + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'comparison.md'))) { + throw 'prepare-skill-evals.ps1 -CollectResults must write comparison.md.' + } + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'report.html'))) { + throw 'prepare-skill-evals.ps1 -CollectResults must write report.html.' + } + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'benchmark.json'))) { + throw 'prepare-skill-evals.ps1 -CollectResults must write benchmark.json.' + } + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'benchmark.md'))) { + throw 'prepare-skill-evals.ps1 -CollectResults must write benchmark.md.' + } + + $firstEntry = @($manifest.evals)[0] + $firstEvalDirectory = Join-Path $iterationDirectory $firstEntry.directory + foreach ($resultFile in @('with-skill.result.json', 'without-skill.result.json')) { + $resultPath = Join-Path (Join-Path $firstEvalDirectory 'results') $resultFile + $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json + $result.model = 'validator-model' + $result.provider = 'validator-provider' + $result.harness = 'validator-harness' + $result.executed_utc = '2026-01-01T00:00:00Z' + $result.output = 'validator output' + $result.transcript = 'validator transcript' + $result.duration_seconds = 1.25 + $result.total_tokens = 123 + $result.tool_calls = 2 + $result.turns = 4 + $result.base_input_tokens = 27 + $result.output_tokens = 123 + $result.cache_read_tokens = 456 + $result.cache_write_1h_tokens = 78 + $result.estimated_cost_usd = 0.12 + $result.model_effort = 'high' + foreach ($grade in @($result.grading)) { + $grade.passed = $true + $grade.evidence = 'validator evidence' + } + $result.isolation.fresh_context = $true + $result.isolation.isolated_home = $true + $result.isolation.isolated_cwd = $true + $result.isolation.filesystem_sandbox = $true + $result.isolation.candidate_skill_exposed = $true + $result.isolation.transcript_captured = $true + [System.IO.File]::WriteAllText($resultPath, (($result | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + } + $metricsOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -CollectResults failed on recorded metrics: $($metricsOutput -join [Environment]::NewLine)" + } + $comparison = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'comparison.md'), $utf8NoBom) + foreach ($needle in @('## Run metrics', '| 1.25 | 123 | 2 | recorded |', '## Isolation reported', 'fresh=Y home=Y cwd=Y fs=Y skill=Y tx=Y')) { + if (-not $comparison.Contains($needle)) { + throw "comparison.md must report available run metric '$needle'." + } + } + $report = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'report.html'), $utf8NoBom) + if (-not $report.Contains('validator output') -or -not $report.Contains('Formal grades') -or -not $report.Contains('Benchmark') -or -not $report.Contains('Base input tokens') -or -not $report.Contains('WITH SKILL') -or -not $report.Contains('WITHOUT SKILL')) { + throw 'report.html must show the first-party side-by-side review with captured output, formal grades, telemetry, and both configurations.' + } + $upstreamReport = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'skill-creator-report.html'), $utf8NoBom) + if (-not $upstreamReport.Contains('Formal Grades') -or -not $upstreamReport.Contains('Benchmark Results') -or -not $upstreamReport.Contains('EMBEDDED_DATA')) { + throw 'skill-creator-report.html must preserve the exact upstream viewer output.' + } + $benchmark = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'benchmark.json'), $utf8NoBom) | ConvertFrom-Json + $expectedEvalCount = @($manifest.evals).Count + if ([int]$benchmark.metadata.runs_per_configuration -ne 1 -or @($benchmark.metadata.evals_run).Count -ne $expectedEvalCount -or @($benchmark.runs).Count -ne 2) { + throw "benchmark.json must use the upstream skill-creator schema for the recorded paired run set (runs_per_configuration=$($benchmark.metadata.runs_per_configuration), evals=$(@($benchmark.metadata.evals_run).Count), runs=$(@($benchmark.runs).Count), expected_evals=$expectedEvalCount)." + } + if ([int]$benchmark.run_summary.with_skill.tokens.mean -ne 123 -or [int]$benchmark.run_summary.without_skill.tokens.mean -ne 123) { + throw "benchmark.json must preserve recorded token metrics through the upstream skill-creator aggregation (with_skill=$($benchmark.run_summary.with_skill.tokens.mean), without_skill=$($benchmark.run_summary.without_skill.tokens.mean))." + } + + $changedOutput = & pwsh -NoProfile -File $scriptPath -Changed -Base 'HEAD' -OutputRoot $packageRoot 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -Changed failed: $($changedOutput -join [Environment]::NewLine)" + } + if (($changedOutput -join ' ') -notmatch 'Changed repo-managed skills in') { + throw 'prepare-skill-evals.ps1 -Changed must report the scope it resolved.' + } + + $insideRepo = Join-Path $repoRoot 'agentic-eval-isolation-check' + $isolationOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo 2>&1 + if ($LASTEXITCODE -eq 0) { + throw 'prepare-skill-evals.ps1 must refuse an output root inside this repository but outside .bot/.' + } + if (Test-Path -LiteralPath $insideRepo) { + Remove-Item -LiteralPath $insideRepo -Recurse -Force + throw 'prepare-skill-evals.ps1 must not create a refused output root.' + } + if (($isolationOutput -join ' ') -notmatch 'must live under \.bot/') { + throw 'prepare-skill-evals.ps1 must explain why an in-repository output root was refused.' + } + + # .bot/ is the sanctioned in-repository home, and it only works while git ignores it. + $botRoot = Join-Path (Join-Path $repoRoot '.bot') 'agentic-eval-bot-check' + try { + $botOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 must accept an output root under .bot/: $($botOutput -join [Environment]::NewLine)" + } + $botStatus = git -C $repoRoot status --porcelain --untracked-files=all -- '.bot' 2>$null + if (@($botStatus | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }).Count -gt 0) { + throw 'An eval package under .bot/ must stay invisible to git; .gitignore no longer covers it.' + } + } finally { + if (Test-Path -LiteralPath $botRoot) { + Remove-Item -LiteralPath $botRoot -Recurse -Force -ErrorAction SilentlyContinue + } + } + } finally { + if (Test-Path -LiteralPath $packageRoot) { + Remove-Item -LiteralPath $packageRoot -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Codebelt xUnit migration and bootstrap contracts' -Action { $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/SKILL.md' -GitRef $Ref $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/FORMS.md' -GitRef $Ref @@ -1249,9 +1653,31 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'ApplicationTest>' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Never emit them in generated or refactored code.' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Never add a process-launching fallback' - Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'zero remaining `WebApplicationFactory`' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'verify-dotnet-test-migration.ps1' + # A wrapped, renamed, or repackaged factory once shipped as a completed migration. The named failure + # modes and the script-produced verdict are what stop that from reading as success again. + foreach ($needle in @('What finishing looks like', 'Wrapping the factory', 'Renaming the seam', 'Bumping packages instead', 'a verdict a script produces rather than a summary you write')) { + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle $needle + } + $verify = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1' -GitRef $Ref + $verifyTest = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1' -GitRef $Ref + foreach ($needle in @('LAUNDERED-FACADE', 'LEGACY-PACKAGE-RETAINED', 'XUNIT-ANCHOR-BREACH', 'CHURN-WITHOUT-CONVERSION', 'project.assets.json')) { + Assert-Contains -Name 'verify-dotnet-test-migration.ps1' -Content $verify -Needle $needle + } + # A gate that only ever fails is noise, so the positive control is part of the contract. + Assert-Contains -Name 'test-verify-dotnet-test-migration.ps1' -Content $verifyTest -Needle 'Positive control' + Assert-Contains -Name 'test-verify-dotnet-test-migration.ps1' -Content $verifyTest -Needle 'completed migration' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Do not invent an endpoint, service, configuration key, or expected result.' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'An MTP executable run may supplement that gate but never replaces it' + # The skill once answered a bare invocation with a capability menu and inspected nothing; these lock the evidence-first contract. + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'You were invoked. That is the request.' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Forbidden as a first response:' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'The test host comes from Codebelt, not from Microsoft' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'it is the fallback for genuine ambiguity, not an intake wizard' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'do not exist below Codebelt xUnit **11.1.0**' + Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle 'This form is a fallback for genuine ambiguity, not an intake step.' + Assert-Contains -Name 'inspect-dotnet-tests.ps1' -Content $inspect -Needle '$managedFixtureFloor = [version]' + Assert-Contains -Name 'dotnet-test/web-functional-tests.md' -Content $web -Needle 'Probing with `if (_application is IAsyncDisposable d)` is dead defensive code' Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle '### project_selection' Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle '### operation_mode' Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle '### test_role' @@ -1275,17 +1701,33 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'https://api.nuget.org/v3/index.json' Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'Test-PackageCompatibility -Packages $trial' Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'combined restore passed' + # xunit.v3 and xunit.runner.visualstudio shipped stable 4.0.0 releases ahead of Codebelt xUnit; "newest stable" must + # never be allowed to outrun the Codebelt package the skill targets. + Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'Resolve-XunitAnchor -BaseAddress' + Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'Select-AnchoredCandidate -PackageId' + Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'at or below major' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Newest is not the ceiling for `xunit*`.' + Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'no candidate above the anchored major may reach a restore' Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'stable candidate resolution should succeed' Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'combined package set' Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'restore evidence' $evalObject = $evals | ConvertFrom-Json - if (@($evalObject.evals).Count -ne 6) { - throw "dotnet-test must define exactly six requested paired eval scenarios; found $(@($evalObject.evals).Count)" + if (@($evalObject.evals).Count -lt 8) { + throw "dotnet-test must define the six paired role scenarios, the bare-invocation immediate-action scenario, and the laundered-migration recovery scenario; found $(@($evalObject.evals).Count)" } foreach ($needle in @('attached Acme.Calculator fixture', 'xUnit v2 project', 'web-cdn-origin-style', 'IClassFixture>', 'ApplicationTestFactory pattern', 'ApplicationTest>')) { Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle } + # A bare invocation must act on inspector evidence instead of answering with a menu; that regression shipped once, so it stays covered. + foreach ($needle in @('Does not present a numbered menu of modes', 'Runs inspect-dotnet-tests.ps1 as the first action')) { + Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle + } + # A run once wrapped WebApplicationFactory in a private nested class and reported the migration done; + # the recovery scenario keeps that exact outcome in the eval set rather than only in a postmortem. + foreach ($needle in @('nested private CdnOriginApplicationFactory', 'including nested, private, and renamed facades', 'verify-dotnet-test-migration.ps1')) { + Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle + } if (@($fixtureFiles | Where-Object { $_ -match '(^|[\\/])(bin|obj)([\\/]|$)' }).Count -gt 0) { throw 'dotnet-test eval fixtures must not include bin/ or obj/ paths' } diff --git a/skills/dotnet-remote-testing/FORMS.md b/skills/dotnet-remote-testing/FORMS.md index 8fea89d..72b3f3b 100644 --- a/skills/dotnet-remote-testing/FORMS.md +++ b/skills/dotnet-remote-testing/FORMS.md @@ -1,6 +1,18 @@ # .NET Remote Testing Input Form -Collect only the fields that are still unresolved after inspecting the request and the repository. Most remote-test requests are fully determined and need **no** questions — for example, "remote test this solution" against a repository with a single applicable environment. Prefer native structured controls when the host provides them; otherwise use the plain-text fallback below without changing field order, defaults, or the final confirmation. +This form is a **fallback for genuine ambiguity, not an intake checklist**. The default path collects nothing: a request to remote test is executed, not surveyed. + +## Autonomy gate — evaluate before presenting any field + +Present a field only when one of these is true: + +1. The runner exited `SelectionRequired` (`16`) — present `environment`, restricted to the `candidates` it returned. +2. The developer explicitly asked to choose something ("let me pick the environment", "which options do I have?"). +3. The developer supplied a value that is genuinely unusable (for example a project path that does not exist). + +If none apply, run with the defaults — auto-resolved target, `Debug`, no coverage — and present **no** fields and **no** confirmation. A single applicable Docker environment in `testenvironments.json` is a resolved answer, not a question. Never walk the field list top-to-bottom to "gather requirements", and never ask `test_scope`, `configuration`, or `coverage` unprompted; those are defaults the developer overrides by saying so. + +Prefer native structured controls when the host provides them; otherwise use the plain-text fallback below without changing field order, defaults, or the final confirmation. ## Fields @@ -8,9 +20,9 @@ Collect only the fields that are still unresolved after inspecting the request a - **type:** single-choice - **prompt:** Which environment should run the tests? -- **choices:** Dynamically list the environments from `remote-test.cs list` — the configured Docker environments from `testenvironments.json`, or the Microsoft-derived environments (for example `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`) when no `testenvironments.json` exists -- **default:** The only applicable Docker environment, or the environment explicitly named by the user (Recommended) -- **required:** true +- **choices:** The `candidates` returned by the runner's `SelectionRequired` result — the configured Docker environments from `testenvironments.json`, or the Microsoft-derived environments (for example `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`) when no `testenvironments.json` exists +- **default:** The environment explicitly named by the user +- **required:** Only when the runner exits `SelectionRequired`. A single applicable Docker environment resolves automatically and is never asked. ### test_scope @@ -21,7 +33,7 @@ Collect only the fields that are still unresolved after inspecting the request a - A specific project - A class or test filter - **default:** Entire solution / auto-resolved target (Recommended) -- **required:** true +- **required:** false — the default applies silently; ask only when the developer asks to narrow the run but does not say how ### project @@ -50,7 +62,7 @@ Collect only the fields that are still unresolved after inspecting the request a - Debug (Recommended) - Release - **default:** Debug (Recommended) -- **required:** true +- **required:** false — `Debug` applies silently unless the developer names a configuration ### coverage @@ -60,7 +72,7 @@ Collect only the fields that are still unresolved after inspecting the request a - No (Recommended) - Yes - **default:** No (Recommended) -- **required:** true +- **required:** false — never ask; coverage is collected only when the developer requests it ### confirmation @@ -70,15 +82,16 @@ Collect only the fields that are still unresolved after inspecting the request a - Yes (Recommended) - No - **default:** Yes (Recommended) -- **required:** true +- **required:** Only when at least one other field was presented. On the autonomous path there is nothing to confirm — the run is the answer. ## Presentation rules -- Infer explicit answers from the request and from `remote-test.cs list`/`plan`; do not ask them again. -- Ask one unresolved field at a time. Never bundle multiple questions. +- Clear the autonomy gate above before presenting anything. In practice most invocations present no fields at all. +- Infer explicit answers from the request and from the runner's own output; do not ask them again. +- Ask one unresolved field at a time. Never bundle multiple questions, and never turn a single blocking choice into a broader intake. - Present the recommended/default choice first and suffix it with `(Recommended)`. -- For the `environment` field, offer the discovered environment names as selectable choices rather than free text. When exactly one environment applies, select it without asking. +- For the `environment` field, offer the runner's `candidates` as selectable choices rather than free text. When exactly one environment applies, select it without asking. - For `project`, offer the auto-resolved target as a selectable choice alongside a free-text path. - In plain-text fallback mode, start immediately with `Field: ` and show numbered choices. Do not add a conversational preamble. - If the user leaves a shown computed/default choice blank, accept it and continue. -- After all fields are resolved, summarize the exact environment, target, configuration, and coverage, then ask `confirmation`. +- When fields were presented, summarize the exact environment, target, configuration, and coverage after they are resolved, then ask `confirmation`. When no field was presented, skip the summary and the confirmation and run. diff --git a/skills/dotnet-remote-testing/SKILL.md b/skills/dotnet-remote-testing/SKILL.md index 349dd67..1d9a698 100644 --- a/skills/dotnet-remote-testing/SKILL.md +++ b/skills/dotnet-remote-testing/SKILL.md @@ -1,71 +1,126 @@ --- name: dotnet-remote-testing description: > - Run .NET tests inside a resolved remote Docker environment and return structured results — Visual Studio's Remote Testing experience (choose an environment, run tests, see results) without hand-writing container plumbing. Use when asked to remote test, run tests in Docker or a container, run tests against a specific .NET SDK, list or select test environments, or honor an existing testenvironments.json. Honors testenvironments.json Docker environments or derives zero-config environments from Microsoft's live .NET release index (LTS, STS, preview) using official mcr.microsoft.com/dotnet/sdk images via the bundled runner scripts/remote-test.cs. Docker only; WSL and SSH are reported unsupported. Do NOT use to author or refactor test code, choose a testing framework, generate Dockerfiles, or run tests on the host. + Run .NET tests inside a resolved remote Docker environment — Visual Studio's Remote Testing without hand-writing container plumbing. Invoking this skill IS the request: run the tests immediately. Never reply with a menu of options or a questionnaire. Use when asked to remote test, run tests in Docker or a container, target a specific .NET SDK, list or select test environments, or honor an existing testenvironments.json. Honors configured Docker environments, or derives them from Microsoft's live .NET release index using mcr.microsoft.com/dotnet/sdk images, plus codebeltnet/ubuntu-testrunner for multi-targeted repos, via the runner scripts/remote-test.cs. Docker only; WSL and SSH are unsupported. Do NOT use to author or refactor test code, choose a testing framework, generate Dockerfiles, or run tests on the host. compatibility: > - Requires the .NET SDK, Docker, and PowerShell 7+. Zero-config discovery needs network access to Microsoft's release index and mcr.microsoft.com; a cache enables offline reuse. + Requires the .NET 10 SDK or later (`dotnet run --file`), a running Docker daemon, and PowerShell 7+. Zero-config discovery needs network access; a cache enables offline reuse. --- # .NET Remote Testing -Give developers the experience Visual Studio's experimental [Remote Testing](https://learn.microsoft.com/en-us/visualstudio/test/remote-testing?view=visualstudio) was meant to provide: +## Do this now + +**You were invoked. That is the request. Run the tests.** + +Your first action is this command — not a question, not a menu, not a summary of what you could do: + +``` +dotnet run --file "/scripts/remote-test.cs" -- run --repo-root "" +``` + +Run it immediately, then report the result. This holds for a bare `/dotnet-remote-testing` with no other words, and for "remote test this repo", "run my tests in Docker", or any equivalent. There is nothing to clarify first: the runner resolves the environment, the target, and the configuration by itself, and it is the *only* thing that decides whether a question is needed. + +**Forbidden as a first response:** listing your capabilities; "here are the typical workflows"; offering `list` / `plan` / `run` as choices; asking which project, environment, .NET version, or filter to use; asking for confirmation. If you are about to write "What would you like me to do?", you have already failed — run the command instead. + +You may ask a question in exactly one situation: the command exited `16` (`SelectionRequired`), which means a genuine choice remains. Then ask one question listing only the `candidates` it returned, and rerun with `-e `. Every other exit code is an outcome to report, never a question. The full mapping is in [Failure handling](#failure-handling). + +Resolve the two placeholders exactly once and reuse them verbatim: + +- `` is the directory containing this `SKILL.md`. Build the path from it; do not guess a relative path from the working directory and do not copy `` through literally. +- `` is the workspace/solution root — the current directory unless the developer named another. Always pass it explicitly rather than relying on the process default, and always quote both paths (Windows paths contain backslashes and often spaces). + +Prerequisites, and the exact way each one fails: + +| Requirement | Why | If missing | +|---|---|---| +| .NET 10 SDK or later | `dotnet run --file` (file-based apps) | The CLI rejects `--file`; report the SDK requirement — do not rewrite the runner into a project | +| Docker, running | Test execution | The runner exits `DockerUnavailable` (`5`); report it, never fall back to the host | +| Network (first run) | Release metadata, image pull | Use `--offline` with `--cache-root` when a cache exists; otherwise exit `15` explains it | + +## Why this skill exists + +Visual Studio's experimental [Remote Testing](https://learn.microsoft.com/en-us/visualstudio/test/remote-testing?view=visualstudio) promised: > **Choose a test environment → run tests → see results.** -Everything between those two actions — configuration discovery, .NET release discovery, image resolution, source staging, NuGet caching, restore, build, test execution, result collection, cancellation, and cleanup — is infrastructure that belongs *behind* the abstraction. Your job is orchestration: understand what the developer means, then hand the work to the deterministic runner. Do not turn a routine "run my tests in .NET 10" into a Docker tutorial, and never compose ad-hoc `docker run` command lines yourself. +Everything between those actions — configuration discovery, .NET release discovery, image resolution, source staging, NuGet caching, restore, build, test execution, result collection, cancellation, and cleanup — is infrastructure that belongs *behind* the abstraction. A developer who reached for this skill has already chosen remote testing; handing that choice back as a questionnaire is the friction this skill exists to remove. Do not turn a routine "run my tests in .NET 10" into a Docker tutorial, and never compose ad-hoc `docker run` command lines yourself. ## Architecture: you orchestrate, the runner executes -The bundled .NET file-based program `scripts/remote-test.cs` is the **execution layer**. It is deterministic and self-tested. You are the **orchestration layer**. Always route execution through it instead of driving Docker directly: +The bundled .NET file-based program `scripts/remote-test.cs` is the **execution layer**. It is deterministic and self-tested (`--self-test`). You are the **orchestration layer**. Always route execution through it instead of driving Docker directly: ``` dotnet run --file "/scripts/remote-test.cs" -- [options] ``` -Commands: `list` (show environments), `plan` (resolve an environment + image and print the execution plan without running), `run` (execute restore/build/test in the resolved container), and `--self-test` (built-in deterministic tests). Add `--json` to any command for machine-readable output. +Commands: `run` (execute restore/build/test in the resolved container — the default action above), `list` (show environments, when the developer asks to *see* them), `plan` (resolve an environment + image and print the execution plan without running, when the developer asks what *would* happen), and `--self-test`. Add `--json` to any command for machine-readable output. + +These are your commands, not a menu for the developer. Never present them as options to choose from. ## Critical - **Never run tests on the host and never silently fall back to local.** If remote testing was requested, tests must execute in the resolved Docker environment. If Docker is unavailable, report that (the runner exits `DockerUnavailable`) — do not run `dotnet test` locally instead. - **`testenvironments.json` is the configuration contract.** Honor Microsoft's existing version-1 schema. Do not invent a competing format, and do not modify `testenvironments.json` unless explicitly asked. -- **Do not generate container plumbing.** Never create a `Dockerfile`, `docker-compose.yml`/`compose.yml`, `.devcontainer/`, `.vscode/`, `Directory.Build.*`, or throwaway scripts to make remote testing work. Zero-configuration testing uses official Microsoft SDK images directly. An *existing* configured `dockerFile` is honored because it is deliberate repository intent. +- **Do not generate container plumbing.** Never create a `Dockerfile`, `docker-compose.yml`/`compose.yml`, `.devcontainer/`, `.vscode/`, `Directory.Build.*`, or throwaway scripts to make remote testing work. Zero-configuration testing uses official Microsoft SDK images directly. An *existing* configured `dockerFile` is honored because it is deliberate repository intent. The runner may derive a cached preparation layer for build tooling the image lacks (see [Build tooling in the image](#build-tooling-in-the-image)) — that happens inside the runner, outside the repository, and is never something you author. - **Do not hardcode .NET versions.** Supported LTS/STS channels and the current preview channel are discovered from Microsoft's release metadata at runtime. `.NET 10`/`.NET 11` are examples, never constants. - **Do not modify the repository to make tests pass.** Never edit `global.json`, project files, target frameworks, or test packages. Report incompatibilities instead. - **Report infrastructure failures as infrastructure, not as failing unit tests.** The runner classifies each phase distinctly; preserve that distinction when you summarize. +## Default action: run the tests + +This restates [Do this now](#do-this-now) because it is the rule most often broken. Autonomy is the default, and the runner — not you — decides when a question is unavoidable: + +- **Resolution succeeds → run, silently.** A single Docker entry in `testenvironments.json`, a single derived environment, or the derived environment matching the repository's own target frameworks — with the auto-resolved target, `Debug`, and no coverage. No questions, no confirmation, no preflight commentary. +- **`SelectionRequired` (exit `16`) → ask exactly one question.** The runner returns the `candidates` it could not choose between. Present those names and nothing else, then rerun with `-e `. +- **Any other failure → report it.** A resolution or infrastructure failure is an outcome to report, not a question to ask. + +Scope, configuration, framework, and coverage are options the developer volunteers — never fields you collect up front. Pass through only what was actually asked for. + ## Step 1: Understand intent and inputs -Read `FORMS.md` and infer everything you can from the request and repository. Most invocations need no questions at all — "remote test this solution" against a repo with one applicable environment is fully determined. Only ask (one field at a time) when a genuine choice remains, such as which environment when several apply. Resolve the workspace/solution root (`--repo-root`, default: current directory). +Infer everything you can from the request and the repository, and resolve the workspace/solution root (`--repo-root`, default: current directory). Then go straight to the command. Read `FORMS.md` only when the runner reported `SelectionRequired` or the developer explicitly asked to choose options; it defines *how* to ask, not a checklist to work through. Typical intents map directly to a command: | The developer says… | You run… | |---|---| +| A bare invocation / "remote test this solution" / "run these tests in .NET 10" | `run` | | "What environments can I test in?" / "list remote environments" | `list` | | "Show me the plan / which image will you use?" | `plan` | -| "Remote test this solution" / "run these tests in .NET 10" | `run` | ## Step 2: List and resolve the environment Resolution is deterministic and follows this precedence, which the runner enforces — do not second-guess it: 1. An environment the user names explicitly (`--environment `). -2. An applicable Docker environment from `testenvironments.json` (authoritative when the file exists — never supplement it with invented environments). -3. Microsoft-derived environments when no `testenvironments.json` exists. +2. An applicable Docker environment from `testenvironments.json` (authoritative when the file exists — never supplement it with invented environments). A single Docker entry is selected outright. +3. A derived environment when no `testenvironments.json` exists. A single derived environment is selected outright; otherwise the repository's own target frameworks choose one: + - **One .NET major** → the Microsoft SDK channel matching it. + - **Several .NET majors** → a Codebelt multi-SDK runner (`codebeltnet/ubuntu-testrunner`, tags like `8-9-10-11`) that provides every one of them. + +That third rule is what makes zero-configuration testing unattended: the source already answered the question. The runner reports the choice (`Selected automatically: …`) so an unattended selection stays auditable, and it deliberately does not approximate — no .NET target framework, two channels for the same major, or no runner covering the required set all fall through to a question rather than guessing an SDK the repository never asked for. -Run `list` to show the choices. When exactly one applicable Docker environment exists, use it. When several exist and the user has not chosen, present the names concisely and let them pick — do not guess intent: +### Why multi-targeting needs a different image + +A Microsoft SDK image ships exactly **one** runtime: `mcr.microsoft.com/dotnet/sdk:10.0` contains only `Microsoft.NETCore.App 10.0.x`. A repository targeting `net9.0;net10.0` therefore *builds* both there and then fails to execute the `net9.0` tests — there is no .NET 9 runtime in the image. The Codebelt runner carries several SDKs at once, so the whole target-framework matrix runs in a single container instead of one container per TFM. + +The runner enforces this rather than leaving it to judgment: pointing a multi-targeted repository at a single-SDK image is reported as `SdkIncompatibility` (`7`) naming the unrunnable frameworks and the remedy. Narrowing the run with `-f/--framework` narrows the environment choice too, so `-f net10.0` on a multi-targeted repository resolves to the ordinary `10` channel. + +The runner performs all of this inside `run` itself, so **do not call `list` as a preflight before running**. When a choice genuinely remains, `run` stops with `SelectionRequired` and hands you the `candidates` — present those names concisely and let the developer pick, then rerun with `-e `. Do not guess between them, and do not ask before the runner says a choice is needed. + +Use `list` when the developer wants to *see* the environments: ``` dotnet run --file "/scripts/remote-test.cs" -- list --repo-root "" ``` -Absence of `testenvironments.json` is **not** an error. In that case the runner derives environments from Microsoft's live release index (`mcr.microsoft.com/dotnet/sdk` images for each supported LTS/STS channel plus the current preview), so no files need to be added to the repository. Environment names look like `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`; the exact set comes from metadata at runtime. +Absence of `testenvironments.json` is **not** an error. In that case the runner derives environments from Microsoft's live release index (`mcr.microsoft.com/dotnet/sdk` images for each supported LTS/STS channel plus the current preview), so no files need to be added to the repository. Environment names look like `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`; for a multi-targeted repository a multi-SDK runner named `ubuntu-testrunner-8-9-10-11` is offered alongside them. The exact set comes from live metadata and the publisher's tag feed at runtime — never a hardcoded list. If the user names a WSL or SSH environment, the runner reports it as unsupported (Docker only for now). Relay that clearly instead of trying to convert it. ## Step 3: Plan when transparency helps -Before a long run — or whenever the developer wants to see what will happen — `plan` resolves the environment, validates the image tag against Microsoft's registry, pre-resolves the immutable digest, inspects target frameworks for SDK compatibility, and prints the deterministic execution plan without touching Docker: +`plan` is for when the developer asks what will happen — not a gate in front of a run they already asked for. Do not insert it before an unambiguous `run`. When it is called for, it resolves the environment, validates the image tag against Microsoft's registry, pre-resolves the immutable digest, inspects target frameworks for SDK compatibility, and prints the deterministic execution plan without touching Docker: ``` dotnet run --file "/scripts/remote-test.cs" -- plan --repo-root "" -e [-p ] [-c Release] --json @@ -85,42 +140,103 @@ Common scoping options (pass through only what the developer asked for): - `--filter ` — a `dotnet test --filter` expression (test class, trait, etc.). - `--test ` — shortcut for a fully-qualified-name filter (a single test or class). - `-c, --configuration ` — build configuration. -- `-f, --framework ` — restrict a multi-targeted test project to one TFM. +- `-f, --framework ` — restrict a multi-targeted test project to one TFM. This also narrows environment resolution, so the run lands on that TFM's single-SDK channel instead of a multi-SDK runner. - `--coverage` — collect coverage when the project already supports it (never add packages to enable it). - `--timeout ` — abort the run after N seconds; the runner still cleans up. The runner establishes an isolated staged workspace (so container builds never leave Linux `bin`/`obj` in the working tree), mounts a persistent NuGet cache outside the repository, pins the image to its digest, runs restore → build → test, collects TRX results, and removes all transient Docker resources afterward. You do not manage any of that. +Two diagnostic options exist for when a result needs explaining, not for routine runs: + +- `--show-log` — print the container's full restore/build/test log. Use it when the summarized failure detail is not enough, never by default. +- `--no-git-metadata` — skip staging `.git` (see [The staged workspace is still a repository](#the-staged-workspace-is-still-a-repository)). Only for a repository whose `.git` is large enough that copying it dominates the run, and only when the developer accepts the fidelity loss. + +### The staged workspace is still a repository + +The staged copy includes the repository's `.git` directory. This is not incidental: a .NET build and the code under test both read it. + +- MinVer, Nerdbank.GitVersioning and GitInfo derive the assembly version from git history. Without `.git` they silently fall back to `0.0.0`, so the container builds a differently-versioned assembly than the host. +- SourceLink stops embedding repository information. +- Application and test code commonly locates the repository root by walking up until a `.git` directory exists. Without it, that probe resolves somewhere else — and every path derived from it changes. + +That last one is why a suite can pass in Visual Studio's remote testing and fail here for reasons that have nothing to do with the container. If a run reports a failure that looks path-dependent, the staged workspace being a real repository is already accounted for; do not "fix" it by editing the repository. + +The runner installs `git` into the image when the image lacks it (see [Build tooling in the image](#build-tooling-in-the-image)) — the two halves belong together: the tooling and the metadata it reads. + +### Build tooling in the image + +The container runs the *build*, not just the tests, and a .NET build routinely shells out to `git` — MinVer, Nerdbank.GitVersioning, GitInfo and SourceLink all do. Microsoft's SDK images ship it; a minimal runner image may not, and the build then fails with `MINVER1007: "git" is not present in PATH` even though nothing is wrong with the code. + +The runner handles this: it probes the resolved image and, when the tooling is missing, layers it on in a cached image tagged `dotnet-remote-testing/prepared:git-` — built in the runner's own temp directory, never in the repository, and reused by every later run against the same base image. The reported `Image`/`Digest` stay the base image; preparation is reported on its own `Tools:` line. If it cannot be added (no package manager, `--offline`), the run proceeds on the base image and says so. + +Relay that line when present, but do not act on it: it is not a repository problem and never a reason to edit `testenvironments.json`, author a `Dockerfile`, or fall back to the host. A recurring `Added git to the image` for a repository's own image is worth mentioning once — the durable fix belongs in that image, not here. + ## Step 5: Report results concisely -Lead with the outcome, not the infrastructure. Mirror the runner's concise result and suppress pull/restore/build log noise unless something failed: +Lead with the outcome, not the infrastructure. Mirror the runner's result and suppress pull/restore/build log noise unless something failed. The runner already reports at the granularity `dotnet test` does — one line per test assembly and target framework, then the totals: ``` Remote Test: dotnet-10-lts +Selected automatically: the only environment matching the repository's target framework 'net10.0'. Image: mcr.microsoft.com/dotnet/sdk:10.0.302 Digest: sha256:... SDK: 10.0.302 +Passed! Cuemon.Core.Tests.dll (net10.0) — 1842 passed, 3 skipped, 0 failed, 21.8 s + Tests: 1842 passed, 3 skipped, 0 failed -Time: 21.8 s +Time: 21.8 s (tests) +Total: 96.4 s (including image pull, restore and build) ``` -When tests fail, prioritize actionable detail — the failing test, its class, the expected/actual message, and location — over container startup output: +Report both durations as the runner does. `Time` is the test execution time from the TRX; `Total` is wall clock for the whole operation. Collapsing them into one number misrepresents a fast suite behind a slow image pull. When the runner explains an automatic environment selection, relay that line — it is what makes an unattended choice auditable. + +When tests fail, relay the runner's failure detail as it stands. It is deliberately shaped like `dotnet test` output — fully-qualified test name, the target framework it failed under, the assertion message, the stack trace and anything the test wrote itself — because that is what makes a red test fixable without a second run: ``` -3 tests failed +Failed! Cuemon.Text.Tests.dll (net10.0) — 110 passed, 0 skipped, 1 failed, 2.0 s +Passed! Cuemon.Text.Tests.dll (net9.0) — 111 passed, 0 skipped, 0 failed, 1.9 s -Cuemon.Text.Tests.StringUtilityTest - Sanitize_WithUnicode_ReturnsExpectedValue - Expected: ... Actual: ... +1 test failed: + + Failed Cuemon.Text.Tests.StringUtilityTest.Sanitize_WithUnicode_ReturnsExpectedValue [net10.0] (314 ms) + Assert.Equal() Failure: Values differ + Expected: 16 + Actual: 2 + Stack trace: + at Cuemon.Text.Tests.StringUtilityTest.Sanitize_WithUnicode_ReturnsExpectedValue() in /workspace/test/…/StringUtilityTest.cs:line 321 ``` +Do not compress this into a bare count. "1 test failed" without the name, the TFM and the message forces the developer to rerun the suite to learn what you already know. Note which target framework failed when a multi-targeted project fails under one TFM and passes under another — that asymmetry is usually the diagnosis. If the detail is still not enough, rerun with `--show-log` rather than guessing. + A run is reproducible in terms of environment, requested image, resolved digest, SDK, architecture, and runner version; include the image and digest so the result can be reproduced. ## Failure handling -The runner distinguishes failure kinds via exit code and the `failureKind` field: `Configuration`, `UnsupportedEnvironment`, `DockerUnavailable`, `ImageResolution`, `SdkIncompatibility`, `SourceStaging`, `Restore`, `Compilation`, `TestHost`, `TestFailure`, `ResultProcessing`, `Cleanup`, `Cancelled`, and `ReleaseMetadataUnavailable`. Report the kind honestly: +**Branch on the exit code, never on the prose.** The exit code is the contract; log text is not. Every outcome maps to exactly one next action, so the same repository produces the same behavior regardless of which model is driving: + +| Exit | `failureKind` | What it means | Your next action | +|---:|---|---|---| +| `0` | — | Tests passed | Report the result | +| `1` | `TestFailure` | Real failing tests | Report the failures — this is **not** an infrastructure problem | +| `2` | — | Invalid arguments | Fix your command line; do not report it as a repository problem | +| `3` | `Configuration` | `testenvironments.json` unusable | Report the diagnostic; never repair the file unprompted | +| `4` | `UnsupportedEnvironment` | WSL/SSH environment named | Report Docker-only; never convert it | +| `5` | `DockerUnavailable` | Docker missing or not running | Report it; **never** fall back to the host | +| `6` | `ImageResolution` | Tag/digest/pull failed | Report the image and the registry error | +| `7` | `SdkIncompatibility` | SDK cannot build the target frameworks | Report the reason; never edit the repository to force it | +| `8` | `SourceStaging` | Workspace could not be staged | Report it as infrastructure | +| `9` | `Restore` | `dotnet restore` failed in the container | Report the restore error, not "tests failed" | +| `10` | `Compilation` | Build failed in the container | Report the compiler errors, not "tests failed" | +| `11` | `TestHost` | Test host crashed | Report as infrastructure with the output tail | +| `12` | `ResultProcessing` | Results unreadable | Report it; results are unknown, not passing | +| `13` | `Cleanup` | Transient resources left behind | Report the exact identifiers the runner names | +| `14` | `Cancelled` | Timeout or interrupt | Report how far it got | +| `15` | `ReleaseMetadataUnavailable` | Release index unreachable, no cache | Report it; suggest `--offline --cache-root` or a named environment | +| `16` | `SelectionRequired` | A real choice remains | **Ask one question** from `candidates`, then rerun with `-e ` | + +`SelectionRequired` is the only exit code that is a question rather than a report. Every other non-zero code is an outcome you relay honestly: - A container/infrastructure problem (image pull, restore, build, test-host crash) is **not** a failing unit test — say which phase failed. - If cleanup leaves resources behind, relay the exact resource identifiers the runner reports. @@ -128,9 +244,10 @@ The runner distinguishes failure kinds via exit code and the `failureKind` field ## What this skill must never do -- Generate a `Dockerfile`, dev container, editor config, or any repository-specific plumbing. (Honor an *existing* configured `dockerFile`; never create one.) +- Generate a `Dockerfile`, dev container, editor config, or any repository-specific plumbing. (Honor an *existing* configured `dockerFile`; never create one. The runner's own cached preparation layer is not repository plumbing and is not yours to write.) - Run privileged containers, mount the Docker socket, mount the whole user profile, forward host credentials indiscriminately, disable TLS validation, expose ports, or print secrets. The runner already avoids these; do not add them. -- Substitute third-party or unofficial images for auto-generated environments (only `mcr.microsoft.com/dotnet/sdk`). An explicit `dockerImage` in `testenvironments.json` is exempt because it is deliberate. +- Answer an invocation with a menu of its own capabilities, a "what would you like me to help you with?" opener, or a confirmation prompt for a run the developer already asked for. +- Reach for an arbitrary image when a recommended one fits. Auto-generated environments use `mcr.microsoft.com/dotnet/sdk` for a single .NET major and `codebeltnet/ubuntu-testrunner` for several; an explicit `dockerImage` in `testenvironments.json` is deliberate intent and is used exactly as written. Other images are permitted but must be a deliberate, stated choice — never a substitution you make on your own. - Fall back to running tests locally. ## References diff --git a/skills/dotnet-remote-testing/evals/evals.json b/skills/dotnet-remote-testing/evals/evals.json index 9302d53..92412e1 100644 --- a/skills/dotnet-remote-testing/evals/evals.json +++ b/skills/dotnet-remote-testing/evals/evals.json @@ -103,6 +103,144 @@ "files": [ "evals/files/offline-cache/cache/releases-index.cache.json" ] + }, + { + "id": 7, + "prompt": "/dotnet-remote-testing", + "expected_output": "The skill treats the bare invocation as a complete request and immediately runs the tests through scripts/remote-test.cs against the single configured Docker environment, reporting the structured result. It does not answer with a menu of capabilities or ask the developer what they would like to do.", + "expectations": [ + "Does NOT respond with a capability menu such as 'list / plan / run / understand your configuration' or 'What would you like me to help you with?'", + "Does NOT ask which environment, scope, configuration, or coverage to use, and does not ask for confirmation before running", + "Runs the tests immediately via scripts/remote-test.cs run against the single configured Docker environment", + "Reports the structured result (environment, image, digest, passed/skipped/failed, durations)", + "Asks a question only if the runner exits SelectionRequired, which does not happen with a single configured environment" + ], + "files": [ + "evals/files/configured/testenvironments.json", + "evals/files/configured/test/Api.Tests/Api.Tests.csproj", + "evals/files/configured/test/Api.Tests/HealthTests.cs" + ] + }, + { + "id": 8, + "prompt": "Remote test this repo.", + "expected_output": "With no testenvironments.json and several Microsoft-derived channels available, the runner selects the channel matching the repository's own target framework (net10.0) without asking, runs the tests in that container, and relays the reported selection reason so the automatic choice is auditable.", + "expectations": [ + "Does not ask which .NET channel to use even though several are derived from release metadata", + "Ends up on the derived environment whose channel major matches the repository's highest .NET target framework (net10.0)", + "Relays the runner's selection reason explaining why that environment was chosen automatically", + "Executes through scripts/remote-test.cs run rather than composing docker commands or running dotnet test on the host", + "Reports both the test duration and the total elapsed time as the runner does" + ], + "files": [ + "evals/files/zero-config/Sample.slnx", + "evals/files/zero-config/src/Sample/Sample.csproj", + "evals/files/zero-config/src/Sample/Calculator.cs", + "evals/files/zero-config/test/Sample.Tests/Sample.Tests.csproj", + "evals/files/zero-config/test/Sample.Tests/CalculatorTests.cs" + ] + }, + { + "id": 9, + "prompt": "Run my tests in a container.", + "expected_output": "Two Docker environments are configured, so the runner exits SelectionRequired and the skill asks exactly one question listing the two candidate names, then reruns with the chosen environment. It does not guess between them and does not expand the interruption into a broader questionnaire.", + "expectations": [ + "Attempts the run first and lets the runner report SelectionRequired rather than pre-emptively interviewing the developer", + "Asks exactly one question, offering only the configured candidate names (linux-dotnet-10-noble, linux-dotnet-10-alpine)", + "Does not additionally ask about scope, build configuration, coverage, or confirmation", + "Does not guess or silently pick one of the two configured environments", + "Reruns with -e after the developer picks" + ], + "files": [ + "evals/files/multi-configured/testenvironments.json", + "evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj", + "evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs" + ] + }, + { + "id": 10, + "prompt": "Remote test this repo.", + "expected_output": "The test project multi-targets net9.0 and net10.0. A Microsoft SDK image ships only one runtime, so the skill resolves a codebeltnet/ubuntu-testrunner multi-SDK environment that provides both majors and runs the entire target-framework matrix in a single container, reporting results for both TFMs.", + "expectations": [ + "Recognizes that the repository multi-targets several .NET majors", + "Does not select a single-SDK mcr.microsoft.com/dotnet/sdk image, which could build but not execute the lower target framework", + "Resolves a codebeltnet/ubuntu-testrunner environment whose combined tag covers both net9.0 and net10.0", + "Runs the whole matrix in one container rather than one container per target framework", + "Reports the automatic selection reason and results covering both target frameworks", + "Does not ask which environment to use, because the target frameworks determine it" + ], + "files": [ + "evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs" + ] + }, + { + "id": 11, + "prompt": "Run only the net10.0 tests in a container.", + "expected_output": "The skill passes --framework net10.0, which narrows both the run and the environment choice, so it resolves the ordinary Microsoft SDK channel for .NET 10 rather than a multi-SDK runner, and executes only the net10.0 target framework.", + "expectations": [ + "Passes -f/--framework net10.0 through to the runner rather than filtering results afterwards", + "Resolves the single-SDK Microsoft channel for .NET 10 because the run is narrowed to one target framework", + "Does not select a multi-SDK runner for a run restricted to a single target framework", + "Does not ask which environment to use", + "Reports results for the net10.0 target framework only" + ], + "files": [ + "evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs" + ] + }, + { + "id": 12, + "prompt": "Remote test this repo.", + "expected_output": "The runner exits 1 with real failing tests. The skill reports them the way dotnet test would: which test assembly and target framework failed, the fully-qualified test name, the assertion message and the stack trace — and treats it as a test outcome to report, not an infrastructure problem and not a question.", + "expectations": [ + "Reports exit code 1 as a genuine test failure rather than an infrastructure or container problem", + "Names the failing test by its fully-qualified name and the target framework it failed under", + "Includes the assertion message and stack trace the runner reported instead of only a failure count", + "Distinguishes the failing test assembly/TFM from the ones that passed when a multi-targeted project fails under only one TFM", + "Does not rerun on the host, edit the repository, or modify the test to make it pass", + "Does not ask a question; a test failure is an outcome to report" + ], + "files": [ + "evals/files/multi-targeted-failing/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted-failing/test/Matrix.Tests/MatrixTests.cs" + ] + }, + { + "id": 13, + "prompt": "These tests pass under Visual Studio's remote testing against the same Docker image, but fail when I run them through you. Why?", + "expected_output": "The skill explains from the runner's own behavior rather than changing the repository: the staged workspace is a disposable copy that includes the repository's .git directory, so repository-root detection, MinVer/Nerdbank version stamping and SourceLink behave as they do on the host. It investigates the actual reported failure instead of assuming the container is at fault, and never edits the repository to force a pass.", + "expectations": [ + "Treats the difference as something to diagnose from the reported failure, not a reason to fall back to the host", + "Knows the staged workspace includes .git, so repository-root probes and version-deriving tools behave as on the host", + "Does not edit global.json, project files, target frameworks, test packages, or testenvironments.json to force a pass", + "Does not create a Dockerfile, dev container, or other repository plumbing", + "Offers --show-log for the full container log rather than guessing at the cause" + ], + "workspace": { + "git": true + }, + "files": [ + "evals/files/multi-targeted-failing/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted-failing/test/Matrix.Tests/MatrixTests.cs" + ] + }, + { + "id": 14, + "prompt": "The persistent NuGet cache contains entries owned by an older container UID. Remote test this repo and make sure a cache permission problem does not break restore.", + "expected_output": "The runner detects that the old cache cannot be reconciled, preserves it under a nuget-stale-* quarantine name, creates a fresh writable cache, and continues the remote restore/build/test run without falling back to the host.", + "expectations": [ + "Does not silently continue with an inaccessible foreign-owned cache", + "Preserves the unrecoverable old cache and creates a fresh writable cache", + "Keeps newly-created cache entries writable across later container UIDs", + "Continues to execute restore/build/test inside Docker and never falls back to the host" + ], + "files": [ + "evals/files/configured/testenvironments.json", + "evals/files/configured/test/Api.Tests/Api.Tests.csproj", + "evals/files/configured/test/Api.Tests/HealthTests.cs" + ] } ] } diff --git a/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj new file mode 100644 index 0000000..946be49 --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj @@ -0,0 +1,12 @@ + + + net10.0 + enable + false + + + + + + + diff --git a/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs new file mode 100644 index 0000000..e5c176c --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs @@ -0,0 +1,9 @@ +using Xunit; + +namespace Catalog.Tests; + +public class CatalogTests +{ + [Fact] + public void Lookup_ReturnsExpectedValue() => Assert.Equal("OK", "OK"); +} diff --git a/skills/dotnet-remote-testing/evals/files/multi-configured/testenvironments.json b/skills/dotnet-remote-testing/evals/files/multi-configured/testenvironments.json new file mode 100644 index 0000000..24d6d0a --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-configured/testenvironments.json @@ -0,0 +1,15 @@ +{ + "version": "1", + "environments": [ + { + "name": "linux-dotnet-10-noble", + "type": "docker", + "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-noble" + }, + { + "name": "linux-dotnet-10-alpine", + "type": "docker", + "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-alpine" + } + ] +} diff --git a/skills/dotnet-remote-testing/evals/files/multi-targeted-failing/test/Matrix.Tests/Matrix.Tests.csproj b/skills/dotnet-remote-testing/evals/files/multi-targeted-failing/test/Matrix.Tests/Matrix.Tests.csproj new file mode 100644 index 0000000..a12326d --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-targeted-failing/test/Matrix.Tests/Matrix.Tests.csproj @@ -0,0 +1,12 @@ + + + net9.0;net10.0 + enable + false + + + + + + + diff --git a/skills/dotnet-remote-testing/evals/files/multi-targeted-failing/test/Matrix.Tests/MatrixTests.cs b/skills/dotnet-remote-testing/evals/files/multi-targeted-failing/test/Matrix.Tests/MatrixTests.cs new file mode 100644 index 0000000..b3ac7c8 --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-targeted-failing/test/Matrix.Tests/MatrixTests.cs @@ -0,0 +1,21 @@ +using Xunit; + +namespace Matrix.Tests; + +public class MatrixTests +{ + [Fact] + public void RunsOnEveryTargetFramework() + { + // This test passes under net10.0 and fails under net9.0 on purpose, so a remote run + // surfaces a genuine, per-target-framework failure with a real assertion message and + // stack trace to report - one TFM failing while the other passes. + var expected = "net10.0"; +#if NET9_0 + var actual = "net9.0"; +#else + var actual = "net10.0"; +#endif + Assert.Equal(expected, actual); + } +} diff --git a/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj new file mode 100644 index 0000000..3e05e2e --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj @@ -0,0 +1,12 @@ + + + net9.0;net10.0 + enable + false + + + + + + + diff --git a/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs new file mode 100644 index 0000000..1b910fc --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs @@ -0,0 +1,9 @@ +using Xunit; + +namespace Matrix.Tests; + +public class MatrixTests +{ + [Fact] + public void RunsOnEveryTargetFramework() => Assert.Equal("OK", "OK"); +} diff --git a/skills/dotnet-remote-testing/references/docker-execution.md b/skills/dotnet-remote-testing/references/docker-execution.md index b174f16..4b80773 100644 --- a/skills/dotnet-remote-testing/references/docker-execution.md +++ b/skills/dotnet-remote-testing/references/docker-execution.md @@ -9,11 +9,12 @@ Every `run` follows the same deterministic sequence: 1. Validate Docker availability. If unavailable, exit `DockerUnavailable` — never fall back to the host. 2. Resolve the environment (configured or Microsoft-derived). 3. Resolve the image and pin it to an immutable digest. -4. Establish an isolated, disposable staged workspace. -5. Prepare deterministic mounts and caches. -6. Execute restore → build → test. -7. Collect structured results (TRX). -8. Clean up transient resources. +4. Ensure the image carries the build tooling (see [Image preparation](#image-preparation)). +5. Establish an isolated, disposable staged workspace. +6. Prepare deterministic mounts and caches. +7. Execute restore → build → test. +8. Collect structured results (TRX). +9. Clean up transient resources. Cleanup is mandatory after success, test failure, build failure, restore failure, cancellation, and exceptions. If cleanup fails, the exact remaining Docker resource identifiers are reported. @@ -23,6 +24,45 @@ Container execution must not pollute or mutate the developer's working tree. The The runner never creates `Dockerfile`, `docker-compose.yml`/`compose.yml`, `.devcontainer/`, `.vscode/`, `Directory.Build.*`, temporary scripts, or generated test configuration in the repository. +## Image preparation + +The container runs the *build*, not only the tests, and a .NET build routinely shells out to host tooling: MinVer, Nerdbank.GitVersioning, GitInfo and SourceLink all invoke `git` during build. An image without it fails the build — MinVer reports `MINVER1007: "git" is not present in PATH` — even though the code is perfectly fine, which surfaces a missing tool as if it were a repository problem. + +Microsoft's SDK images ship `git`; a minimal runner image may not. So after the image is resolved and pinned, the runner probes it (`command -v git`) and, when the tooling is missing, derives a single layer from the resolved base: + +```dockerfile +FROM +USER root +RUN … install git with whichever package manager the base image ships (apt-get / apk / microdnf / dnf / yum) +USER +``` + +This preparation layer is: + +- **outside the repository** — the Dockerfile is written into the run's own temporary directory, never into the working tree, so the "never generate container plumbing" rule is intact; +- **content-addressed and cached** — tagged `dotnet-remote-testing/prepared:git-`, so it is built once per base image and reused by every later run (`Reused prepared image providing git.`); +- **transparent** — the reported `Image`/`Digest` remain the resolved base image (reproducibility identity), with the preparation reported separately; +- **identity-preserving** — installing packages needs root, but the image's own `USER` is restored afterwards, so a base image that runs as a non-root user keeps doing so and file ownership and permission-sensitive tests behave as they do in the configured image; +- **best effort** — if the tooling cannot be added (no package manager, `--offline`, no network), the base image is used anyway and the reason is reported, because a repository that never invokes `git` runs fine without it. + +## Git metadata in the workspace + +The repository's `.git` directory is copied into the staged workspace alongside the source. The copy is disposable, so the container may write to it freely; the developer's real repository is never mounted and never touched. + +This matters because staging the source without its git metadata is not a neutral omission — it changes observable build and test behavior: + +| Consumer | Without `.git` | +|---|---| +| MinVer / Nerdbank.GitVersioning / GitInfo | Silently fall back to `0.0.0`, so the container builds differently-versioned assemblies than the host | +| SourceLink | Stops embedding repository information | +| Repository-root probes (`walk up until a .git directory exists`) | Resolve to a different directory, changing every path derived from them — including paths the tests under it read | + +The third row is the subtle one: a suite that passes under Visual Studio's Remote Testing and fails here, for reasons unrelated to the container, is very often a repository-root probe landing somewhere else. Installing `git` into the image (above) and staging `.git` are two halves of one guarantee: the build tooling is present *and* the metadata it reads is present. + +A linked worktree or submodule stores `.git` as a `gitdir: ` pointer file; the runner resolves the pointer and stages the real directory, because the path it names does not exist inside the container. A linked worktree's git directory is only half a repository — it holds per-worktree state (`HEAD`, `index`, logs) and points at a shared `commondir` that holds the objects, refs and config — so the runner stages the shared half first, layers the per-worktree files over it, and drops the `commondir`/`gitdir` pointers. The staged copy is then an ordinary standalone repository, which is what git-based versioning and SourceLink need; staging the near half alone would leave a git directory git cannot read, and versioning would fall back to `0.0.0` exactly as if `.git` had been skipped. A source tree with no git metadata at all stages normally and silently — that is an ordinary case, not a degraded one. + +`--no-git-metadata` opts out for a repository whose `.git` is large enough that copying it dominates the run. The run then reports the fidelity loss rather than hiding it. + ## Mounts Three bind mounts, nothing more: @@ -33,7 +73,7 @@ Three bind mounts, nothing more: | NuGet cache | `/nuget` | Persistent package cache owned outside the repository | Yes | | Results | `/results` | TRX output read back by the host | No (removed after the run) | -The dependency cache (`/nuget`, via `NUGET_PACKAGES`) is deliberately separated from the per-execution build/test workspace: the immutable, reusable dependency cache may persist for fast feedback, while the build/test workspace is isolated per execution so results never depend on stale source. +The dependency cache (`/nuget`, via `NUGET_PACKAGES`) is deliberately separated from the per-execution build/test workspace: the immutable, reusable dependency cache may persist for fast feedback, while the build/test workspace is isolated per execution so results never depend on stale source. `NUGET_PACKAGES` is exported with a trailing slash (`/nuget/`) because NuGet's package root becomes an MSBuild `SourceRoot`, and SourceLink fails the build on a `SourceRoot` that does not end in a separator. The in-container `umask 000` keeps newly-created entries usable by a later run with a different UID. Before mounting, the host adds the required write bits; if an older foreign-owned cache cannot be reconciled, the runner preserves it under a `nuget-stale-*` name and starts a fresh cache instead of letting restore fail later with an opaque permission error. ## In-container phases @@ -53,13 +93,21 @@ Supported through options that pass straight into the phases: entire solution, a ## Result collection and classification -The runner parses every TRX in `/results` (multi-targeted projects emit one per TFM) into a single structured result: passed, skipped, failed counts, duration, and actionable failure detail (test name, class, message, location). It prioritizes machine-readable results and suppresses pull/restore/build noise on success. +The runner parses every TRX in `/results` into a single structured result, at the granularity `dotnet test` itself reports: + +- **Per test assembly and target framework** — a multi-targeted project emits one TRX per TFM, and the assembly path recorded in the TRX is the only thing that distinguishes them. Each becomes its own `Passed!`/`Failed!` line with its own counts and duration, so a failure is attributable to one test project under one TFM instead of a pooled total. +- **Aggregate counts and duration** across every assembly. +- **Per failing test**: fully-qualified name, owning class, target framework, elapsed time, assertion message, stack trace, and anything the test wrote to its own output helper. Detail is capped (15 failures, 10 stack frames, 15 output lines) so a wholesale failure stays readable; `--json` always carries the complete set. + +Pull/restore/build noise is suppressed on success. `--show-log` prints the container log in full when the summarized detail is not enough. Failures are classified into distinct kinds so a container/infrastructure problem is never misreported as a failing unit test: `Configuration`, `UnsupportedEnvironment`, `DockerUnavailable`, `ImageResolution`, `SdkIncompatibility`, `SourceStaging`, `Restore`, `Compilation`, `TestHost`, `TestFailure`, `ResultProcessing`, `Cleanup`, `Cancelled`, `ReleaseMetadataUnavailable`. A non-zero `dotnet test` exit with a TRX containing failures is a `TestFailure`; a non-zero exit with no failing results (crash, no discovered tests, missing adapter) is a `TestHost` failure. +Each phase emits a machine-readable end marker, so the log between two markers is exactly that phase's output. An infrastructure failure is reported with *its own* phase's log rather than a tail of everything — a build failure names the offending file and compiler error instead of trailing test-runner chatter. Any failing tests already recorded in a TRX are reported first even when the phase failed for another reason, so an assertion failure followed by a test-host crash does not disappear behind the crash. + ## Cancellation and cleanup -The container is given a deterministic, knowable name so it can always be targeted for cleanup — even after Ctrl+C or a `--timeout`. On cancellation the runner force-removes the container and deletes the staged workspace and results directory; the persistent NuGet cache is kept. `docker run --rm` also auto-removes the container on normal completion. +The container is given a deterministic, knowable name so it can always be targeted for cleanup — even after Ctrl+C or a `--timeout`. On cancellation the runner force-removes the container and deletes the staged workspace and results directory; the persistent NuGet cache and any cached preparation image are kept — both are reusable assets, not leftovers. `docker run --rm` also auto-removes the container on normal completion. ## Security posture diff --git a/skills/dotnet-remote-testing/references/release-discovery.md b/skills/dotnet-remote-testing/references/release-discovery.md index 601db1b..eb2b0d7 100644 --- a/skills/dotnet-remote-testing/references/release-discovery.md +++ b/skills/dotnet-remote-testing/references/release-discovery.md @@ -37,7 +37,20 @@ The runner prefers an **exact SDK-version image tag** derived from `latest-sdk` Because a version string does not always transform mechanically into a valid tag, the selected tag is validated against Microsoft's official SDK image metadata (the `mcr.microsoft.com` registry) before execution. If the exact tag is unavailable, the channel tag (`10.0`) is tried as a fallback candidate. -Third-party images, unofficial Docker Hub images, and locally discovered look-alike images are never substituted for auto-generated environments. An explicit `dockerImage` in `testenvironments.json` is the only exception, because it is deliberate configuration. +Auto-generated environments use one of the two recommended publishers — `mcr.microsoft.com/dotnet/sdk` for a single .NET major, or `codebeltnet/ubuntu-testrunner` when several majors must be present at once (see below). Other images are not forbidden, but they are never substituted on the runner's own initiative: an image outside those two comes from an explicit `dockerImage` in `testenvironments.json`, which is deliberate configuration and is used as written. `plan` reports `image.recommendedPublisher` so the provenance of any image is visible. + +## Multi-SDK runners for multi-targeted repositories + +A Microsoft SDK image contains exactly one runtime. `mcr.microsoft.com/dotnet/sdk:10.0` provides `Microsoft.NETCore.App 10.0.x` and nothing else, so a repository targeting `net9.0;net10.0` compiles both target frameworks there and then cannot execute the `net9.0` tests. Building is not running. + +`codebeltnet/ubuntu-testrunner` publishes combined tags carrying several SDKs — `8-9-10-11` provides .NET 8, 9, 10 and 11 in a single image — so the whole target-framework matrix runs in one container rather than one container per TFM. + +- Tags are discovered at runtime from the publisher's tag feed (`https://hub.docker.com/v2/repositories/codebeltnet/ubuntu-testrunner/tags`), cached outside the repository like release metadata, and overridable with `--multi-sdk-tags-file` for offline or deterministic runs. +- Only the **major-only combined form** (`8-9-10-11`) is used. Single-major tags are already covered by Microsoft's images, and pinned combination forms move with each patch. +- The **tightest covering tag wins**: the fewest extra SDKs that still provide every required major, breaking ties on the tag name so resolution is stable. +- A multi-SDK environment declares its majors explicitly, and compatibility is judged on that list rather than on a single SDK version. +- Pointing a multi-targeted repository at a single-SDK image is reported as an SDK incompatibility naming the unrunnable target frameworks and the remedy, instead of starting a run that cannot finish. +- `--framework` narrows the environment choice as well as the run, so restricting to one TFM resolves the ordinary single-SDK channel. ## Immutable image identity @@ -60,6 +73,29 @@ The runner inspects the solution/projects being tested and will not select an SD Multi-targeted projects are accounted for. The runner never edits `global.json`, project files, or target frameworks to make remote testing succeed. +### Target frameworks also resolve the environment + +Compatibility is not the only use of this inspection. When several environments are derived and the caller named no environment, the repository's own target frameworks select one outright, so zero-configuration testing runs unattended instead of stopping to ask which .NET to use. + +- **One .NET major** → the derived channel matching it exactly. +- **Several .NET majors** → the multi-SDK runner providing all of them, because every runtime must be present for the tests to execute. + +The rule is exact-match by design and never approximates: + +| Repository targets | Available | Outcome | +|---|---|---| +| `net10.0` | channels 8, 9, 10, 11-preview | `dotnet-10-lts` selected automatically | +| `net11.0` | channels 8, 9, 10, 11-preview | `dotnet-11-preview` selected | +| `net9.0;net10.0` | channels + runner tags `9-10`, `8-9-10-11` | `ubuntu-testrunner-9-10` selected (tightest cover) | +| `net8.0;net10.0` | channels + runner tag `8-9-10-11` | `ubuntu-testrunner-8-9-10-11` selected | +| `net9.0;net10.0` with `-f net10.0` | channels 8, 9, 10, 11-preview | `dotnet-10-lts` — the run was narrowed to one TFM | +| `net8.0;net10.0` | no covering runner tag | `SelectionRequired` — never a single-SDK image that cannot run both | +| `net7.0` (EOL) | channels 8, 9, 10, 11-preview | `SelectionRequired` — no matching channel | +| `netstandard2.0` only | channels 8, 9, 10, 11-preview | `SelectionRequired` — no .NET target to match | +| `net10.0` | two channels for major 10 | `SelectionRequired` — the match is not unique | + +The selection is reported in both human and JSON output (`environment.selectionReason`) so an unattended choice remains auditable. This applies only to derived environments; a `testenvironments.json` with several Docker entries is deliberate developer intent and always asks. + ## Offline behavior and caching Successfully retrieved release metadata is cached outside the repository together with the retrieval timestamp. When Microsoft cannot be reached: diff --git a/skills/dotnet-remote-testing/references/testenvironments-json.md b/skills/dotnet-remote-testing/references/testenvironments-json.md index caa47c3..4e2f09e 100644 --- a/skills/dotnet-remote-testing/references/testenvironments-json.md +++ b/skills/dotnet-remote-testing/references/testenvironments-json.md @@ -44,7 +44,7 @@ Following Microsoft's rule, a `docker` environment must specify **either** `dock ### Configured images are deliberate -An explicit `dockerImage` in `testenvironments.json` is exempt from the Microsoft-only restriction that governs auto-generated environments, because it represents intentional repository configuration. The runner uses it as written (after pulling and resolving its digest). Auto-generated environments, by contrast, always use `mcr.microsoft.com/dotnet/sdk`. +An explicit `dockerImage` in `testenvironments.json` is intentional repository configuration, so the runner uses it as written (after pulling and resolving its digest) whatever its publisher. Auto-generated environments come from the two recommended publishers instead: `mcr.microsoft.com/dotnet/sdk` for a single .NET major, and `codebeltnet/ubuntu-testrunner` when the repository multi-targets several majors and needs all their runtimes in one image. ### Configured Dockerfiles are honored, never created diff --git a/skills/dotnet-remote-testing/scripts/remote-test.cs b/skills/dotnet-remote-testing/scripts/remote-test.cs index c6a4154..7df7f5a 100644 --- a/skills/dotnet-remote-testing/scripts/remote-test.cs +++ b/skills/dotnet-remote-testing/scripts/remote-test.cs @@ -36,6 +36,14 @@ internal static class RemoteTestProgram internal const string ReleasesIndexUrl = "https://raw.githubusercontent.com/dotnet/core/refs/heads/main/release-notes/releases-index.json"; + // Microsoft's SDK images carry exactly one runtime, so a repository that multi-targets several .NET + // majors cannot execute its lower target frameworks there — it builds, then fails for want of a + // runtime. The Codebelt test runner ships several SDKs in one image (tags such as "8-9-10-11"), so + // one container covers every target framework in a single run. + internal const string MultiSdkRepository = "codebeltnet/ubuntu-testrunner"; + internal const string MultiSdkTagsUrl = + "https://hub.docker.com/v2/repositories/codebeltnet/ubuntu-testrunner/tags?page_size=100"; + internal static readonly JsonSerializerOptions JsonOut = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase, @@ -140,6 +148,7 @@ internal sealed class Options public string? ConfigPath { get; private set; } public string? EnvironmentName { get; private set; } public string? ReleasesIndexFile { get; private set; } + public string? MultiSdkTagsFile { get; private set; } public string? CacheRoot { get; private set; } // Test scoping. These flow into the container command plan; they never mutate the repository. @@ -151,6 +160,10 @@ internal sealed class Options public bool Coverage { get; private set; } public int TimeoutSeconds { get; private set; } + // Staging/reporting fidelity switches. Defaults reproduce what the developer sees locally. + public bool NoGitMetadata { get; private set; } + public bool ShowLog { get; private set; } + public static Options Parse(string[] args) { var o = new Options(); @@ -168,10 +181,13 @@ public static Options Parse(string[] args) case "--offline": o.Offline = true; break; case "--no-registry-check": o.NoRegistryCheck = true; break; case "--coverage": o.Coverage = true; break; + case "--no-git-metadata": o.NoGitMetadata = true; break; + case "--show-log": o.ShowLog = true; break; case "--repo-root": o.RepoRoot = Path.GetFullPath(Next(args, ref i, a)); break; case "--config-path": o.ConfigPath = Next(args, ref i, a); break; case "--environment" or "-e": o.EnvironmentName = Next(args, ref i, a); break; case "--releases-index-file": o.ReleasesIndexFile = Next(args, ref i, a); break; + case "--multi-sdk-tags-file": o.MultiSdkTagsFile = Next(args, ref i, a); break; case "--cache-root": o.CacheRoot = Next(args, ref i, a); break; case "--project" or "-p": o.Project = Next(args, ref i, a); break; case "--filter": o.Filter = Next(args, ref i, a); break; @@ -258,15 +274,20 @@ Test scoping (plan/run): -f, --framework Restrict multi-targeted test projects to one TFM. --coverage Collect code coverage (XPlat Code Coverage) when the project supports it. --timeout Abort the container run after N seconds (0 = no timeout). + --no-git-metadata Do not stage .git into the workspace (faster for a very large + repository, but repository-root detection, MinVer/Nerdbank + versioning and SourceLink will differ from the host). Release discovery / networking: --offline Use cached release metadata only; never reach the network. --no-registry-check Skip Docker registry tag validation and digest pre-resolution. --releases-index-file Load Microsoft release metadata from a local file instead of the network. + --multi-sdk-tags-file Load Codebelt multi-SDK runner tags from a local file instead of the network. --cache-root Override the metadata/NuGet cache root (outside the repository). Output: --json Emit machine-readable JSON. + --show-log Print the container's restore/build/test log in full. -h, --help Show this help. Exit codes: 0 success, 1 test failures, 2 invalid args, 3 configuration, 4 unsupported environment, @@ -610,6 +631,181 @@ public static IReadOnlyList CandidateTags(SdkVersion sdk) public static string SdkImageReference(string tag) => $"{RemoteTestProgram.SdkRepository}:{tag}"; } +// --------------------------------------------------------------------------------------------------- +// Multi-SDK runner discovery (codebeltnet/ubuntu-testrunner). +// +// A Microsoft SDK image contains one runtime. That is fine for a single-target repository, but a +// repository multi-targeting several .NET majors can only *build* the lower targets there — executing +// their tests needs the matching runtimes. The Codebelt runner publishes combined tags ("8-9-10-11") +// carrying several SDKs, so the whole target-framework matrix runs in one container. +// +// The available tags are discovered from the published tag feed at runtime. Nothing here is hardcoded: +// when a new major joins the combined tags, it is picked up without a skill change. +// --------------------------------------------------------------------------------------------------- + +internal sealed record MultiSdkRunner +{ + public required string Tag { get; init; } + public IReadOnlyList Majors { get; init; } = []; + + public string Reference => $"{RemoteTestProgram.MultiSdkRepository}:{Tag}"; + + public bool Covers(IReadOnlyList requiredMajors) => requiredMajors.All(Majors.Contains); +} + +internal static class MultiSdkTagReader +{ + // Only the major-only combined form ("8-9-10-11") is used. It is a moving tag that tracks the + // current patch of each major, and it is the form the publisher documents for consumers. Single + // majors ("10"), channel forms ("10.0"), and fully pinned combinations are deliberately ignored + // here — single majors are already covered by Microsoft's images. + private static readonly Regex CombinedMajorTag = new(@"^\d+(?:-\d+)+$", RegexOptions.Compiled); + + public static IReadOnlyList Parse(string json) + { + var runners = new List(); + JsonDocument document; + try + { + document = JsonDocument.Parse(json); + } + catch (JsonException) + { + return runners; + } + + using (document) + { + if (!document.RootElement.TryGetProperty("results", out var results) || + results.ValueKind != JsonValueKind.Array) + { + return runners; + } + + foreach (var entry in results.EnumerateArray()) + { + if (!entry.TryGetProperty("name", out var nameElement) || + nameElement.GetString() is not { } name || + !CombinedMajorTag.IsMatch(name)) + { + continue; + } + + var majors = new List(); + var usable = true; + foreach (var part in name.Split('-')) + { + if (int.TryParse(part, NumberStyles.Integer, CultureInfo.InvariantCulture, out var major) && major > 0) + { + majors.Add(major); + } + else + { + usable = false; + break; + } + } + + if (usable && majors.Count > 1) + { + runners.Add(new MultiSdkRunner { Tag = name, Majors = [.. majors.Distinct().OrderBy(m => m)] }); + } + } + } + + return runners; + } + + // Tightest fit wins: the fewest extra SDKs that still cover every required major. Ties break on the + // tag name so the same repository always resolves to the same image. + public static MultiSdkRunner? Select(IReadOnlyList runners, IReadOnlyList requiredMajors) + { + if (requiredMajors.Count < 2) + { + return null; + } + + return runners + .Where(r => r.Covers(requiredMajors)) + .OrderBy(r => r.Majors.Count) + .ThenBy(r => r.Tag, StringComparer.Ordinal) + .FirstOrDefault(); + } +} + +internal sealed record MultiSdkResult(IReadOnlyList Runners, string? Error); + +internal static class MultiSdkRunnerStore +{ + private static readonly HttpClient Http = new() { Timeout = TimeSpan.FromSeconds(15) }; + + private static string CacheFile(string cacheRoot) => Path.Combine(cacheRoot, "multi-sdk-tags.cache.json"); + + public static async Task LoadAsync(Options options, CancellationToken ct) + { + // An explicit local file is a deliberate input (used by the deterministic test harness). + if (!string.IsNullOrWhiteSpace(options.MultiSdkTagsFile)) + { + return File.Exists(options.MultiSdkTagsFile) + ? new MultiSdkResult(MultiSdkTagReader.Parse(await File.ReadAllTextAsync(options.MultiSdkTagsFile, ct)), null) + : new MultiSdkResult([], $"multi-SDK tags file not found: {options.MultiSdkTagsFile}"); + } + + var cacheFile = CacheFile(options.CacheDirectory); + if (options.Offline) + { + return LoadFromCache(cacheFile, "Offline mode: "); + } + + try + { + var json = await Http.GetStringAsync(RemoteTestProgram.MultiSdkTagsUrl, ct); + var runners = MultiSdkTagReader.Parse(json); + if (runners.Count == 0) + { + return LoadFromCache(cacheFile, "Multi-SDK tag feed returned no combined tags: "); + } + + TryWriteCache(cacheFile, json); + return new MultiSdkResult(runners, null); + } + catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException or IOException) + { + return LoadFromCache(cacheFile, $"Could not reach the multi-SDK tag feed ({ex.Message}); "); + } + } + + private static MultiSdkResult LoadFromCache(string cacheFile, string prefix) + { + if (!File.Exists(cacheFile)) + { + return new MultiSdkResult([], prefix + "no cached multi-SDK runner tags are available."); + } + + try + { + return new MultiSdkResult(MultiSdkTagReader.Parse(File.ReadAllText(cacheFile)), null); + } + catch (IOException ex) + { + return new MultiSdkResult([], prefix + $"the cached multi-SDK runner tags could not be read ({ex.Message})."); + } + } + + private static void TryWriteCache(string cacheFile, string json) + { + try + { + Directory.CreateDirectory(Path.GetDirectoryName(cacheFile)!); + File.WriteAllText(cacheFile, json); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + // Caching is an optimization; never fail discovery because the cache could not be written. + } + } +} + // --------------------------------------------------------------------------------------------------- // Environment resolution. // A resolved environment is what the runner actually executes against, whether it came from @@ -632,9 +828,34 @@ internal sealed record ResolvedEnvironment public string? LocalRoot { get; init; } - // Whether this environment's image is exempt from the Microsoft-only restriction (configured images - // are deliberate; generated images must come from mcr.microsoft.com/dotnet/sdk). + // The .NET majors this environment can both build and *run*. Empty means "single SDK, inferred from + // Channel/Sdk". A multi-SDK runner states them explicitly, which is what makes it usable for a + // repository whose target frameworks span several majors. + public IReadOnlyList SupportedMajors { get; init; } = []; + + public bool IsMultiSdk => SupportedMajors.Count > 1; + + // Whether this environment's image came from the repository's own configuration rather than being + // generated. Configured images are deliberate intent and are always used exactly as written. public bool ImageIsConfigured => Origin == EnvironmentOrigin.Configured; + + // The .NET major version this environment's channel represents ("10.0" -> 10), or 0 when the channel + // is unknown (configured environments carry no channel). + public int ChannelMajor + { + get + { + if (string.IsNullOrWhiteSpace(Channel)) + { + return 0; + } + + var head = Channel.Split('.')[0]; + return int.TryParse(head, NumberStyles.Integer, CultureInfo.InvariantCulture, out var major) && major > 0 + ? major + : 0; + } + } } internal static class GeneratedEnvironments @@ -677,6 +898,17 @@ public static IReadOnlyList FromMetadata(ReleaseMetadata me return result; } + // A multi-SDK runner environment. It carries no single channel: its value is that every listed major + // is present, so a multi-targeted repository runs its whole matrix in one container. + public static ResolvedEnvironment FromMultiSdkRunner(MultiSdkRunner runner) => new() + { + Name = $"ubuntu-testrunner-{runner.Tag}", + Origin = EnvironmentOrigin.Generated, + ReleaseType = "Multi-SDK", + DockerImage = runner.Reference, + SupportedMajors = runner.Majors, + }; + public static ResolvedEnvironment FromConfigured(EnvironmentDefinition def) => new() { Name = def.Name, @@ -695,6 +927,10 @@ internal sealed record EnvironmentResolution public ResolvedEnvironment? Environment { get; init; } public IReadOnlyList Candidates { get; init; } = []; public string? Message { get; init; } + + // Why this environment was chosen when the caller did not name one. Surfaced so an automatic + // selection is always explainable rather than looking arbitrary. + public string? SelectionReason { get; init; } } internal static class EnvironmentResolver @@ -702,11 +938,14 @@ internal static class EnvironmentResolver // Deterministic precedence: // 1. An environment explicitly named by the user (configured first, then generated). // 2. An applicable Docker environment from testenvironments.json (authoritative when present). - // 3. Microsoft-derived environments when no testenvironments.json exists. + // 3. Microsoft-derived environments when no testenvironments.json exists. When several are + // derived, the repository's own highest .NET target framework picks exactly one of them + // (see SelectByTargetFramework) so "run my tests" does not need a question to answer. public static EnvironmentResolution Resolve( TestEnvironmentsConfig? config, IReadOnlyList generated, - string? requestedName) + string? requestedName, + TargetFrameworkInfo? repoTargets = null) { var configured = config?.SupportedDockerEnvironments ?? []; @@ -789,6 +1028,14 @@ public static EnvironmentResolution Resolve( return Resolved(generated[0]); } + // Several channels are available. The repository already states which .NET it targets, so use + // that instead of asking a question the source code has already answered. + var byTargetFramework = SelectByTargetFramework(generated, repoTargets); + if (byTargetFramework is not null) + { + return byTargetFramework; + } + return new EnvironmentResolution { Status = ResolutionStatus.Ambiguous, @@ -797,6 +1044,68 @@ public static EnvironmentResolution Resolve( }; } + // Deterministic tie-break: the repository's highest .NET target framework major must match exactly + // one derived channel. Highest wins because an SDK builds its own major and every lower one, so the + // newest target is the only channel guaranteed to build the whole repository. + // + // This deliberately does not "pick something close". No target frameworks, no .NET target (only + // netstandard/net48), or more than one channel for the same major all fall through to a question — + // guessing an SDK the repository never asked for is worse than asking once. + private static EnvironmentResolution? SelectByTargetFramework( + IReadOnlyList generated, + TargetFrameworkInfo? repoTargets) + { + if (repoTargets is null) + { + return null; + } + + var majors = repoTargets.NetCoreMajors; + if (majors.Count == 0) + { + return null; + } + + // Multi-targeted repositories need every runtime present, not just the newest SDK, so a runner + // covering the whole matrix wins outright when one is available. + if (majors.Count > 1) + { + var covering = generated.Where(e => e.IsMultiSdk && majors.All(e.SupportedMajors.Contains)).ToList(); + if (covering.Count == 0) + { + return null; + } + + var runner = covering + .OrderBy(e => e.SupportedMajors.Count) + .ThenBy(e => e.Name, StringComparer.Ordinal) + .First(); + + var targeted = string.Join(", ", majors.Select(m => $"net{m}.0")); + return Resolved(runner) with + { + SelectionReason = + $"Selected automatically: the repository targets {targeted}, and this runner provides every one of them, " + + "so the whole target-framework matrix runs in a single container.", + }; + } + + var targetMajor = majors.Max(); + var matches = generated.Where(e => !e.IsMultiSdk && e.ChannelMajor == targetMajor).ToList(); + if (matches.Count != 1) + { + return null; + } + + var tfm = repoTargets.TargetFrameworks + .FirstOrDefault(t => TargetFrameworkInspector.NetMajor(t) == targetMajor) ?? $"net{targetMajor}.0"; + + return Resolved(matches[0]) with + { + SelectionReason = $"Selected automatically: the only environment matching the repository's target framework '{tfm}'.", + }; + } + private static EnvironmentResolution Resolved(ResolvedEnvironment env) => new() { Status = ResolutionStatus.Resolved, Environment = env }; } @@ -925,19 +1234,33 @@ public static TargetFrameworkInfo Inspect(string sourceRoot, string? project) // Can a channel (identified by its SDK version) build these target frameworks? An SDK builds its own // major and every lower one; it cannot build a newer runtime major, and the Linux SDK cannot build // .NET Framework (net4x) targets. - public static SdkCompatibility CanBuild(SdkVersion? channelSdk, TargetFrameworkInfo tfms) + public static SdkCompatibility CanBuild( + SdkVersion? channelSdk, + TargetFrameworkInfo tfms, + IReadOnlyList? supportedMajors = null) { - if (channelSdk is null) - { - return new SdkCompatibility(false, "The environment SDK version could not be determined."); - } - if (tfms.HasNetFramework) { return new SdkCompatibility(false, "The project targets .NET Framework (net4x), which cannot be built by a Linux .NET SDK container."); } + // An environment that states its majors explicitly (a multi-SDK runner) is judged on that list: + // every target framework must be present, because presence is what allows the tests to run. + if (supportedMajors is { Count: > 0 }) + { + var missing = tfms.NetCoreMajors.Where(m => !supportedMajors.Contains(m)).ToList(); + return missing.Count == 0 + ? new SdkCompatibility(true, null) + : new SdkCompatibility(false, + $"The project targets {string.Join(", ", missing.Select(m => $"net{m}.0"))}, which the selected image does not provide."); + } + + if (channelSdk is null) + { + return new SdkCompatibility(false, "The environment SDK version could not be determined."); + } + foreach (var major in tfms.NetCoreMajors) { if (major > channelSdk.Major) @@ -947,6 +1270,18 @@ public static SdkCompatibility CanBuild(SdkVersion? channelSdk, TargetFrameworkI } } + // A single-SDK image ships exactly one runtime. Lower target frameworks compile there but have + // no runtime to execute on, so a multi-targeted repository needs a multi-SDK runner instead of + // a silently doomed run. + var unrunnable = tfms.NetCoreMajors.Where(m => m != channelSdk.Major).ToList(); + if (unrunnable.Count > 0) + { + return new SdkCompatibility(false, + $"The project targets {string.Join(", ", unrunnable.Select(m => $"net{m}.0"))} in addition to net{channelSdk.Major}.0, " + + $"but the selected image ships only the {channelSdk.Major}.x runtime. Use a multi-SDK runner image " + + $"({RemoteTestProgram.MultiSdkRepository}) or restrict the run with --framework."); + } + // Honor an explicit global.json pin: the container SDK major must satisfy it. var pinned = SdkVersion.TryParse(tfms.GlobalJsonSdkVersion); if (pinned is not null @@ -1035,6 +1370,12 @@ internal static class ContainerPlanner return string.IsNullOrWhiteSpace(test) ? null : $"FullyQualifiedName~{test}"; } + // NuGet's package root becomes an MSBuild SourceRoot, and SourceLink rejects a SourceRoot that does + // not end in a separator ("SourceRoot paths are required to end with a slash or backslash"). The + // mount target stays clean; only the environment value carries the trailing slash. + public static string NuGetPackagesPath(string containerDir) => + containerDir.EndsWith('/') ? containerDir : containerDir + "/"; + // The in-container script. Phases run in order; each emits a machine-readable end marker with its // exit code so the host can classify restore vs build vs test outcomes precisely. restore/build stop // the run on failure; test always runs to completion so a TRX is produced even when tests fail. @@ -1048,10 +1389,13 @@ public static string BuildEntrypoint(TestCommandOptions o) var sb = new StringBuilder(); sb.Append("set -o pipefail\n"); - sb.Append($"export NUGET_PACKAGES={Shell.Quote(o.NuGetDir)}\n"); + sb.Append($"export NUGET_PACKAGES={Shell.Quote(NuGetPackagesPath(o.NuGetDir))}\n"); sb.Append("export DOTNET_CLI_TELEMETRY_OPTOUT=1\n"); sb.Append("export DOTNET_NOLOGO=1\n"); sb.Append("export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1\n"); + // Keep files created by a container user writable by the host and by a later run using a + // different container UID. The host-side reconciliation below cannot chmod foreign-owned files. + sb.Append("umask 000\n"); sb.Append($"cd {Shell.Quote(o.WorkDir)} || {{ echo '{PhaseMarkerPrefix}staging:1##'; exit 8; }}\n"); sb.Append("run_phase() { name=\"$1\"; shift; \"$@\"; code=$?; echo \"" + PhaseMarkerPrefix + "${name}:${code}##\"; return $code; }\n"); sb.Append($"run_phase restore dotnet restore{target}{fw} || exit 9\n"); @@ -1071,7 +1415,7 @@ public static ContainerPlan Build( { ["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1", ["DOTNET_NOLOGO"] = "1", - ["NUGET_PACKAGES"] = test.NuGetDir, + ["NUGET_PACKAGES"] = NuGetPackagesPath(test.NuGetDir), }; var entrypoint = BuildEntrypoint(test); @@ -1195,7 +1539,30 @@ private static string Relative(string root, string path) => // emit one per TFM) into a single structured result, prioritizing actionable failure detail. // --------------------------------------------------------------------------------------------------- -internal sealed record TestFailureDetail(string TestName, string? ClassName, string? Message, string? StackTrace); +// One failing test, with everything `dotnet test` would have printed about it: where it lives, what it +// asserted, where it threw, and whatever the test itself wrote to the output helper. +internal sealed record TestFailureDetail( + string TestName, + string? ClassName, + string? Message, + string? StackTrace, + string? Output = null, + string? Assembly = null, + string? Framework = null, + double DurationSeconds = 0); + +// One test assembly/TFM pair — the unit `dotnet test` reports a Passed!/Failed! line for. +internal sealed record TestAssemblyResult( + string Assembly, + string? Framework, + int Total, + int Passed, + int Failed, + int Skipped, + double DurationSeconds) +{ + public string Display => Framework is null ? Assembly : $"{Assembly} ({Framework})"; +} internal sealed record TestRunResult { @@ -1206,6 +1573,7 @@ internal sealed record TestRunResult public int NotExecuted { get; init; } public double DurationSeconds { get; init; } public IReadOnlyList Failures { get; init; } = []; + public IReadOnlyList Assemblies { get; init; } = []; public int TrxFilesParsed { get; init; } public static TestRunResult Empty => new(); @@ -1219,33 +1587,92 @@ internal sealed record TestRunResult NotExecuted = NotExecuted + other.NotExecuted, DurationSeconds = DurationSeconds + other.DurationSeconds, Failures = [.. Failures, .. other.Failures], + Assemblies = [.. Assemblies, .. other.Assemblies], TrxFilesParsed = TrxFilesParsed + other.TrxFilesParsed, }; } +// VSTest records the test assembly's path in lower case, so a TRX alone would report +// "acme.tests.dll" for an assembly the developer knows as "Acme.Tests.dll". The repository's own +// project files carry the authoritative casing. +internal static class AssemblyNameIndex +{ + private static readonly string[] ProjectPatterns = ["*.csproj", "*.fsproj", "*.vbproj"]; + + public static IReadOnlyDictionary Build(string sourceRoot) + { + var map = new Dictionary(StringComparer.OrdinalIgnoreCase); + if (!Directory.Exists(sourceRoot)) + { + return map; + } + + foreach (var pattern in ProjectPatterns) + { + IEnumerable files; + try + { + files = Directory.EnumerateFiles(sourceRoot, pattern, SearchOption.AllDirectories); + } + catch (Exception) + { + continue; + } + + foreach (var project in files) + { + var name = Path.GetFileNameWithoutExtension(project) + ".dll"; + map[name] = name; + } + } + + return map; + } +} + internal static class TrxParser { private static readonly XNamespace Ns = "http://microsoft.com/schemas/VisualStudio/TeamTest/2010"; - public static TestRunResult ParseFile(string path) => Parse(File.ReadAllText(path)); + public static TestRunResult ParseFile(string path, IReadOnlyDictionary? assemblyNames = null) => + Parse(File.ReadAllText(path), assemblyNames); - public static TestRunResult Parse(string trxXml) + public static TestRunResult Parse(string trxXml, IReadOnlyDictionary? assemblyNames = null) { var doc = XDocument.Parse(trxXml); var root = doc.Root ?? throw new InvalidOperationException("TRX has no root element."); - // Map testId -> class name via TestDefinitions so failures carry their owning class. + // Map testId -> class name / storage via TestDefinitions so failures carry their owning class and + // the assembly they came from. Multi-targeted projects emit one TRX per TFM, and only the storage + // path distinguishes them. var classById = new Dictionary(StringComparer.OrdinalIgnoreCase); + string? storage = null; + string? classAssembly = null; foreach (var ut in root.Descendants(Ns + "UnitTest")) { var id = ut.Attribute("id")?.Value; var className = ut.Element(Ns + "TestMethod")?.Attribute("className")?.Value; - if (id is not null && className is not null) + if (className is not null) { - classById[id] = className.Split(',')[0]; + var parts = className.Split(',', 2); + if (id is not null) + { + classById[id] = parts[0]; + } + + if (classAssembly is null && parts.Length == 2) + { + classAssembly = parts[1].Trim(); + } } + + storage ??= ut.Attribute("storage")?.Value + ?? ut.Element(Ns + "TestMethod")?.Attribute("codeBase")?.Value; } + var assemblyName = AssemblyDisplayName(storage, classAssembly, assemblyNames); + var framework = FrameworkFrom(storage); + int passed = 0, failed = 0, skipped = 0, notExecuted = 0, total = 0; var failures = new List(); double duration = 0; @@ -1262,18 +1689,24 @@ public static TestRunResult Parse(string trxXml) foreach (var r in root.Descendants(Ns + "UnitTestResult")) { var outcome = r.Attribute("outcome")?.Value ?? ""; - duration += ParseDuration(r.Attribute("duration")?.Value); + var testDuration = ParseDuration(r.Attribute("duration")?.Value); + duration += testDuration; if (string.Equals(outcome, "Failed", StringComparison.OrdinalIgnoreCase)) { var testId = r.Attribute("testId")?.Value; var testName = r.Attribute("testName")?.Value ?? "(unknown test)"; - var error = r.Element(Ns + "Output")?.Element(Ns + "ErrorInfo"); + var output = r.Element(Ns + "Output"); + var error = output?.Element(Ns + "ErrorInfo"); failures.Add(new TestFailureDetail( testName, testId is not null && classById.TryGetValue(testId, out var cls) ? cls : null, error?.Element(Ns + "Message")?.Value?.Trim(), - error?.Element(Ns + "StackTrace")?.Value?.Trim())); + error?.Element(Ns + "StackTrace")?.Value?.Trim(), + output?.Element(Ns + "StdOut")?.Value?.Trim(), + assemblyName, + framework, + Math.Round(testDuration, 3))); } else if (outcome is "NotExecuted" or "Skipped") { @@ -1301,11 +1734,14 @@ public static TestRunResult Parse(string trxXml) NotExecuted = notExecuted, DurationSeconds = Math.Round(duration, 3), Failures = failures, + Assemblies = assemblyName is null + ? [] + : [new TestAssemblyResult(assemblyName, framework, total, passed, failed, skipped, Math.Round(duration, 3))], TrxFilesParsed = 1, }; } - public static TestRunResult ParseDirectory(string directory) + public static TestRunResult ParseDirectory(string directory, IReadOnlyDictionary? assemblyNames = null) { var result = TestRunResult.Empty; if (!Directory.Exists(directory)) @@ -1317,7 +1753,7 @@ public static TestRunResult ParseDirectory(string directory) { try { - result = result.Merge(ParseFile(file)); + result = result.Merge(ParseFile(file, assemblyNames)); } catch (Exception) { @@ -1325,7 +1761,61 @@ public static TestRunResult ParseDirectory(string directory) } } - return result; + return result with + { + Assemblies = [.. result.Assemblies.OrderBy(a => a.Assembly, StringComparer.OrdinalIgnoreCase).ThenBy(a => a.Framework, StringComparer.OrdinalIgnoreCase)], + }; + } + + // "/workspace/test/Acme.Tests/bin/Debug/net10.0/Acme.Tests.dll" -> "Acme.Tests.dll". + internal static string? AssemblyNameFrom(string? storage) => + string.IsNullOrWhiteSpace(storage) ? null : Path.GetFileName(storage.Replace('\\', '/')); + + // VSTest lower-cases the storage path it records, which would report "acme.tests.dll" for an assembly + // the developer knows as "Acme.Tests.dll". Two sources restore the real casing, in order of + // authority: the repository's own project files, then the class name's assembly part + // ("Namespace.Type, Acme.Tests"), which some loggers include and which keeps its casing. + internal static string? AssemblyDisplayName( + string? storage, string? classAssembly, IReadOnlyDictionary? assemblyNames = null) + { + var fromStorage = AssemblyNameFrom(storage); + if (fromStorage is not null && assemblyNames is not null && assemblyNames.TryGetValue(fromStorage, out var known)) + { + return known; + } + + var simpleName = classAssembly?.Split(',')[0].Trim(); + if (string.IsNullOrWhiteSpace(simpleName)) + { + return fromStorage; + } + + var candidate = simpleName + ".dll"; + return fromStorage is null || string.Equals(candidate, fromStorage, StringComparison.OrdinalIgnoreCase) + ? candidate + : fromStorage; + } + + // The TFM is the output folder the test assembly was built into; it is the only place a TRX records + // which target framework produced it. + internal static string? FrameworkFrom(string? storage) + { + if (string.IsNullOrWhiteSpace(storage)) + { + return null; + } + + var segments = storage.Replace('\\', '/').Split('/', StringSplitOptions.RemoveEmptyEntries); + for (var i = segments.Length - 2; i >= 0; i--) + { + if (Regex.IsMatch(segments[i], @"^net(standard|coreapp|framework)?\d+(\.\d+)*(-[a-z0-9.]+)?$", RegexOptions.IgnoreCase) + || Regex.IsMatch(segments[i], @"^net\d{2,3}$", RegexOptions.IgnoreCase)) + { + return segments[i]; + } + } + + return null; } private static int IntAttr(XElement e, string name) => @@ -1445,14 +1935,42 @@ public static ExecutionOutcome Classify( public static IReadOnlyDictionary ParsePhaseMarkers(string output) { var map = new Dictionary(StringComparer.OrdinalIgnoreCase); - foreach (Match m in Regex.Matches(output, - Regex.Escape(ContainerPlanner.PhaseMarkerPrefix) + @"(?[a-zA-Z]+):(?-?\d+)##")) + foreach (Match m in MarkerPattern.Matches(output)) { map[m.Groups["name"].Value] = int.Parse(m.Groups["code"].Value, CultureInfo.InvariantCulture); } return map; } + + // The container emits one marker per phase, so the text between two markers is exactly that phase's + // log. Reporting the failing phase's own output — instead of a tail of everything — is what turns + // "the build failed" into "this file, this line, this compiler error". + public static string PhaseOutput(string output, string phase) + { + if (string.IsNullOrEmpty(output)) + { + return ""; + } + + var start = 0; + foreach (Match m in MarkerPattern.Matches(output)) + { + if (string.Equals(m.Groups["name"].Value, phase, StringComparison.OrdinalIgnoreCase)) + { + return output[start..m.Index].Trim('\r', '\n'); + } + + start = m.Index + m.Length; + } + + // The phase never completed (crash, cancellation): everything after the last marker is its log. + return output[start..].Trim('\r', '\n'); + } + + private static readonly Regex MarkerPattern = new( + Regex.Escape(ContainerPlanner.PhaseMarkerPrefix) + @"(?[a-zA-Z]+):(?-?\d+)##", + RegexOptions.Compiled); } // --------------------------------------------------------------------------------------------------- @@ -1753,6 +2271,19 @@ public static Task PullAsync(string image, CancellationToken ct) return r.ExitCode == 0 ? r.StdOut.Trim() : null; } + // The user an image is configured to run as. Empty means the image sets none, i.e. root. + public static async Task ResolveUserAsync(string image, CancellationToken ct) + { + var r = await ProcessRunner.RunAsync("docker", ["inspect", "--format", "{{.Config.User}}", image], null, ct); + if (r.ExitCode != 0) + { + return null; + } + + var user = r.StdOut.Trim(); + return user.Length == 0 ? null : user; + } + public static Task BuildAsync(string dockerfile, string context, string tag, CancellationToken ct) => ProcessRunner.RunAsync("docker", ["build", "-f", dockerfile, "-t", tag, context], null, ct); @@ -1774,26 +2305,160 @@ public static async Task RemoveContainerAsync(string name, CancellationTok } // --------------------------------------------------------------------------------------------------- -// Source staging — copy the source into an isolated, disposable workspace so container builds never -// pollute the developer's working tree with Linux bin/obj artifacts. +// Image preparation — a .NET build routinely shells out to host tooling: MinVer, Nerdbank.GitVersioning, +// GitInfo and SourceLink all invoke `git` while building. An image without it fails the build (MinVer +// reports MINVER1007) even though the code compiles fine, which reads as a repository problem when it is +// really a missing tool in the image. Microsoft's SDK images ship git; a minimal runner image may not. +// When it is missing, one thin layer is derived from the resolved base image and cached under a +// digest-addressed tag, so the cost is paid once per image and never inside the repository. // --------------------------------------------------------------------------------------------------- -internal sealed record StagingResult(string? StagedPath, string? Error, int FileCount); +internal sealed record PreparedImage(string Reference, bool Provisioned, string? Note = null); -internal static class SourceStager +internal static class ImageProvisioner { - private static readonly string[] ExcludedDirs = ["bin", "obj", ".git", ".vs", ".vscode", "node_modules", "TestResults"]; + // Tooling the container must expose on PATH for a build to behave the way it does on the host. + public static readonly string[] RequiredTools = ["git"]; - public static async Task StageAsync(string sourceRoot, string stagingRoot, CancellationToken ct) - { - if (!Directory.Exists(sourceRoot)) - { - return new StagingResult(null, $"Source root does not exist: {sourceRoot}", 0); - } + private const string TagPrefix = "dotnet-remote-testing/prepared"; - try - { - Directory.CreateDirectory(stagingRoot); + public static string ToolList => string.Join(", ", RequiredTools); + + // Content-addressed tag: the same base image and tool set always produce the same prepared image, + // so a later run reuses the cached layer instead of rebuilding it. + public static string DerivedTag(string baseReference, string? digest) + { + var key = digest is not null && digest.Contains(':', StringComparison.Ordinal) + ? digest[(digest.IndexOf(':', StringComparison.Ordinal) + 1)..] + : StableHash(baseReference); + var shortKey = key.Length > 16 ? key[..16] : key; + return $"{TagPrefix}:{string.Join('-', RequiredTools)}-{shortKey}"; + } + + // Verifies the tools are on PATH inside the image, without assuming a specific shell or entrypoint. + public static string ProbeCommand() => + string.Join(" && ", RequiredTools.Select(t => $"command -v {t} >/dev/null 2>&1")); + + // A single RUN that adapts to whichever package manager the base image ships. The Dockerfile is + // written to the run's own temporary directory — never into the repository being tested. + // + // Installing packages needs root, but the identity the tests run under is part of the environment + // being reproduced: a base image that runs as a non-root user must keep doing so, or the prepared + // image writes build and test output with different ownership than the configured image would. + // baseUser is that image's configured user, or null when it sets none (already root). + public static string Dockerfile(string baseReference, string? baseUser = null) + { + var tools = string.Join(' ', RequiredTools); + var restore = string.IsNullOrWhiteSpace(baseUser) ? string.Empty : $"USER {baseUser.Trim()}\n"; + return $""" + FROM {baseReference} + USER root + RUN set -e; \ + if command -v apt-get >/dev/null 2>&1; then \ + apt-get update && apt-get install -y --no-install-recommends {tools} && rm -rf /var/lib/apt/lists/*; \ + elif command -v apk >/dev/null 2>&1; then \ + apk add --no-cache {tools}; \ + elif command -v microdnf >/dev/null 2>&1; then \ + microdnf install -y {tools} && microdnf clean all; \ + elif command -v dnf >/dev/null 2>&1; then \ + dnf install -y {tools} && dnf clean all; \ + elif command -v yum >/dev/null 2>&1; then \ + yum install -y {tools} && yum clean all; \ + else \ + echo 'No supported package manager in the base image.' >&2; exit 1; \ + fi + {restore} + """; + } + + // Best effort by design: when the tooling cannot be added, the base image is used anyway and the + // reason is reported, because a repository that never invokes git still runs perfectly well there. + public static async Task EnsureAsync( + string baseReference, string? digest, string workRoot, bool offline, CancellationToken ct) + { + var tag = DerivedTag(baseReference, digest); + + if (await DockerClient.ResolveImageIdAsync(tag, ct) is not null) + { + return new PreparedImage(tag, true, $"Reused prepared image providing {ToolList}."); + } + + var probe = await DockerClient.RunAsync( + ["run", "--rm", "--entrypoint", "sh", baseReference, "-c", ProbeCommand()], ct); + if (probe.ExitCode == 0) + { + return new PreparedImage(baseReference, false); + } + + if (offline) + { + return new PreparedImage(baseReference, false, + $"The image does not provide {ToolList} and adding it needs network access (--offline). " + + "A build that invokes it will fail."); + } + + // Read the identity off the base image before deriving from it, so the prepared image keeps + // running as whoever the configured image runs as instead of silently switching to root. + var baseUser = await DockerClient.ResolveUserAsync(baseReference, ct); + + var contextDir = Path.Combine(workRoot, "image-prep"); + Directory.CreateDirectory(contextDir); + var dockerfile = Path.Combine(contextDir, "Dockerfile"); + await File.WriteAllTextAsync(dockerfile, Dockerfile(baseReference, baseUser), ct); + + var build = await DockerClient.BuildAsync(dockerfile, contextDir, tag, ct); + if (build.ExitCode != 0) + { + var reason = build.StdErr.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .LastOrDefault() ?? "docker build failed."; + return new PreparedImage(baseReference, false, + $"The image does not provide {ToolList} and it could not be added: {reason}"); + } + + return new PreparedImage(tag, true, $"Added {ToolList} to the image (cached for later runs)."); + } + + private static string StableHash(string value) + { + var bytes = System.Security.Cryptography.SHA256.HashData(Encoding.UTF8.GetBytes(value)); + return Convert.ToHexStringLower(bytes); + } +} + +// --------------------------------------------------------------------------------------------------- +// Source staging — copy the source into an isolated, disposable workspace so container builds never +// pollute the developer's working tree with Linux bin/obj artifacts. +// +// The staged copy must still *be* a repository. Dropping .git changes observable behavior: MinVer and +// Nerdbank.GitVersioning fall back to 0.0.0, SourceLink stops embedding, and any repository-root probe +// ("walk up until a .git directory exists") resolves somewhere else entirely — which silently changes +// what the tests under it see. That is the difference between "it passes in Visual Studio's remote +// testing and fails here", so .git is staged too, as a disposable copy the container may freely write. +// --------------------------------------------------------------------------------------------------- + +internal sealed record StagingResult( + string? StagedPath, + string? Error, + int FileCount, + bool GitMetadataStaged = false, + long GitMetadataBytes = 0, + string? GitMetadataNote = null); + +internal static class SourceStager +{ + private static readonly string[] ExcludedDirs = ["bin", "obj", ".git", ".vs", ".vscode", "node_modules", "TestResults"]; + + public static async Task StageAsync( + string sourceRoot, string stagingRoot, CancellationToken ct, bool includeGitMetadata = true) + { + if (!Directory.Exists(sourceRoot)) + { + return new StagingResult(null, $"Source root does not exist: {sourceRoot}", 0); + } + + try + { + Directory.CreateDirectory(stagingRoot); // Prefer git to enumerate tracked + untracked-not-ignored files; this keeps ignored build // output out of the staged copy without reimplementing .gitignore. @@ -1802,7 +2467,13 @@ public static async Task StageAsync(string sourceRoot, string sta ? CopyEnumerated(sourceRoot, stagingRoot, files) : CopyRecursive(sourceRoot, stagingRoot); - return new StagingResult(stagingRoot, null, count); + var git = includeGitMetadata + ? StageGitMetadata(sourceRoot, stagingRoot) + : new GitStagingResult(false, 0, "Git metadata staging disabled; repository-root detection and version stamping will differ from the host."); + + MakeWritableForContainer(stagingRoot); + + return new StagingResult(stagingRoot, null, count, git.Staged, git.Bytes, git.Note); } catch (Exception ex) { @@ -1810,9 +2481,220 @@ public static async Task StageAsync(string sourceRoot, string sta } } + private sealed record GitStagingResult(bool Staged, long Bytes, string? Note); + + // Copy the repository's git directory verbatim into the staged workspace. Best-effort by design: a + // missing or unreadable .git is a fidelity note, never a reason to fail a run that can still execute. + private static GitStagingResult StageGitMetadata(string sourceRoot, string stagingRoot) + { + var gitDir = ResolveGitDirectory(sourceRoot); + if (gitDir is null) + { + return new GitStagingResult(false, 0, null); + } + + var destination = Path.Combine(stagingRoot, ".git"); + try + { + // A linked worktree stages its "gitdir:" pointer file as ordinary content; the real git + // directory has to replace it, because the path it points at does not exist in the container. + if (File.Exists(destination)) + { + File.Delete(destination); + } + + // A linked worktree's git directory holds only per-worktree state (HEAD, index, logs). The + // objects, refs and config live in the shared directory its "commondir" points at, outside + // the staged copy. Staging the worktree half alone produces a git directory git cannot read, + // so versioning falls back to 0.0.0 and SourceLink stops embedding — the exact fidelity loss + // staging .git exists to prevent. The shared half is copied first and the per-worktree files + // are layered over it, which collapses the pair into an ordinary standalone repository. + var commonDir = ResolveCommonDirectory(gitDir); + if (commonDir is not null) + { + // "worktrees/" only registers linked worktrees by host path; none of them exist in the + // container, and this worktree's own entry is exactly what is being flattened here. + CopyDirectory(commonDir, destination, excludeTopLevelDirectory: "worktrees"); + } + + var bytes = CopyDirectory(gitDir, destination); + if (commonDir is not null) + { + // The staged repository is standalone now; leaving the pointers behind would send git + // back out to host paths that do not exist in the container. + foreach (var pointer in new[] { "commondir", "gitdir" }) + { + var stale = Path.Combine(destination, pointer); + if (File.Exists(stale)) + { + File.Delete(stale); + } + } + + bytes = MeasureDirectory(destination); + } + + return new GitStagingResult(true, bytes, null); + } + catch (Exception ex) + { + return new GitStagingResult(false, 0, + $"Git metadata could not be staged ({ex.Message}); repository-root detection and version stamping may differ from the host."); + } + } + + // .git is a directory in an ordinary clone and a "gitdir: " pointer file in a linked worktree + // or submodule. Both resolve to a real directory that carries the repository state. + private static string? ResolveGitDirectory(string sourceRoot) + { + var candidate = Path.Combine(sourceRoot, ".git"); + if (Directory.Exists(candidate)) + { + return candidate; + } + + if (!File.Exists(candidate)) + { + return null; + } + + var pointer = File.ReadAllText(candidate).Trim(); + const string Prefix = "gitdir:"; + if (!pointer.StartsWith(Prefix, StringComparison.OrdinalIgnoreCase)) + { + return null; + } + + var target = pointer[Prefix.Length..].Trim(); + if (!Path.IsPathRooted(target)) + { + target = Path.GetFullPath(Path.Combine(sourceRoot, target)); + } + + return Directory.Exists(target) ? target : null; + } + + // A linked worktree's git directory carries a "commondir" file naming the shared repository + // directory that actually holds objects, refs and config. An ordinary clone has no such file. + private static string? ResolveCommonDirectory(string gitDir) + { + var marker = Path.Combine(gitDir, "commondir"); + if (!File.Exists(marker)) + { + return null; + } + + var target = File.ReadAllText(marker).Trim(); + if (target.Length == 0) + { + return null; + } + + if (!Path.IsPathRooted(target)) + { + target = Path.GetFullPath(Path.Combine(gitDir, target)); + } + + return Directory.Exists(target) ? target : null; + } + + private static long MeasureDirectory(string root) => + Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories).Sum(f => new FileInfo(f).Length); + + // Bind mounts keep the host's ownership of the mounted directory; Docker only reconciles ownership + // for named volumes, never for bind mounts. A prepared image restores the base image's configured + // user (see ImageProvisioner), and that user's UID/GID commonly differs from whoever staged this + // directory on the host, which leaves restore/build/test unable to write into it. Opening group and + // other write access sidesteps the mismatch without needing to know the container's UID up front. + // No-op on Windows, where Docker Desktop's bind-mount layer does not enforce host UID/GID at all. + internal static void MakeWritableForContainer(string root) + { + if (OperatingSystem.IsWindows() || !Directory.Exists(root)) + { + return; + } + + try + { + AddWriteBits(root, isDirectory: true); + foreach (var dir in Directory.EnumerateDirectories(root, "*", SearchOption.AllDirectories)) + { + AddWriteBits(dir, isDirectory: true); + } + + foreach (var file in Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories)) + { + AddWriteBits(file, isDirectory: false); + } + } + catch (IOException) + { + // Best effort: a container user permission problem surfaces on its own via the run's exit + // code, which is a clearer signal than failing the run here over a chmod race. + } + } + + private static void AddWriteBits(string path, bool isDirectory) + { + var mode = File.GetUnixFileMode(path); + var required = UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.OtherRead | UnixFileMode.OtherWrite; + if (isDirectory) + { + required |= UnixFileMode.GroupExecute | UnixFileMode.OtherExecute; + } + + // chmod itself requires ownership. If a previous container UID already left the required + // access bits in place, reuse those entries without attempting a chmod that must fail. + if ((mode & required) == required) + { + return; + } + + File.SetUnixFileMode(path, mode | required); + } + + private static long CopyDirectory(string source, string destination, string? excludeTopLevelDirectory = null) + { + long bytes = 0; + var stack = new Stack<(string Source, string Destination)>(); + stack.Push((source, destination)); + while (stack.Count > 0) + { + var (from, to) = stack.Pop(); + Directory.CreateDirectory(to); + foreach (var file in Directory.EnumerateFiles(from)) + { + var target = Path.Combine(to, Path.GetFileName(file)); + File.Copy(file, target, overwrite: true); + // The source .git may be read-only in places (packed objects); the staged copy is + // disposable and the container must be able to write to it. + new FileInfo(target).IsReadOnly = false; + bytes += new FileInfo(target).Length; + } + + foreach (var dir in Directory.EnumerateDirectories(from)) + { + var name = Path.GetFileName(dir); + if (excludeTopLevelDirectory is not null + && string.Equals(from, source, StringComparison.Ordinal) + && string.Equals(name, excludeTopLevelDirectory, StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + stack.Push((dir, Path.Combine(to, name))); + } + } + + return bytes; + } + private static async Task?> TryGitEnumerateAsync(string sourceRoot, CancellationToken ct) { - if (!Directory.Exists(Path.Combine(sourceRoot, ".git"))) + // .git is a directory in an ordinary clone and a "gitdir:" pointer file in a linked worktree; + // git enumerates both, and skipping the pointer form would stage ignored build output. + var marker = Path.Combine(sourceRoot, ".git"); + if (!Directory.Exists(marker) && !File.Exists(marker)) { return null; } @@ -1895,7 +2777,8 @@ internal sealed record ResolveContext( ReleaseMetadata? Metadata, string? MetadataError, IReadOnlyList Generated, - EnvironmentResolution Resolution); + EnvironmentResolution Resolution, + string? MultiSdkError = null); internal static class Commands { @@ -1928,8 +2811,47 @@ bool Match(EnvironmentDefinition e) => } } - var resolution = EnvironmentResolver.Resolve(config, generated, options.EnvironmentName); - return new ResolveContext(config, metadata, metadataError, generated, resolution); + // Generated environments are resolved against the repository's own target frameworks. Configured + // environments never need this (they are deliberate intent), and a named environment short-circuits + // before it is used, so the inspection only runs when it can actually decide something. + TargetFrameworkInfo? repoTargets = null; + string? multiSdkError = null; + if (options.EnvironmentName is null && config?.SourcePath is null && generated.Count > 1) + { + repoTargets = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(options.RepoRoot, options.Project), options.Framework); + + // Only a genuinely multi-targeted repository needs the multi-SDK runner feed; do not spend a + // network call to answer a question a single target framework already answers. + if (repoTargets.NetCoreMajors.Count > 1) + { + var multi = await MultiSdkRunnerStore.LoadAsync(options, ct); + multiSdkError = multi.Error; + var runner = MultiSdkTagReader.Select(multi.Runners, repoTargets.NetCoreMajors); + if (runner is not null) + { + generated = [.. generated, GeneratedEnvironments.FromMultiSdkRunner(runner)]; + } + } + } + + var resolution = EnvironmentResolver.Resolve(config, generated, options.EnvironmentName, repoTargets); + return new ResolveContext(config, metadata, metadataError, generated, resolution, multiSdkError); + } + + // --framework narrows what actually runs, so it must narrow what the environment is chosen for too. + // Without this, "-f net10.0" on a multi-targeted repository would still be resolved as multi-targeted. + private static TargetFrameworkInfo NarrowToRequestedFramework(TargetFrameworkInfo info, string? framework) + { + if (string.IsNullOrWhiteSpace(framework)) + { + return info; + } + + var match = info.TargetFrameworks.FirstOrDefault( + t => string.Equals(t, framework, StringComparison.OrdinalIgnoreCase)); + + return info with { TargetFrameworks = match is null ? [framework] : [match] }; } public static async Task ListAsync(Options options) @@ -1956,6 +2878,20 @@ public static async Task ListAsync(Options options) metadata = result.Metadata; metadataError = result.Error; environments = metadata is not null ? GeneratedEnvironments.FromMetadata(metadata) : []; + + // A multi-targeted repository cannot execute its lower target frameworks on a single-SDK + // image, so offer the runner that can — listing only what cannot work would be misleading. + var repoMajors = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(options.RepoRoot, options.Project), options.Framework).NetCoreMajors; + if (repoMajors.Count > 1) + { + var multi = await MultiSdkRunnerStore.LoadAsync(options, cts.Token); + var runner = MultiSdkTagReader.Select(multi.Runners, repoMajors); + if (runner is not null) + { + environments = [GeneratedEnvironments.FromMultiSdkRunner(runner), .. environments]; + } + } } if (options.Json) @@ -1970,6 +2906,7 @@ public static async Task ListAsync(Options options) { e.Name, origin = e.Origin.ToString(), e.Channel, e.ReleaseType, e.Sdk, image = e.DockerImage, dockerFile = e.DockerFile, + supportedMajors = e.SupportedMajors.Count > 0 ? e.SupportedMajors : null, }), unsupported = unsupported.Select(e => new { e.Name, type = e.RawType }), configDiagnostics = config?.Diagnostics.Select(d => new { d.Code, d.Message, environment = d.EnvironmentName }), @@ -1999,6 +2936,11 @@ public static async Task ListAsync(Options options) Console.WriteLine($" {e.ReleaseType}"); } + if (e.SupportedMajors.Count > 0) + { + Console.WriteLine($" Provides .NET {string.Join(", ", e.SupportedMajors)} in one image"); + } + if (e.Sdk is not null) { Console.WriteLine($" SDK {e.Sdk}"); @@ -2046,7 +2988,8 @@ public static async Task PlanAsync(Options options) var env = ctx.Resolution.Environment!; var sourceRoot = ResolveSourceRoot(options, env); - var tfmInfo = TargetFrameworkInspector.Inspect(sourceRoot, options.Project); + var tfmInfo = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(sourceRoot, options.Project), options.Framework); // Image identity: for generated environments, validate candidate tags against MCR and pre-resolve // the digest without pulling. Offline / --no-registry-check skips the network probe. @@ -2060,6 +3003,14 @@ public static async Task PlanAsync(Options options) { imageNote = $"Configured Dockerfile '{env.DockerFile}' will be built into a local image."; } + else if (env.IsMultiSdk) + { + // The tag came from the publisher's own tag feed, so it exists by construction. Its digest is + // resolved at pull time in `run`; there is no Microsoft registry probe to make here. + requestedTag = env.DockerImage?.Split(':').Last(); + imageNote = $"Multi-SDK runner providing .NET {string.Join(", ", env.SupportedMajors)}; " + + "the whole target-framework matrix runs in one container."; + } else if (env.Origin == EnvironmentOrigin.Generated && channelSdk is not null) { var candidates = ImageTagResolver.CandidateTags(channelSdk); @@ -2087,7 +3038,7 @@ public static async Task PlanAsync(Options options) } } - var compatibility = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo); + var compatibility = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo, env.SupportedMajors); var compatibilityBlocking = SdkCompatibilityPolicy.IsBlocking(env.Origin, compatibility.Compatible); // Configured environments trust their image SDK (validated at run time); do not present or fail // them as incompatible just because the SDK could not be determined statically. @@ -2110,7 +3061,15 @@ public static async Task PlanAsync(Options options) Console.WriteLine(JsonSerializer.Serialize(new { tool = RemoteTestProgram.ToolName, - environment = new { env.Name, origin = env.Origin.ToString(), env.Channel, env.ReleaseType, env.Sdk }, + environment = new + { + env.Name, + origin = env.Origin.ToString(), + env.Channel, + env.ReleaseType, + env.Sdk, + selectionReason = ctx.Resolution.SelectionReason, + }, image = new { requested = env.DockerImage ?? reference, @@ -2119,7 +3078,7 @@ public static async Task PlanAsync(Options options) dockerFile = env.DockerFile, digest, digestResolved = digest is not null, - microsoftOnlyEnforced = env.Origin == EnvironmentOrigin.Generated, + recommendedPublisher = IsRecommendedPublisher(env.DockerImage ?? reference), note = imageNote, }, targetFrameworks = new @@ -2138,6 +3097,11 @@ public static async Task PlanAsync(Options options) else { Console.WriteLine($"Plan: {env.Name}"); + if (ctx.Resolution.SelectionReason is not null) + { + Console.WriteLine($" Selection: {ctx.Resolution.SelectionReason}"); + } + Console.WriteLine($" Image: {env.DockerImage ?? reference ?? "(from Dockerfile)"}"); if (requestedTag is not null) { @@ -2171,6 +3135,14 @@ public static async Task PlanAsync(Options options) Coverage = options.Coverage, }; + // Recommended publishers for auto-generated environments: Microsoft's official SDK images and the + // Codebelt multi-SDK test runner. This is reported, not enforced — an image from anywhere else is + // allowed (a configured dockerImage is deliberate intent), it simply is not one we vouch for. + private static bool IsRecommendedPublisher(string? reference) => + reference is not null + && (reference.StartsWith(RemoteTestProgram.SdkRepository + ":", StringComparison.Ordinal) + || reference.StartsWith(RemoteTestProgram.MultiSdkRepository + ":", StringComparison.Ordinal)); + private static string ResolveSourceRoot(Options options, ResolvedEnvironment env) { if (string.IsNullOrWhiteSpace(env.LocalRoot)) @@ -2206,6 +3178,7 @@ private static int ReportResolutionProblem(Options options, ResolveContext ctx) message, candidates = res.Candidates, metadataError = ctx.MetadataError, + multiSdkError = ctx.MultiSdkError, }, RemoteTestProgram.JsonOut)); } else @@ -2215,6 +3188,11 @@ private static int ReportResolutionProblem(Options options, ResolveContext ctx) { Console.Error.WriteLine("Available: " + string.Join(", ", res.Candidates)); } + + if (ctx.MultiSdkError is not null) + { + Console.Error.WriteLine($"Multi-SDK runner discovery: {ctx.MultiSdkError}"); + } } return (int)kind; @@ -2226,7 +3204,9 @@ private sealed record RunImage( string? RequestedTag = null, string? Sdk = null, FailureKind Kind = FailureKind.None, - string? Error = null); + string? Error = null, + string? PreparedReference = null, + string? ProvisionNote = null); private sealed record CleanupReport(bool ContainerRemoved, bool WorkspaceRemoved, IReadOnlyList Leftovers); @@ -2242,6 +3222,10 @@ public static async Task RunAsync(Options options) void OnCancel(object? _, ConsoleCancelEventArgs e) { e.Cancel = true; cancelled = true; cts.Cancel(); } Console.CancelKeyPress += OnCancel; + // Wall clock for the whole operation. Reported alongside the test duration from the TRX so a + // fast test suite behind a slow image pull never looks like the run itself took no time. + var wallClock = Stopwatch.StartNew(); + var containerName = ContainerPlanner.ContainerName(Guid.NewGuid().ToString("N")[..8]); var runRoot = Path.Combine(Path.GetTempPath(), "dotnet-remote-testing", containerName); string? stagingRoot = null; @@ -2263,9 +3247,10 @@ public static async Task RunAsync(Options options) } var sourceRoot = ResolveSourceRoot(options, env); - var tfmInfo = TargetFrameworkInspector.Inspect(sourceRoot, options.Project); + var tfmInfo = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(sourceRoot, options.Project), options.Framework); var channelSdk = SdkVersion.TryParse(env.Sdk); - var compat = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo); + var compat = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo, env.SupportedMajors); if (SdkCompatibilityPolicy.IsBlocking(env.Origin, compat.Compatible)) { return Error(options, FailureKind.SdkIncompatibility, compat.Reason ?? "The selected SDK cannot build the requested target framework."); @@ -2277,19 +3262,54 @@ public static async Task RunAsync(Options options) return Error(options, image.Kind, image.Error); } + // The image runs the build, not just the tests, so it must carry what the build shells out to. + var prepared = await ImageProvisioner.EnsureAsync( + image.Reference!, image.Digest, runRoot, options.Offline, cts.Token); + image = image with { PreparedReference = prepared.Provisioned ? prepared.Reference : null, ProvisionNote = prepared.Note }; + var resultsRoot = Path.Combine(runRoot, "results"); stagingRoot = Path.Combine(runRoot, "workspace"); Directory.CreateDirectory(resultsRoot); + SourceStager.MakeWritableForContainer(resultsRoot); - var staging = await SourceStager.StageAsync(sourceRoot, stagingRoot, cts.Token); + var staging = await SourceStager.StageAsync(sourceRoot, stagingRoot, cts.Token, includeGitMetadata: !options.NoGitMetadata); if (staging.Error is not null || staging.StagedPath is null) { return Error(options, FailureKind.SourceStaging, staging.Error ?? "Source staging produced no workspace."); } - // NuGet packages cache persists across runs and lives outside the repository. + // NuGet packages cache persists across runs and lives outside the repository. Files already + // in it may carry an older run's container UID, so it is reconciled on every run rather than + // only when created. A cache whose modes cannot be repaired by this host is quarantined and + // rebuilt; continuing with it would make restore fail later inside the non-root container. var nugetCache = Path.Combine(options.CacheDirectory, "nuget"); Directory.CreateDirectory(nugetCache); + try + { + SourceStager.MakeWritableForContainer(nugetCache); + } + catch (UnauthorizedAccessException ex) + { + var quarantine = Path.Combine( + options.CacheDirectory, + "nuget-stale-" + DateTime.UtcNow.ToString("yyyyMMddHHmmssfff", CultureInfo.InvariantCulture) + + "-" + Guid.NewGuid().ToString("N")); + try + { + Directory.Move(nugetCache, quarantine); + Directory.CreateDirectory(nugetCache); + SourceStager.MakeWritableForContainer(nugetCache); + Console.Error.WriteLine( + $"NuGet cache entries could not be made writable because they are owned by another UID; " + + $"the old cache was preserved at '{quarantine}' and a fresh cache will be used."); + } + catch (Exception recoveryError) when (recoveryError is IOException or UnauthorizedAccessException) + { + return Error(options, FailureKind.Restore, + $"NuGet cache '{nugetCache}' is not writable and could not be quarantined for recovery: " + + recoveryError.Message + $" Original permission error: {ex.Message}"); + } + } var testOptions = BuildTestOptions(options, sourceRoot); var mounts = new[] @@ -2298,7 +3318,7 @@ public static async Task RunAsync(Options options) new ContainerMount(nugetCache, "/nuget", ReadOnly: false), new ContainerMount(resultsRoot, "/results", ReadOnly: false), }; - var plan = ContainerPlanner.Build(image.Reference!, containerName, mounts, testOptions); + var plan = ContainerPlanner.Build(prepared.Reference, containerName, mounts, testOptions); ProcessResult proc; try @@ -2313,13 +3333,15 @@ public static async Task RunAsync(Options options) cancelled |= proc.TimedOut; - var results = TrxParser.ParseDirectory(resultsRoot); + var results = TrxParser.ParseDirectory(resultsRoot, AssemblyNameIndex.Build(sourceRoot)); var phaseMarkers = FailureClassifier.ParsePhaseMarkers(proc.StdOut); var outcome = FailureClassifier.Classify(phaseMarkers, proc.ExitCode, cancelled, results); var cleanup = await CleanupAsync(containerName, runRoot, cts.Token); - return EmitRunResult(options, env, image, tfmInfo, results, outcome, proc, cleanup); + return EmitRunResult( + options, env, image, tfmInfo, results, outcome, proc, cleanup, + ctx.Resolution.SelectionReason, wallClock.Elapsed.TotalSeconds, staging); } catch (OperationCanceledException) { @@ -2377,7 +3399,9 @@ private static async Task ResolveImageForRunAsync( } else { - // Configured dockerImage is deliberate repository intent — exempt from the Microsoft-only rule. + // A configured dockerImage (deliberate repository intent) or a multi-SDK runner tag taken + // from the publisher's tag feed. Both are used exactly as written; the digest is resolved + // from the pulled image below. reference = env.DockerImage!; } @@ -2426,7 +3450,10 @@ private static int EmitRunResult( TestRunResult results, ExecutionOutcome outcome, ProcessResult proc, - CleanupReport cleanup) + CleanupReport cleanup, + string? selectionReason = null, + double? elapsedSeconds = null, + StagingResult? staging = null) { var exit = FailureClassifier.ToExitCode(outcome.Kind); if (outcome.Kind == FailureKind.None && cleanup.Leftovers.Count > 0) @@ -2450,19 +3477,38 @@ private static int EmitRunResult( failureKind = outcome.Kind == FailureKind.None ? null : outcome.Kind.ToString(), phase = outcome.Phase, message = outcome.Message, - environment = new { env.Name, origin = env.Origin.ToString(), env.Channel, env.ReleaseType }, - image = new { requested = image.RequestedTag, reference = image.Reference, digest = image.Digest, sdk = image.Sdk }, + environment = new { env.Name, origin = env.Origin.ToString(), env.Channel, env.ReleaseType, selectionReason }, + image = new + { + requested = image.RequestedTag, + reference = image.Reference, + digest = image.Digest, + sdk = image.Sdk, + prepared = image.PreparedReference, + provisioning = image.ProvisionNote, + }, targetFrameworks = tfmInfo.TargetFrameworks, + workspace = new { gitMetadataStaged = staging?.GitMetadataStaged, gitMetadataNote = staging?.GitMetadataNote }, tests = new { results.Total, results.Passed, results.Skipped, results.Failed, results.DurationSeconds, trxFiles = results.TrxFilesParsed }, - failures = results.Failures.Select(f => new { f.TestName, f.ClassName, f.Message, f.StackTrace }), + testAssemblies = results.Assemblies.Select(a => new { a.Assembly, a.Framework, a.Total, a.Passed, a.Failed, a.Skipped, a.DurationSeconds }), + elapsedSeconds = elapsedSeconds is null ? (double?)null : Math.Round(elapsedSeconds.Value, 1), + failures = results.Failures.Select(f => new { f.TestName, f.ClassName, f.Assembly, f.Framework, f.DurationSeconds, f.Message, f.StackTrace, f.Output }), cleanup = new { cleanup.ContainerRemoved, cleanup.WorkspaceRemoved, cleanup.Leftovers }, - diagnostics = status == "error" ? new { containerExitCode = proc.ExitCode, stdoutTail = LastLines(proc.StdOut, 30), stderrTail = LastLines(proc.StdErr, 20) } : null, + diagnostics = status == "error" + ? new { containerExitCode = proc.ExitCode, phaseLogTail = LastLines(FailureClassifier.PhaseOutput(proc.StdOut, outcome.Phase), 40), stderrTail = LastLines(proc.StdErr, 20) } + : null, + containerLog = options.ShowLog ? proc.StdOut : null, }, RemoteTestProgram.JsonOut)); return (int)exit; } // Concise human result: environment → image/digest/sdk → tests → duration, actionable on failure. Console.WriteLine($"Remote Test: {env.Name}"); + if (selectionReason is not null) + { + Console.WriteLine(selectionReason); + } + Console.WriteLine(); Console.WriteLine($"Image: {image.Reference}"); if (image.Digest is not null) @@ -2475,38 +3521,72 @@ private static int EmitRunResult( Console.WriteLine($"SDK: {image.Sdk}"); } + if (image.ProvisionNote is not null) + { + Console.WriteLine($"Tools: {image.ProvisionNote}"); + } + + if (staging?.GitMetadataNote is not null) + { + Console.WriteLine($"Note: {staging.GitMetadataNote}"); + } + Console.WriteLine(); if (outcome.Kind is FailureKind.None or FailureKind.TestFailure) { - Console.WriteLine($"Tests: {results.Passed} passed, {results.Skipped} skipped, {results.Failed} failed"); - Console.WriteLine($"Time: {results.DurationSeconds.ToString("0.0", CultureInfo.InvariantCulture)} s"); - if (results.Failed > 0) + // Per assembly/TFM first — the same unit `dotnet test` reports on, so a failure is + // immediately attributable to one test project and one target framework. + var width = results.Assemblies.Count == 0 ? 0 : results.Assemblies.Max(a => a.Display.Length); + foreach (var a in results.Assemblies) + { + var verdict = a.Failed > 0 ? "Failed!" : "Passed!"; + Console.WriteLine( + $"{verdict,-8} {a.Display.PadRight(width)} — {a.Passed} passed, {a.Skipped} skipped, {a.Failed} failed, {a.DurationSeconds.ToString("0.0", CultureInfo.InvariantCulture)} s"); + } + + if (results.Assemblies.Count > 0) { Console.WriteLine(); - Console.WriteLine($"{results.Failed} test(s) failed:"); - foreach (var f in results.Failures) - { - Console.WriteLine(); - Console.WriteLine($" {f.ClassName}"); - Console.WriteLine($" {f.TestName}"); - if (f.Message is not null) - { - Console.WriteLine($" {f.Message}"); - } - } } + + Console.WriteLine($"Tests: {results.Passed} passed, {results.Skipped} skipped, {results.Failed} failed"); + Console.WriteLine($"Time: {results.DurationSeconds.ToString("0.0", CultureInfo.InvariantCulture)} s (tests)"); + if (elapsedSeconds is not null) + { + Console.WriteLine($"Total: {elapsedSeconds.Value.ToString("0.0", CultureInfo.InvariantCulture)} s (including image pull, restore and build)"); + } + + WriteFailureDetail(results); } else { Console.Error.WriteLine($"{outcome.Kind}: {outcome.Message}"); - var tail = LastLines(proc.StdErr, 20); - if (!string.IsNullOrWhiteSpace(tail)) + + // Any TRX that was produced before the failure still names real failing tests; report those + // first so a test-host crash after a genuine assertion failure is not reduced to a log tail. + WriteFailureDetail(results); + + // The container writes compiler/restore/test diagnostics to stdout, so a stderr-only tail + // would leave the developer with a verdict and no cause. Prefer the failing phase's own log. + var phaseLog = FailureClassifier.PhaseOutput(proc.StdOut, outcome.Phase); + var detail = FirstNonEmpty( + ErrorLines(phaseLog, 20), LastLines(phaseLog, 40), LastLines(proc.StdErr, 20), LastLines(proc.StdOut, 40)); + if (!string.IsNullOrWhiteSpace(detail)) { - Console.Error.WriteLine(tail); + Console.Error.WriteLine(); + Console.Error.WriteLine($"--- {outcome.Phase} output ---"); + Console.Error.WriteLine(detail); } } + if (options.ShowLog && !string.IsNullOrWhiteSpace(proc.StdOut)) + { + Console.WriteLine(); + Console.WriteLine("--- container log ---"); + Console.WriteLine(proc.StdOut.TrimEnd()); + } + if (cleanup.Leftovers.Count > 0) { Console.Error.WriteLine("Cleanup left resources: " + string.Join(", ", cleanup.Leftovers)); @@ -2515,6 +3595,75 @@ private static int EmitRunResult( return (int)exit; } + // Caps so a suite that fails wholesale stays readable; the counts above remain authoritative and the + // full detail is always available in --json. + private const int MaxReportedFailures = 15; + private const int MaxStackFrames = 10; + private const int MaxOutputLines = 15; + + // The detail a developer actually needs to fix a red test: fully-qualified name, which TFM, the + // assertion message, the stack, and whatever the test wrote to its output helper. + private static void WriteFailureDetail(TestRunResult results) + { + if (results.Failures.Count == 0) + { + return; + } + + Console.WriteLine(); + Console.WriteLine(results.Failures.Count == 1 ? "1 test failed:" : $"{results.Failures.Count} tests failed:"); + + foreach (var f in results.Failures.Take(MaxReportedFailures)) + { + var name = f.ClassName is not null && !f.TestName.StartsWith(f.ClassName, StringComparison.Ordinal) + ? $"{f.ClassName}.{f.TestName}" + : f.TestName; + var where = f.Framework is null ? "" : $" [{f.Framework}]"; + var took = f.DurationSeconds > 0 ? $" ({(f.DurationSeconds * 1000).ToString("0", CultureInfo.InvariantCulture)} ms)" : ""; + + Console.WriteLine(); + Console.WriteLine($" Failed {name}{where}{took}"); + WriteIndented(f.Message, " ", int.MaxValue); + + if (!string.IsNullOrWhiteSpace(f.StackTrace)) + { + Console.WriteLine(" Stack trace:"); + WriteIndented(f.StackTrace, " ", MaxStackFrames); + } + + if (!string.IsNullOrWhiteSpace(f.Output)) + { + Console.WriteLine(" Output:"); + WriteIndented(f.Output, " ", MaxOutputLines); + } + } + + if (results.Failures.Count > MaxReportedFailures) + { + Console.WriteLine(); + Console.WriteLine($" … and {results.Failures.Count - MaxReportedFailures} more failing test(s); rerun with --json for the full list."); + } + } + + private static void WriteIndented(string? text, string indent, int maxLines) + { + if (string.IsNullOrWhiteSpace(text)) + { + return; + } + + var lines = text.Replace("\r\n", "\n", StringComparison.Ordinal).Split('\n'); + foreach (var line in lines.Take(maxLines)) + { + Console.WriteLine(indent + line.TrimEnd()); + } + + if (lines.Length > maxLines) + { + Console.WriteLine($"{indent}… {lines.Length - maxLines} more line(s)"); + } + } + private static int Error(Options options, FailureKind kind, string message) { if (options.Json) @@ -2554,6 +3703,21 @@ private static string LastLines(string s, int count) var lines = s.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); return string.Join('\n', lines.TakeLast(count)); } + + // MSBuild/NuGet diagnostics, distilled from the build log so a failure names its cause. + private static string ErrorLines(string s, int count) + { + var lines = s.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .Where(l => l.Contains(" error ", StringComparison.OrdinalIgnoreCase) + || l.Contains(": error", StringComparison.OrdinalIgnoreCase) + || l.StartsWith("error", StringComparison.OrdinalIgnoreCase)) + .Distinct(StringComparer.Ordinal) + .ToList(); + return string.Join('\n', lines.TakeLast(count)); + } + + private static string FirstNonEmpty(params string[] candidates) => + candidates.FirstOrDefault(c => !string.IsNullOrWhiteSpace(c)) ?? ""; } // --------------------------------------------------------------------------------------------------- @@ -2576,9 +3740,12 @@ public static int Run() ReleaseMetadataTests(); SdkAndImageTagTests(); EnvironmentSelectionTests(); + MultiSdkRunnerTests(); UnsupportedEnvironmentTests(); TargetFrameworkTests(); CommandPlanningTests(); + ImagePreparationTests(); + SourceStagingTests(); ResultParsingTests(); FailureClassificationTests(); CancellationAndCleanupTests(); @@ -2740,6 +3907,135 @@ private static void EnvironmentSelectionTests() var notFound = EnvironmentResolver.Resolve(config, generated, "does-not-exist"); Check("unknown name reported as not found", notFound.Status == ResolutionStatus.NotFound); + + // Deterministic tie-break: the repository's own target framework answers the question. + var net10 = new TargetFrameworkInfo { TargetFrameworks = ["net10.0"] }; + var byTfm = EnvironmentResolver.Resolve(null, generated, null, net10); + Check("target framework selects the matching channel without asking", + byTfm.Status == ResolutionStatus.Resolved && byTfm.Environment!.Name == "dotnet-10-lts"); + Check("automatic selection explains itself", byTfm.SelectionReason is not null && byTfm.SelectionReason.Contains("net10.0")); + + // A single-SDK image ships one runtime, so a multi-targeted repository must not be silently + // pointed at the newest channel — those lower target frameworks would build and then fail to run. + var multiTargeted = new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net9.0", "net10.0"] }; + Check("multi-targeted repository is not sent to a single-SDK image", + EnvironmentResolver.Resolve(null, generated, null, multiTargeted).Status == ResolutionStatus.Ambiguous); + + var runner = GeneratedEnvironments.FromMultiSdkRunner( + new MultiSdkRunner { Tag = "8-9-10-11", Majors = [8, 9, 10, 11] }); + var withRunner = EnvironmentResolver.Resolve(null, [.. generated, runner], null, multiTargeted); + Check("multi-targeted repository selects the covering multi-SDK runner", + withRunner.Status == ResolutionStatus.Resolved && withRunner.Environment!.Name == "ubuntu-testrunner-8-9-10-11"); + Check("multi-SDK selection explains the whole-matrix benefit", + withRunner.SelectionReason is not null && withRunner.SelectionReason.Contains("single container")); + + Check("single-target repository still prefers the matching single-SDK channel", + EnvironmentResolver.Resolve(null, [.. generated, runner], null, net10).Environment!.Name == "dotnet-10-lts"); + + var uncovered = new TargetFrameworkInfo { TargetFrameworks = ["net7.0", "net10.0"] }; + Check("a runner that does not cover every target framework is not selected", + EnvironmentResolver.Resolve(null, [.. generated, runner], null, uncovered).Status == ResolutionStatus.Ambiguous); + + var previewOnly = new TargetFrameworkInfo { TargetFrameworks = ["net11.0"] }; + Check("preview channel is selectable by target framework", + EnvironmentResolver.Resolve(null, generated, null, previewOnly).Environment!.Name == "dotnet-11-preview"); + + var unsupportedMajor = new TargetFrameworkInfo { TargetFrameworks = ["net7.0"] }; + Check("target framework with no supported channel still asks", + EnvironmentResolver.Resolve(null, generated, null, unsupportedMajor).Status == ResolutionStatus.Ambiguous); + + var noNetTarget = new TargetFrameworkInfo { TargetFrameworks = ["netstandard2.0"] }; + Check("non-.NET target framework does not guess a channel", + EnvironmentResolver.Resolve(null, generated, null, noNetTarget).Status == ResolutionStatus.Ambiguous); + + Check("empty target framework info does not guess a channel", + EnvironmentResolver.Resolve(null, generated, null, new TargetFrameworkInfo()).Status == ResolutionStatus.Ambiguous); + + var duplicateMajor = new List(generated) + { + generated.First(e => e.ChannelMajor == 10) with { Name = "dotnet-10-alt" }, + }; + Check("two channels for the same major stay ambiguous", + EnvironmentResolver.Resolve(null, duplicateMajor, null, net10).Status == ResolutionStatus.Ambiguous); + + Check("configured environments are never auto-selected by target framework", + EnvironmentResolver.Resolve(twoConfig, generated, null, net10).Status == ResolutionStatus.Ambiguous); + + Check("channel major parsed from channel version", + generated.First(e => e.Name == "dotnet-10-lts").ChannelMajor == 10); + Check("configured environment has no channel major", + GeneratedEnvironments.FromConfigured(config.SupportedDockerEnvironments[0]).ChannelMajor == 0); + } + + private const string SampleMultiSdkTags = """ + { + "count": 8, + "results": [ + { "name": "11.0.100-preview.7" }, + { "name": "10" }, + { "name": "10.0" }, + { "name": "8-9-10-11" }, + { "name": "8.0-9.0-10.0-11.0" }, + { "name": "9-10" }, + { "name": "8.0.421-9.0.314-10.0.300-11.0.100-preview.4" }, + { "name": "mono-net8.0.418-9.0.311-10.0.103" } + ] + } + """; + + private static void MultiSdkRunnerTests() + { + Section("Multi-SDK runner discovery"); + + var runners = MultiSdkTagReader.Parse(SampleMultiSdkTags); + var tags = runners.Select(r => r.Tag).ToList(); + Check("combined major tags are discovered", tags.Contains("8-9-10-11") && tags.Contains("9-10")); + Check("single-major tags are ignored", !tags.Contains("10") && !tags.Contains("10.0")); + Check("channel and pinned combination forms are ignored", + !tags.Contains("8.0-9.0-10.0-11.0") && !tags.Contains("8.0.421-9.0.314-10.0.300-11.0.100-preview.4")); + Check("prefixed tags are ignored", !tags.Any(t => t.StartsWith("mono", StringComparison.Ordinal))); + Check("majors parsed from the tag", + runners.Single(r => r.Tag == "8-9-10-11").Majors.SequenceEqual([8, 9, 10, 11])); + Check("image reference built from the publisher repository", + runners.Single(r => r.Tag == "9-10").Reference == "codebeltnet/ubuntu-testrunner:9-10"); + + Check("malformed feed yields no runners", MultiSdkTagReader.Parse("not json").Count == 0); + Check("feed without results yields no runners", MultiSdkTagReader.Parse("""{ "count": 0 }""").Count == 0); + + // Tightest fit: cover every required major without dragging in SDKs the repository never asked for. + Check("tightest covering tag wins", + MultiSdkTagReader.Select(runners, [9, 10])!.Tag == "9-10"); + Check("wider tag used when the tight one does not cover", + MultiSdkTagReader.Select(runners, [8, 10])!.Tag == "8-9-10-11"); + Check("no covering tag returns null", + MultiSdkTagReader.Select(runners, [7, 10]) is null); + Check("single major never selects a multi-SDK runner", + MultiSdkTagReader.Select(runners, [10]) is null); + + var env = GeneratedEnvironments.FromMultiSdkRunner(runners.Single(r => r.Tag == "8-9-10-11")); + Check("runner environment is named after its tag", env.Name == "ubuntu-testrunner-8-9-10-11"); + Check("runner environment is multi-SDK", env.IsMultiSdk && env.SupportedMajors.SequenceEqual([8, 9, 10, 11])); + Check("runner environment carries no single channel", env.Channel is null && env.ChannelMajor == 0); + + // Compatibility is judged on declared majors, because presence is what lets the tests run. + var spread = new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net10.0"] }; + Check("multi-SDK image is compatible with every provided target", + TargetFrameworkInspector.CanBuild(null, spread, env.SupportedMajors).Compatible); + Check("multi-SDK image rejects a target it does not provide", + !TargetFrameworkInspector.CanBuild(null, new TargetFrameworkInfo { TargetFrameworks = ["net7.0"] }, env.SupportedMajors).Compatible); + Check("multi-SDK image still cannot build .NET Framework", + !TargetFrameworkInspector.CanBuild(null, new TargetFrameworkInfo { TargetFrameworks = ["net48"] }, env.SupportedMajors).Compatible); + + // The runtime gap that motivates the multi-SDK runner in the first place. + var sdk10 = SdkVersion.TryParse("10.0.302"); + var singleSdkSpread = TargetFrameworkInspector.CanBuild(sdk10, spread); + Check("single-SDK image is incompatible with a multi-targeted repository", !singleSdkSpread.Compatible); + Check("the incompatibility names the missing runtime and the remedy", + singleSdkSpread.Reason is not null + && singleSdkSpread.Reason.Contains("net8.0") + && singleSdkSpread.Reason.Contains(RemoteTestProgram.MultiSdkRepository)); + Check("single-SDK image remains compatible with its own single target", + TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net10.0"] }).Compatible); } private static void UnsupportedEnvironmentTests() @@ -2783,7 +4079,11 @@ private static void TargetFrameworkTests() Check("global.json parsed", sdk == "10.0.302" && roll == "latestFeature"); var sdk10 = SdkVersion.TryParse("10.0.302"); - Check("sdk builds equal/lower target", TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net10.0"] }).Compatible); + Check("sdk builds and runs its own target", TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net10.0"] }).Compatible); + // A lower target compiles on a newer SDK but has no runtime in that image, so it is not runnable + // there. This is why a multi-targeted repository needs a multi-SDK runner. + Check("sdk alone cannot run a lower target it can build", + !TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net10.0"] }).Compatible); Check("sdk cannot build newer target", !TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net11.0"] }).Compatible); Check("linux sdk cannot build net framework", !TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net48"] }).Compatible); Check("global.json disable pin mismatch is incompatible", !TargetFrameworkInspector.CanBuild(sdk10, @@ -2813,7 +4113,12 @@ private static void CommandPlanningTests() Check("entrypoint runs restore/build/test in order", entry.IndexOf("run_phase restore", StringComparison.Ordinal) < entry.IndexOf("run_phase build", StringComparison.Ordinal) && entry.IndexOf("run_phase build", StringComparison.Ordinal) < entry.IndexOf("run_phase test", StringComparison.Ordinal)); - Check("entrypoint sets NUGET_PACKAGES to the cache mount", entry.Contains("export NUGET_PACKAGES='/nuget'")); + // Trailing slash is required: NuGet's package root becomes an MSBuild SourceRoot and SourceLink + // fails the build without it. + Check("entrypoint sets NUGET_PACKAGES to the cache mount", entry.Contains("export NUGET_PACKAGES='/nuget/'")); + Check("entrypoint keeps cache entries writable across container UIDs", entry.Contains("umask 000")); + Check("nuget package root ends with a separator", + ContainerPlanner.NuGetPackagesPath("/nuget") == "/nuget/" && ContainerPlanner.NuGetPackagesPath("/nuget/") == "/nuget/"); Check("entrypoint uses --no-restore/--no-build to reuse phases", entry.Contains("--no-restore") && entry.Contains("--no-build")); Check("entrypoint honors configuration/framework/filter/coverage", entry.Contains("-c 'Release'") && entry.Contains("--framework 'net10.0'") && entry.Contains("--filter 'Category=Unit'") && entry.Contains("XPlat Code Coverage")); @@ -2857,6 +4162,136 @@ private static void CommandPlanningTests() } } + private static void ImagePreparationTests() + { + Section("Image preparation"); + + const string digest = "sha256:990d47a4f925dedf27c875271c8b592e201666536f955befef9147745652f29f"; + var tag = ImageProvisioner.DerivedTag("codebeltnet/ubuntu-testrunner:8-9-10-11", digest); + Check("prepared tag is derived from the base digest", tag == "dotnet-remote-testing/prepared:git-990d47a4f925dedf"); + Check("prepared tag is stable for the same image", ImageProvisioner.DerivedTag("other:tag", digest) == tag); + Check("prepared tag changes with the base image", + ImageProvisioner.DerivedTag("x:1", "sha256:abcdef0123456789abcdef") != tag); + Check("prepared tag needs no digest", ImageProvisioner.DerivedTag("x:1", null).StartsWith("dotnet-remote-testing/prepared:git-", StringComparison.Ordinal)); + + Check("probe asks the image for the tooling", ImageProvisioner.ProbeCommand() == "command -v git >/dev/null 2>&1"); + + var dockerfile = ImageProvisioner.Dockerfile("codebeltnet/ubuntu-testrunner:8-9-10-11"); + Check("provisioning layers onto the resolved base image", dockerfile.StartsWith("FROM codebeltnet/ubuntu-testrunner:8-9-10-11", StringComparison.Ordinal)); + Check("provisioning installs the required tooling", dockerfile.Contains("install -y --no-install-recommends git")); + Check("provisioning adapts to the image's package manager", + dockerfile.Contains("command -v apt-get") && dockerfile.Contains("command -v apk") && dockerfile.Contains("command -v microdnf")); + Check("provisioning fails loudly on an unknown package manager", dockerfile.Contains("No supported package manager")); + + // Installing needs root, but the prepared image must still run as whoever the base image runs + // as; switching the image to root changes file ownership and permission-sensitive results. + Check("provisioning does not leave a root-only image as root", !dockerfile.TrimEnd().EndsWith("USER root", StringComparison.Ordinal)); + var nonRoot = ImageProvisioner.Dockerfile("acme/runner:1", "app"); + Check("provisioning restores the base image's user", nonRoot.TrimEnd().EndsWith("USER app", StringComparison.Ordinal)); + Check("provisioning still installs as root", nonRoot.Contains("USER root", StringComparison.Ordinal)); + Check("provisioning adds no user line when the base image sets none", + !ImageProvisioner.Dockerfile("acme/runner:1", " ").Contains("USER app", StringComparison.Ordinal)); + } + + private static void SourceStagingTests() + { + Section("Source staging"); + + var root = Path.Combine(Path.GetTempPath(), "rt-stage-" + Guid.NewGuid().ToString("N")[..8]); + var source = Path.Combine(root, "repo"); + try + { + Directory.CreateDirectory(Path.Combine(source, "src")); + Directory.CreateDirectory(Path.Combine(source, "bin")); + Directory.CreateDirectory(Path.Combine(source, ".git", "refs")); + File.WriteAllText(Path.Combine(source, "src", "App.csproj"), ""); + File.WriteAllText(Path.Combine(source, "bin", "stale.dll"), "x"); + File.WriteAllText(Path.Combine(source, ".git", "HEAD"), "ref: refs/heads/main"); + + var staged = Path.Combine(root, "staged"); + var result = SourceStager.StageAsync(source, staged, CancellationToken.None).GetAwaiter().GetResult(); + + Check("staging succeeds", result.Error is null && result.StagedPath == staged); + Check("sources are staged", File.Exists(Path.Combine(staged, "src", "App.csproj"))); + + var index = AssemblyNameIndex.Build(source); + Check("project files provide the authoritative assembly casing", index["app.dll"] == "App.dll"); + Check("host build output is not staged", !Directory.Exists(Path.Combine(staged, "bin"))); + + // Without .git the staged workspace stops being a repository: MinVer/Nerdbank fall back to + // 0.0.0, SourceLink stops embedding, and any "walk up to the .git directory" repository-root + // probe resolves elsewhere — which silently changes what the tests under it observe. + Check("git metadata is staged", result.GitMetadataStaged && Directory.Exists(Path.Combine(staged, ".git"))); + Check("git metadata is staged verbatim", + File.ReadAllText(Path.Combine(staged, ".git", "HEAD")) == "ref: refs/heads/main" + && Directory.Exists(Path.Combine(staged, ".git", "refs"))); + Check("git metadata size is reported", result.GitMetadataBytes > 0); + + var without = Path.Combine(root, "staged-no-git"); + var opted = SourceStager.StageAsync(source, without, CancellationToken.None, includeGitMetadata: false).GetAwaiter().GetResult(); + Check("git metadata can be opted out", !opted.GitMetadataStaged && !Directory.Exists(Path.Combine(without, ".git"))); + Check("opting out is explained, not silent", opted.GitMetadataNote is not null); + + // A linked worktree or submodule stores .git as a "gitdir:" pointer file, not a directory. + var linked = Path.Combine(root, "linked"); + Directory.CreateDirectory(linked); + File.WriteAllText(Path.Combine(linked, "a.txt"), "a"); + File.WriteAllText(Path.Combine(linked, ".git"), $"gitdir: {Path.Combine(source, ".git")}"); + var linkedStaged = Path.Combine(root, "staged-linked"); + var linkedResult = SourceStager.StageAsync(linked, linkedStaged, CancellationToken.None).GetAwaiter().GetResult(); + Check("gitdir pointer file is resolved to the real git directory", + linkedResult.GitMetadataStaged && File.Exists(Path.Combine(linkedStaged, ".git", "HEAD"))); + + // A real linked worktree splits its git directory in two: per-worktree state here, objects + // and refs in the shared "commondir". Staging only the near half leaves a git directory git + // cannot read, so MinVer/Nerdbank fall back to 0.0.0 and SourceLink stops embedding. + var common = Path.Combine(root, "main", ".git"); + var worktreeGit = Path.Combine(common, "worktrees", "wt"); + Directory.CreateDirectory(Path.Combine(common, "objects", "pack")); + Directory.CreateDirectory(Path.Combine(common, "refs", "heads")); + Directory.CreateDirectory(worktreeGit); + File.WriteAllText(Path.Combine(common, "HEAD"), "ref: refs/heads/main"); + File.WriteAllText(Path.Combine(common, "config"), "[core]\n\tbare = false"); + File.WriteAllText(Path.Combine(common, "objects", "pack", "pack-1.pack"), "objects"); + File.WriteAllText(Path.Combine(common, "refs", "heads", "main"), "0123456789abcdef"); + File.WriteAllText(Path.Combine(worktreeGit, "HEAD"), "ref: refs/heads/feature"); + File.WriteAllText(Path.Combine(worktreeGit, "commondir"), "../.."); + File.WriteAllText(Path.Combine(worktreeGit, "gitdir"), Path.Combine(root, "wt", ".git")); + + var worktree = Path.Combine(root, "wt"); + Directory.CreateDirectory(worktree); + File.WriteAllText(Path.Combine(worktree, "a.txt"), "a"); + File.WriteAllText(Path.Combine(worktree, ".git"), $"gitdir: {worktreeGit}"); + + var worktreeStaged = Path.Combine(root, "staged-worktree"); + var worktreeResult = SourceStager.StageAsync(worktree, worktreeStaged, CancellationToken.None).GetAwaiter().GetResult(); + var stagedGit = Path.Combine(worktreeStaged, ".git"); + + Check("worktree staging carries the shared objects and refs", + worktreeResult.GitMetadataStaged + && File.Exists(Path.Combine(stagedGit, "objects", "pack", "pack-1.pack")) + && File.Exists(Path.Combine(stagedGit, "refs", "heads", "main")) + && File.Exists(Path.Combine(stagedGit, "config"))); + Check("worktree staging keeps the worktree's own HEAD", + File.ReadAllText(Path.Combine(stagedGit, "HEAD")) == "ref: refs/heads/feature"); + Check("worktree staging drops pointers to host paths", + !File.Exists(Path.Combine(stagedGit, "commondir")) && !File.Exists(Path.Combine(stagedGit, "gitdir"))); + Check("worktree staging does not register host worktrees", !Directory.Exists(Path.Combine(stagedGit, "worktrees"))); + Check("worktree staging reports the merged size", worktreeResult.GitMetadataBytes > 0); + + // A repository with no git metadata at all is ordinary, not an error. + var plain = Path.Combine(root, "plain"); + Directory.CreateDirectory(plain); + File.WriteAllText(Path.Combine(plain, "a.txt"), "a"); + var plainResult = SourceStager.StageAsync(plain, Path.Combine(root, "staged-plain"), CancellationToken.None).GetAwaiter().GetResult(); + Check("a non-git source stages without a note", plainResult.Error is null && !plainResult.GitMetadataStaged && plainResult.GitMetadataNote is null); + } + finally + { + try { Directory.Delete(root, recursive: true); } catch (Exception) { /* temp cleanup is best-effort */ } + } + } + private static void ResultParsingTests() { Section("Result parsing"); @@ -2867,12 +4302,12 @@ private static void ResultParsingTests() - Expected: foo Actual: barat StringUtilityTest.cs:line 142 + probing /workspace/tuningExpected: foo Actual: barat StringUtilityTest.cs:line 142 - + @@ -2884,9 +4319,41 @@ private static void ResultParsingTests() Check("failure detail captured", result.Failures.Count == 1 && result.Failures[0].TestName == "Sanitize_WithUnicode_ReturnsExpectedValue"); Check("failure class resolved from TestDefinitions", result.Failures[0].ClassName == "Cuemon.Text.Tests.StringUtilityTest"); Check("failure message captured", result.Failures[0].Message == "Expected: foo Actual: bar"); + Check("failure stack trace captured", result.Failures[0].StackTrace == "at StringUtilityTest.cs:line 142"); + Check("test-written output captured", result.Failures[0].Output == "probing /workspace/tuning"); + Check("failure carries its assembly and framework", + result.Failures[0].Assembly == "Cuemon.Text.Tests.dll" && result.Failures[0].Framework == "net10.0"); + + Check("per-assembly summary produced", + result.Assemblies.Count == 1 && result.Assemblies[0] is { Assembly: "Cuemon.Text.Tests.dll", Framework: "net10.0", Failed: 1 }); + Check("assembly summary displays framework", result.Assemblies[0].Display == "Cuemon.Text.Tests.dll (net10.0)"); + + Check("framework derived from the output path", + TrxParser.FrameworkFrom("/w/bin/Debug/net9.0/A.dll") == "net9.0" + && TrxParser.FrameworkFrom(@"C:\w\bin\Release\net10.0-windows\A.dll") == "net10.0-windows" + && TrxParser.FrameworkFrom("/w/bin/Debug/netstandard2.0/A.dll") == "netstandard2.0"); + Check("framework absent when the path has none", TrxParser.FrameworkFrom("/w/A.dll") is null); + Check("assembly name derived from the storage path", TrxParser.AssemblyNameFrom(@"C:\w\bin\Debug\net10.0\A.Tests.dll") == "A.Tests.dll"); + + // VSTest lower-cases the storage path; the developer knows the assembly by its real casing. + var known = new Dictionary(StringComparer.OrdinalIgnoreCase) { ["Acme.Tests.dll"] = "Acme.Tests.dll" }; + Check("assembly casing recovered from the repository's project files", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/acme.tests.dll", null, known) == "Acme.Tests.dll"); + Check("an unknown assembly keeps the name the TRX recorded", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/other.tests.dll", null, known) == "other.tests.dll"); + Check("assembly casing recovered from the class name", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/acme.tests.dll", "Acme.Tests") == "Acme.Tests.dll"); + Check("a qualified display name is reduced to its simple name", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/acme.tests.dll", "Acme.Tests, Version=1.0.0.0, Culture=neutral") == "Acme.Tests.dll"); + Check("a class name from another assembly never overrides storage", + TrxParser.AssemblyDisplayName("/w/bin/Debug/net10.0/Acme.Tests.dll", "Shared.Fixtures") == "Acme.Tests.dll"); + Check("class name alone still names the assembly", + TrxParser.AssemblyDisplayName(null, "Acme.Tests") == "Acme.Tests.dll"); + Check("neither source yields no assembly name", TrxParser.AssemblyDisplayName(null, null) is null); var merged = result.Merge(TrxParser.Parse(trx)); Check("multiple trx files aggregate", merged is { Total: 6, Failed: 2, TrxFilesParsed: 2 }); + Check("assembly summaries aggregate too", merged.Assemblies.Count == 2); } private static void FailureClassificationTests() @@ -2914,6 +4381,15 @@ private static void FailureClassificationTests() var markers = FailureClassifier.ParsePhaseMarkers("noise\n##RT_PHASE_END:restore:0##\nmore\n##RT_PHASE_END:build:2##\n"); Check("phase markers parsed from output", markers["restore"] == 0 && markers["build"] == 2); + + // Reporting the failing phase's own log — not a tail of everything — is what makes a build + // failure name the offending file instead of trailing test-runner chatter. + const string log = "restoring\n##RT_PHASE_END:restore:0##\nApp.cs(3,5): error CS1002: ; expected\n##RT_PHASE_END:build:1##\ntest chatter\n"; + Check("phase log isolates restore", FailureClassifier.PhaseOutput(log, "restore") == "restoring"); + Check("phase log isolates build", FailureClassifier.PhaseOutput(log, "build") == "App.cs(3,5): error CS1002: ; expected"); + Check("an incomplete phase yields everything after the last marker", + FailureClassifier.PhaseOutput(log, "test") == "test chatter"); + Check("phase log is empty when there is no output", FailureClassifier.PhaseOutput("", "build") == ""); } private static void CancellationAndCleanupTests() diff --git a/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 b/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 index 913600a..e8d464d 100644 --- a/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 +++ b/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 @@ -147,6 +147,97 @@ try { $badJson = Get-Json $bad.Output $codes = @($badJson.configDiagnostics | ForEach-Object { $_.Code }) Write-Result ($codes -contains 'CONFLICTING_DOCKER_SOURCE') 'reports CONFLICTING_DOCKER_SOURCE' ($codes -join ',') + + Write-Host '' + Write-Host '== 8. Unattended selection: the repository answers the environment question ==' + # $emptyRoot already contains a net10.0 project from section 6, and the injected index derives four + # channels. Resolution must land on the matching channel without -e and without asking. + $auto = Invoke-Runner @('plan', '--repo-root', $emptyRoot, '--offline', '--releases-index-file', $indexPath, '--json') + $autoJson = Get-Json $auto.Output + Write-Result ($auto.ExitCode -eq 0 -and $autoJson.environment.name -eq 'dotnet-10-lts') ` + 'target framework selects the channel with no --environment' ("exit=$($auto.ExitCode) env=$($autoJson.environment.name)") + Write-Result ([string]::IsNullOrWhiteSpace($autoJson.environment.selectionReason) -eq $false -and $autoJson.environment.selectionReason -match 'net10\.0') ` + 'automatic selection is explained in the output' $autoJson.environment.selectionReason + + Write-Host '' + Write-Host '== 9. Genuine ambiguity still stops with SelectionRequired and candidates ==' + $twoRoot = Join-Path $workspace 'two-docker' + New-Item -ItemType Directory -Path $twoRoot -Force | Out-Null + @' +{ + "version": "1", + "environments": [ + { "name": "noble", "type": "docker", "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-noble" }, + { "name": "alpine", "type": "docker", "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-alpine" } + ] +} +'@ | Set-Content -Path (Join-Path $twoRoot 'testenvironments.json') -Encoding utf8 + $twoPlan = Invoke-Runner @('plan', '--repo-root', $twoRoot, '--offline', '--json') + $twoJson = Get-Json $twoPlan.Output + Write-Result ($twoPlan.ExitCode -eq 16 -and $twoJson.failureKind -eq 'SelectionRequired') ` + 'two configured docker envs exit SelectionRequired (16)' ("exit=$($twoPlan.ExitCode) kind=$($twoJson.failureKind)") + $twoCandidates = @($twoJson.candidates) + Write-Result ($twoCandidates -contains 'noble' -and $twoCandidates -contains 'alpine') ` + 'SelectionRequired returns the candidate names to ask about' ($twoCandidates -join ',') + + Write-Host '' + Write-Host '== 10. No .NET target framework means ask, never guess ==' + $nsRoot = Join-Path $workspace 'netstandard-only' + $nsProj = Join-Path $nsRoot 'src\Lib' + New-Item -ItemType Directory -Path $nsProj -Force | Out-Null + 'netstandard2.0' | Set-Content -Path (Join-Path $nsProj 'Lib.csproj') -Encoding utf8 + $ns = Invoke-Runner @('plan', '--repo-root', $nsRoot, '--offline', '--releases-index-file', $indexPath, '--json') + $nsJson = Get-Json $ns.Output + Write-Result ($ns.ExitCode -eq 16 -and $nsJson.failureKind -eq 'SelectionRequired') ` + 'netstandard-only repository does not get a guessed channel' ("exit=$($ns.ExitCode) kind=$($nsJson.failureKind)") + + Write-Host '' + Write-Host '== 11. Multi-targeted repositories resolve to a multi-SDK runner ==' + # A Microsoft SDK image carries one runtime, so a repository spanning majors must not be pointed at + # a single-SDK image: it would build and then fail for want of a runtime. + $tagsPath = Join-Path $workspace 'multi-sdk-tags.json' + @' +{ + "count": 4, + "results": [ + { "name": "10" }, + { "name": "9-10" }, + { "name": "8-9-10-11" }, + { "name": "8.0-9.0-10.0-11.0" } + ] +} +'@ | Set-Content -Path $tagsPath -Encoding utf8 + + $multiRoot = Join-Path $workspace 'multi-targeted' + $multiProj = Join-Path $multiRoot 'test\Multi' + New-Item -ItemType Directory -Path $multiProj -Force | Out-Null + 'net9.0;net10.0' | Set-Content -Path (Join-Path $multiProj 'Multi.csproj') -Encoding utf8 + + $multi = Invoke-Runner @('plan', '--repo-root', $multiRoot, '--offline', '--releases-index-file', $indexPath, '--multi-sdk-tags-file', $tagsPath, '--json') + $multiJson = Get-Json $multi.Output + Write-Result ($multi.ExitCode -eq 0 -and $multiJson.environment.name -eq 'ubuntu-testrunner-9-10') ` + 'multi-targeted repo selects the tightest covering runner' ("exit=$($multi.ExitCode) env=$($multiJson.environment.name)") + Write-Result ($multiJson.image.reference -eq 'codebeltnet/ubuntu-testrunner:9-10') ` + 'runner image reference comes from the publisher feed' $multiJson.image.reference + Write-Result ($multiJson.compatibility.Compatible -eq $true) ` + 'every target framework is compatible with the runner' + + Write-Host '' + Write-Host '== 12. --framework narrows the environment choice too ==' + $narrowed = Invoke-Runner @('plan', '--repo-root', $multiRoot, '-f', 'net10.0', '--offline', '--releases-index-file', $indexPath, '--multi-sdk-tags-file', $tagsPath, '--json') + $narrowedJson = Get-Json $narrowed.Output + Write-Result ($narrowed.ExitCode -eq 0 -and $narrowedJson.environment.name -eq 'dotnet-10-lts') ` + 'restricting to one TFM resolves the matching single-SDK channel' ("exit=$($narrowed.ExitCode) env=$($narrowedJson.environment.name)") + + Write-Host '' + Write-Host '== 13. A single-SDK image is reported incompatible with a multi-targeted repo ==' + $forced = Invoke-Runner @('plan', '--repo-root', $multiRoot, '-e', 'dotnet-10-lts', '--offline', '--releases-index-file', $indexPath, '--json') + $forcedJson = Get-Json $forced.Output + # plan emits the full plan payload and signals the verdict through the exit code plus compatibility. + Write-Result ($forced.ExitCode -eq 7 -and $forcedJson.compatibility.Compatible -eq $false) ` + 'naming a single-SDK env for a multi-targeted repo exits SdkIncompatibility (7)' ("exit=$($forced.ExitCode) compatible=$($forcedJson.compatibility.Compatible)") + Write-Result ($forcedJson.compatibility.Reason -match 'ubuntu-testrunner') ` + 'the incompatibility points at the multi-SDK remedy' $forcedJson.compatibility.Reason } finally { Remove-Item $workspace -Recurse -Force -ErrorAction SilentlyContinue diff --git a/skills/dotnet-remote-testing/scripts/validate-skill.ps1 b/skills/dotnet-remote-testing/scripts/validate-skill.ps1 index d5ccb43..e51fdda 100644 --- a/skills/dotnet-remote-testing/scripts/validate-skill.ps1 +++ b/skills/dotnet-remote-testing/scripts/validate-skill.ps1 @@ -27,7 +27,24 @@ $contracts = @( 'Do not generate container plumbing', 'Do not hardcode .NET versions', 'never silently fall back to local', - 'reproduce' + 'reproduce', + # Devex contract: invoking the skill is the request. A capability menu instead of a test run is the + # regression these guards exist to prevent. + 'Default action: run the tests', + 'Forbidden as a first response', + 'You were invoked. That is the request. Run the tests.', + 'SelectionRequired', + 'Branch on the exit code', + # A Microsoft SDK image carries one runtime, so multi-targeted repositories need a multi-SDK runner. + # Losing this guidance means silently planning runs that build and then cannot execute. + 'codebeltnet/ubuntu-testrunner', + 'ships exactly **one** runtime', + # A staged workspace without .git stops being a repository: version stamping falls back to 0.0.0 and + # repository-root probes resolve elsewhere, which makes a suite fail here that passes in Visual + # Studio's remote testing. Losing this guidance sends the agent chasing the repository instead. + 'The staged workspace is still a repository', + # Reporting contract: a red test must be actionable from the report alone. + 'Do not compress this into a bare count' ) foreach ($needle in $contracts) { if (-not $skill.Contains($needle, [System.StringComparison]::Ordinal)) { @@ -35,12 +52,50 @@ foreach ($needle in $contracts) { } } +# Smaller models act on whatever they read first. The imperative must lead the file, ahead of any +# command enumeration they could mistake for a menu to offer the developer. +# Offsets are measured from the start of the body, not the file, so the frontmatter length does not +# affect the verdict. +$bodyMatch = [regex]::Match($skill, '(?ms)^---\r?\n.*?\r?\n---\r?\n') +$bodyStart = if ($bodyMatch.Success) { $bodyMatch.Index + $bodyMatch.Length } else { 0 } +$body = $skill.Substring($bodyStart) + +$doThisNow = $body.IndexOf('## Do this now', [System.StringComparison]::Ordinal) +$commands = $body.IndexOf('Commands: ', [System.StringComparison]::Ordinal) +if ($doThisNow -lt 0) { + throw 'SKILL.md must open with a "## Do this now" section so the default action is read first.' +} +if ($commands -ge 0 -and $commands -lt $doThisNow) { + throw 'SKILL.md lists the command surface before "## Do this now"; the imperative must come first.' +} +if ($doThisNow -gt 100) { + throw "SKILL.md places '## Do this now' too late (body offset $doThisNow); it must lead the document body." +} + +# The exit-code decision table is what makes behavior identical across models. Every documented exit +# code must be present, so no outcome is left to improvisation. +foreach ($code in 0..16) { + if (-not [regex]::IsMatch($skill, "(?m)^\|\s*``$code``\s*\|")) { + throw "SKILL.md exit-code decision table is missing exit code $code." + } +} + # Guard the governing principle: Docker complexity must not be the boundary of the capability. if (-not $skill.Contains('orchestration', [System.StringComparison]::Ordinal)) { throw 'SKILL.md must describe the skill as the orchestration layer over a deterministic runner.' } $forms = [System.IO.File]::ReadAllText((Join-Path $skillRoot 'FORMS.md')) + +# The form must gate itself. Without this, the field list reads as an intake checklist and the skill +# interrogates developers who already stated what they want. +$formsContracts = @('Autonomy gate', 'not an intake checklist') +foreach ($needle in $formsContracts) { + if (-not $forms.Contains($needle, [System.StringComparison]::Ordinal)) { + throw "FORMS.md is missing required autonomy contract: $needle" + } +} + $projectField = [regex]::Match($forms, '(?ms)^### project\s*(?.*?)(?=^### |\z)') if (-not $projectField.Success -or -not $projectField.Groups['body'].Value.Contains('- **choices:**', [System.StringComparison]::Ordinal) -or diff --git a/skills/dotnet-segregated-assets/scripts/segregate-assets.cs b/skills/dotnet-segregated-assets/scripts/segregate-assets.cs index 9a3d05e..2827282 100644 --- a/skills/dotnet-segregated-assets/scripts/segregate-assets.cs +++ b/skills/dotnet-segregated-assets/scripts/segregate-assets.cs @@ -1155,9 +1155,10 @@ public static Selection Select(string repoRoot, string projectPath, IEnumerable< // Nothing names the selected project. A file that names a different web project is that // project's topology; a lone remaining file in a sanctioned location is unattributed and - // safe to use when the project directory could not appear in it at all. + // safe to use only when there is no other web project it could instead belong to — with + // siblings in the repository, a file naming none of them is opaque, not "obviously ours". var unattributed = candidates.Where(candidate => !NamesAnotherProject(candidate.Text)).ToList(); - if (selectedDirectory.Length > 0 && unattributed.Count == 1) + if (selectedDirectory.Length > 0 && otherDirectories.Count == 0 && unattributed.Count == 1) return new Selection(unattributed[0].Path, false); return new Selection(null, true); @@ -2765,6 +2766,23 @@ private static void TestComposeFileSelectorAcceptsUnambiguousLayouts(string root Assert("compose-select: sole unattributed candidate is used", ComposeFileSelector.Select(opaque, Path.Combine(opaqueWeb, "Web.csproj"), new[] { "app/Web.csproj" }).ComposeFile is not null); + // Regression: in a multi-project repo, a root-level file naming none of the web projects is + // opaque, not "obviously ours" — it must not be silently attributed to whichever project asks. + var opaqueMulti = Path.Combine(root, "compose-opaque-multi"); + var opaqueSite = Path.Combine(opaqueMulti, "src", "Acme.Site"); + var opaqueApi = Path.Combine(opaqueMulti, "src", "Acme.Api"); + Directory.CreateDirectory(opaqueSite); + Directory.CreateDirectory(opaqueApi); + File.WriteAllText(Path.Combine(opaqueSite, "Acme.Site.csproj"), ""); + File.WriteAllText(Path.Combine(opaqueApi, "Acme.Api.csproj"), ""); + File.WriteAllText(Path.Combine(opaqueMulti, SegregateAssetsProgram.ComposeFileName), + "services:\n app-assets:\n image: codebeltnet/web-cdn-origin:2.0.0\n volumes:\n - /elsewhere:/cdnroot:ro\n"); + var opaqueMultiProjects = new[] { "src/Acme.Site/Acme.Site.csproj", "src/Acme.Api/Acme.Api.csproj" }; + Assert("compose-select: opaque root file is not attributed to a sibling project", + ComposeFileSelector.Select(opaqueMulti, Path.Combine(opaqueSite, "Acme.Site.csproj"), opaqueMultiProjects).ComposeFile is null); + Assert("compose-select: opaque root file in a multi-project repo is reported, not silently absent", + ComposeFileSelector.Select(opaqueMulti, Path.Combine(opaqueSite, "Acme.Site.csproj"), opaqueMultiProjects).BelongsToAnotherProject); + // Segment-boundary safety: Acme.Api must not match Acme.ApiGateway. var prefix = Path.Combine(root, "compose-prefix"); var gateway = Path.Combine(prefix, "src", "Acme.ApiGateway"); diff --git a/skills/dotnet-test/FORMS.md b/skills/dotnet-test/FORMS.md index 7f65217..a9c2d42 100644 --- a/skills/dotnet-test/FORMS.md +++ b/skills/dotnet-test/FORMS.md @@ -1,6 +1,10 @@ # .NET Test Input Form -Collect only unresolved fields. Prefer native structured controls when the host provides them. Otherwise use the plain-text fallback below without changing field order or defaults. +This form is a fallback for genuine ambiguity, not an intake step. `scripts/inspect-dotnet-tests.ps1` already answers every field below from the repository, so in the normal case you run it, resolve the fields from its JSON, and never open this file. The mapping from inspector output to field is in `SKILL.md` Step 1. + +Ask a field only when the inspector's evidence leaves it genuinely open. When that happens, ask that one field on its own, say what made it ambiguous, and keep the resolved fields silent — re-asking something the JSON already stated reads as if the inspection never ran. + +Prefer native structured controls when the host provides them. Otherwise use the plain-text fallback below without changing field order or defaults. ## Fields @@ -11,6 +15,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - **choices:** Dynamically list discovered test `.csproj` files relative to the repository root - **default:** The only discovered test project, or the project explicitly named by the user (Recommended) - **required:** true +- **resolved_by:** the request naming a project, or `projects[]` holding exactly one ### operation_mode @@ -21,6 +26,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Bootstrap test coverage (Recommended when no selected test project exists or it has no behavior tests) - **default:** Compute from the selected project - **required:** true +- **resolved_by:** tests present in the selected project (refactor) or absent (bootstrap) — a repository fact, never a preference to poll ### test_role @@ -33,6 +39,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Console or worker functional test - **default:** Auto-classify from repository evidence (Recommended) - **required:** true +- **resolved_by:** `projects[].role` ### application_adaptation @@ -43,6 +50,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Application and test code are both in scope - **default:** Test code only; report the required application adaptation (Recommended) - **required:** true +- **resolved_by:** `referencedApplications[].genericHost` — when true, no adaptation is needed and the field does not apply - **show_when:** `test_role` is `Console or worker functional test`, or auto-classification reports a missing Generic Host blocker ### host_ownership @@ -55,6 +63,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Shared xUnit class fixture - **default:** Auto-classify from current factory/fixture usage and isolation requirements (Recommended) - **required:** true +- **resolved_by:** existing usage — a factory per test method is focused; `IClassFixture` or one shared host is shared - **show_when:** `test_role` is `ASP.NET Core functional test` or `Console or worker functional test`, and repository evidence does not already decide focused versus shared ownership ### confirmation @@ -66,9 +75,11 @@ Collect only unresolved fields. Prefer native structured controls when the host - No - **default:** Yes (Recommended) - **required:** true +- **resolved_by:** the request itself; ask only when mutation would exceed the scope it authorized ## Presentation rules +- `required: true` means the field must be **settled** before mutation, not that it must be asked. A field settled from inspector evidence is satisfied. - Infer explicit answers from the request and inspection output; do not ask them again. - Ask one unresolved field at a time. - Present the recommended/default choice first and suffix it with `(Recommended)`. diff --git a/skills/dotnet-test/SKILL.md b/skills/dotnet-test/SKILL.md index 327e9d1..c095854 100644 --- a/skills/dotnet-test/SKILL.md +++ b/skills/dotnet-test/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-test description: > - Bootstrap or refactor .NET xUnit test projects to Codebelt conventions. Use for unit-test setup, xUnit v2-to-v3 modernization, Microsoft Testing Platform adoption, ASP.NET Core WebApplicationFactory migration, entrypoint-owned managed fixtures, reusable functional-test harnesses, and in-process console or worker tests. Classify the selected project, preserve behavior and test names, resolve compatible stable packages from NuGet, and validate restore/build/test. Do not use for NUnit/MSTest-only work, production refactoring without a test-project goal, or process-launching end-to-end harnesses. + Move .NET xUnit test projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's WebApplicationFactory and hand-rolled host plumbing with WebApplicationTestFactory, WebApplicationTest, ApplicationTestFactory, and ApplicationTest — for ASP.NET Core, console, and worker applications alike. Invoking this skill IS the request: inspect the repository and refactor immediately, never opening with a menu, a capability list, or a questionnaire. Use for WebApplicationFactory migration, xUnit v2-to-v3 modernization, Microsoft Testing Platform adoption, managed fixtures, reusable functional-test harnesses, in-process console or worker tests, and unit-test bootstrap. Preserve behavior, test names, and package ownership, then validate restore/build/test. Do NOT use for NUnit/MSTest-only work, production refactoring without a test-project goal, or process-launching end-to-end harnesses. compatibility: > Requires .NET SDK, PowerShell 7+, and network access to NuGet for dynamic package resolution. --- @@ -10,6 +10,44 @@ compatibility: > Bootstrap and refactor xUnit projects using the tested patterns from [Codebelt xUnit](https://github.com/codebeltnet/xunit) and the matching application-host patterns from [Codebelt Bootstrapper](https://github.com/codebeltnet/bootstrapper). +## This skill has one job + +**The test host comes from Codebelt, not from Microsoft, and not from a builder you write in the test project.** + +Microsoft ships `WebApplicationFactory`, and it only covers ASP.NET Core. Everything else — console apps, workers, hosted services — has no Microsoft equivalent, so teams hand-roll a `HostBuilder` in the test project and end up testing a composition root that no deployed process ever runs. Codebelt xUnit closes both gaps with one family of abstractions where the application's own entry point owns startup: + +| What the test needs | Codebelt gives you | Instead of | +|---|---|---| +| One host per test / per narrow harness (web) | `WebApplicationTestFactory.Create(..., new ManagedWebApplicationFixture())` | `new MyFactory() : WebApplicationFactory` | +| One host shared by a test class (web) | `WebApplicationTest>` | `IClassFixture>` | +| One host per test / per narrow harness (console, worker) | `ApplicationTestFactory.Create(..., new ManagedApplicationFixture())` | a hand-built `HostBuilder` in the test project | +| One host shared by a test class (console, worker) | `ApplicationTest>` | a static host cached in a test helper | + +That substitution is the deliverable. A run that leaves `WebApplicationFactory` in place, or that swaps the type while quietly rebuilding the host in test code, has not done the job no matter how green the test run looks. + +### What finishing looks like + +One thing has to be true at the end: the Codebelt abstraction constructs the host, and nothing in the selected project derives from `WebApplicationFactory` any more. How you get there — file names, helper shapes, where settings come from — is yours to choose. + +These rewrites feel like migrations and change nothing: + +- **Wrapping the factory.** Keeping `WebApplicationFactory` as a private nested class, a renamed facade, or a field inside a new `...TestApplication` type. Microsoft's host still starts the application; the wrapper only hides that from the diff. +- **Renaming the seam.** Turning `new CdnOriginTestApplication()` into `CdnOriginTestApplication.Create()` across every test file. Every call site changes and the composition root does not. +- **Importing the namespace.** Adding `using Codebelt.Extensions.Xunit;` without ever calling `WebApplicationTestFactory.Create` or deriving from `WebApplicationTest<,>`. +- **Bumping packages instead.** Raising `xunit*` or unrelated pins produces a busy diff that reads as effort. It is not the deliverable, and moving `xunit*` past the anchor in [Step 3](#step-3-resolve-packages-without-hardcoding-latest) breaks the very API you are migrating onto. + +Every one of these shipped from a real run of this skill and was reported back as a successful migration, which is the point: from inside the run, a wrapper looks like progress, and the tests stay green because the host never changed. That is why [Step 7](#step-7-validate-and-loop) ends in a verdict a script produces rather than a summary you write. Either `WebApplicationTestFactory`, `WebApplicationTest<,>`, `ApplicationTestFactory`, or `ApplicationTest<,>` appears in the project's own source, or the migration did not happen. + +## Do this now + +**You were invoked. That is the request.** Your first action is the inspector in [Step 1](#step-1-gather-evidence-before-asking-anything) — not a question, not a menu, not a plan. + +The inspector answers, from the repository itself, essentially every question you might be tempted to ask: which test projects exist, what role each one plays, whether it is already on xUnit v3 and Microsoft Testing Platform, who owns each package version, every `WebApplicationFactory` and managed-fixture usage with file and line, whether the referenced application has a Generic Host seam, and what the recommended migration is. Asking the developer to hand-type answers the JSON already contains costs them a turn and tells you nothing new. + +**Forbidden as a first response:** a numbered menu of things this skill could do; "bootstrap vs refactor vs improve coverage"; "would you like me to run a diagnostic scan first?"; listing capabilities; asking which project, role, or host ownership to use before the inspector has run. If you are about to write "What do you want to do?", run the inspector instead — its output makes the question obsolete. + +There is exactly one shape of legitimate question, and it comes *after* the evidence: the inspector reported a real blocker, or its evidence genuinely contradicts what the request asked for. See [Step 1](#step-1-gather-evidence-before-asking-anything). + ## Critical - Inspect before editing. Run `scripts/inspect-dotnet-tests.ps1` against the selected project and treat its role, package ownership, `WebApplicationFactory` inventory, and blockers as the starting contract. @@ -18,25 +56,39 @@ Bootstrap and refactor xUnit projects using the tested patterns from [Codebelt x - Use entrypoint-owned `ManagedWebApplicationFixture` and `ManagedApplicationFixture` for new and migrated functional tests. Do not emit their deprecated blocking variants; they are scheduled for removal. - Do not reconstruct an application entry point inside test code with `WebApplication.CreateBuilder`, `WebHostBuilder`, `HostBuilder`, `UseTestServer`, copied service registrations, or copied middleware. That creates a second composition root which can pass while the real `Program` is broken. - Preserve target frameworks, central package management, unrelated MSBuild configuration, existing test names, and test isolation. -- Replace every selected `WebApplicationFactory` usage. A partial migration that leaves a selected usage or package reference behind is incomplete. +- Never take an `xunit*` package past the major the Codebelt xUnit release depends on. The resolver anchors that ceiling in [Step 3](#step-3-resolve-packages-without-hardcoding-latest); newest-on-NuGet is not it. +- Edit files you are actually changing, in place. Rewriting a file wholesale flips its line endings and makes `git status` report churn that reviewers must read to discover it means nothing; a file with no semantic change must not appear in the diff at all. +- Replace every selected `WebApplicationFactory` usage. A partial migration that leaves a selected usage or package reference behind is incomplete, and a usage moved inside a wrapper type is still a usage. +- Finish on the verdict from `scripts/verify-dotnet-test-migration.ps1`, quoted as it printed. Reporting a migration complete without it leaves the one claim that matters unverified. - Keep functional testing in-process. Never add a process-launching fallback for console or worker applications. - If a selected executable has no Generic Host, adapt production startup only when application adaptation is explicitly in scope. Otherwise report the exact missing host seam and stop before changing production startup. - During bootstrap, add at least one test derived from real source behavior. Placeholder assertions such as `Assert.True(true)` do not satisfy the task. - When the request requires restore/build/test, `dotnet test` must discover the expected non-zero test count and report zero failures. An MTP executable run may supplement that gate but never replaces it; if `dotnet test` discovers zero tests, add or restore the repository-appropriate `xunit.runner.visualstudio` adapter and rerun. -## Step 1: Resolve scope and inputs - -Read `FORMS.md`. Infer fields already answered by the request or repository. Ask only for unresolved fields, one at a time, and confirm the final summary before mutation. +## Step 1: Gather evidence before asking anything -Resolve the repository root and selected `.csproj` path. Do not broaden a single-project request to every test project. - -Run: +Run the inspector first. Omit `-ProjectPath` when the request did not name a project — the inspector then discovers and classifies every test project itself: ```powershell -pwsh -NoProfile -File "/scripts/inspect-dotnet-tests.ps1" -RepoRoot "" -ProjectPath "" +pwsh -NoProfile -File "/scripts/inspect-dotnet-tests.ps1" -RepoRoot "" [-ProjectPath ""] ``` -Keep stdout as JSON. Treat a non-zero exit or a reported blocker as a real stop condition. +`` is the directory containing this `SKILL.md`; quote both paths. Keep stdout as JSON. A non-zero exit or a reported blocker is a real stop condition. + +Now read the JSON and resolve the `FORMS.md` fields from it. Almost always, all of them resolve and you proceed straight to Step 2 without asking anything: + +| `FORMS.md` field | Resolved by | Ask only when | +|---|---|---| +| `project_selection` | the request naming a project, or `projects[]` containing exactly one | `projects[]` has several and the request does not narrow them | +| `operation_mode` | tests present in the selected project → refactor; none → bootstrap | never — this is a fact about the repository, not a preference | +| `test_role` | `projects[].role` | `role` contradicts what the request explicitly asked for | +| `application_adaptation` | `referencedApplications[].genericHost` is `true` → not applicable | a missing-Generic-Host blocker is reported | +| `host_ownership` | a factory constructed per test method → focused; `IClassFixture` or one shared host → shared | usage is genuinely mixed and the request does not say | +| `confirmation` | the request itself | mutation would exceed the scope the request authorized | + +Read `FORMS.md` only when a row above actually lands in its "ask" column; it is the fallback for genuine ambiguity, not an intake wizard to run up front. When you do ask, ask that one field alone, state the evidence that made it ambiguous, and carry every already-resolved field forward silently. + +Do not broaden a single-project request to every test project. ## Step 2: Classify the project @@ -55,13 +107,25 @@ If the evidence conflicts with the requested role, report the conflict and ask b Run the resolver for the selected target frameworks and role: ```powershell -pwsh -NoProfile -File "/scripts/resolve-test-package-versions.ps1" -TargetFramework -Role +pwsh -NoProfile -File "/scripts/resolve-test-package-versions.ps1" -TargetFramework -Role [-XunitAnchorVersion ] ``` The resolver queries NuGet stable versions, tries newer candidates first, and verifies each candidate against the selected package set through isolated compatibility-project restores; it emits only a set whose combined package restore passes. If it fails, report the package, target frameworks, and restore evidence instead of guessing. +**Newest is not the ceiling for `xunit*`.** xUnit versions its own packages on its own schedule — `xunit.v3` and `xunit.runner.visualstudio` are both past 4.0.0 while [Codebelt xUnit](https://github.com/codebeltnet/xunit) still builds against the 3.x line — so "latest stable" would push a test project a whole xUnit generation past the Codebelt API it is supposed to use. The resolver therefore anchors every `xunit*` id to the Codebelt package for the role (`Codebelt.Extensions.Xunit` for `Unit`, `Codebelt.Extensions.Xunit.App` otherwise), reading the anchor's own published nuspec dependencies: + +- an id the anchor declares — today `xunit.v3.assert` and `xunit.v3.extensibility.core` — resolves **1:1** to the exact version the anchor declares; +- every other `xunit*` id resolves to the newest minor/patch **at or below the anchor's major**; +- nothing else is capped, and the anchored evidence is reported back under `xunitAnchor` plus a per-package `constraint`. + +Pass `-XunitAnchorVersion` with the `Codebelt.Extensions.Xunit*` version the repository already references — the inspector reports it under `packageOwnership` — whenever that pin is being kept, so the resolved xUnit generation matches the Codebelt release actually in use rather than the newest one on NuGet. Omit it to anchor on the newest Codebelt release. Never hand-pick an `xunit*` version above the reported anchor major; if a project genuinely needs the next xUnit generation, the Codebelt package has to move there first. + When a benchmark or smoke harness provides `DOTNET_TEST_MAXIMUM_CANDIDATES`, `DOTNET_TEST_RESOLVER_CACHE_DIR`, or `DOTNET_TEST_RESOLVER_TRACE_FILE`, honor that measured scope instead of widening the live search again. Use a small explicit candidate limit for ordinary eval smoke runs; fallback and combined-package behavior stay covered by `scripts/test-resolve-test-package-versions.ps1`. +The managed fixtures this skill targets do not exist below Codebelt xUnit **11.1.0**. A project pinned under that floor restores and builds fine today and then fails to compile the moment you write the pattern, so treat the inspector's version-floor recommendation as a prerequisite edit rather than advice — raise it in the owning props file before Step 4. + +Apply the smallest change that makes the target pattern compile. The resolver returns a complete latest-compatible set because it verifies the set as a whole, not because every member needs to move; adopting all of it turns a test-host migration into a repo-wide dependency bump the request never asked for. Bump what the pattern requires, leave working pins alone, and mention the rest as available rather than applying it. + Preserve package ownership: - Central Package Management: update or add `PackageVersion` in the owning `Directory.Packages.props`; keep project `PackageReference` items versionless. @@ -142,14 +206,23 @@ Replace every placeholder with repository evidence. Do not invent an endpoint, s ## Step 7: Validate and loop -Run the narrowest authoritative sequence that covers the selected change: +For a migration, run the gate before anything expensive — it is static analysis and it fails in seconds, so there is no reason to spend a restore and a full test run discovering that the host never moved: + +```powershell +pwsh -NoProfile -File "/scripts/verify-dotnet-test-migration.ps1" -RepoRoot "" -ProjectPath "" -ExpectedWebPattern +``` + +Use `-ExpectedApplicationPattern ` instead for a console or worker migration. The gate reruns the inspector under that postcondition and adds the checks that can only exist once the edits do: a type still deriving from `WebApplicationFactory`, a retained `Microsoft.AspNetCore.Mvc.Testing` reference, `xunit*` pins past the major the restored Codebelt package declares, and files changed under the project while the target pattern appears zero times. Exit 0 prints `PASSED`, exit 1 prints `FAILED` with numbered violations and their file and line, and exit 2 means the gate itself could not run. + +Treat `FAILED` as the answer to "is this done", not as advice. Each violation names what to change; fix them and rerun. Do not restate the verdict in your own words, and do not move on to the completion report while it still says `FAILED`. + +Then run the narrowest authoritative sequence that covers the selected change: -1. rerun `inspect-dotnet-tests.ps1`; for a web migration, pass `-ExpectedWebPattern Focused` or `-ExpectedWebPattern Shared`; for a console/worker migration, pass `-ExpectedApplicationPattern Focused` or `-ExpectedApplicationPattern Shared`. These gates require the selected Codebelt pattern with an entrypoint-owned managed fixture and reject legacy factories, deprecated blocking fixtures, and direct replacement-host construction; -2. restore the selected test project; -3. build the selected test project; -4. run `dotnet test` when restore/build/test was requested and confirm the expected non-zero test count with zero failures; a zero-discovery exit code is a failure and `dotnet run` is not a substitute; -5. for migrations, search the selected scope and confirm zero remaining `WebApplicationFactory` usages; do not treat that zero count as sufficient without the expected-pattern postcondition; -6. inspect the final diff for target-framework, package-owner, test-name, and unrelated-change drift. +1. restore the selected test project; +2. build the selected test project; +3. run `dotnet test` when restore/build/test was requested and confirm the expected non-zero test count with zero failures; a zero-discovery exit code is a failure and `dotnet run` is not a substitute; +4. inspect the final diff for target-framework, package-owner, test-name, and unrelated-change drift; +5. rerun the gate as the last action, after the final edit. An earlier `PASSED` describes an earlier state of the files, and the verdict you quote has to describe the ones you are handing over. If tests expose a migration regression, repair the preserved lifecycle or configuration behavior rather than weakening assertions. @@ -159,8 +232,9 @@ Report: - selected project and classified role; - mode and whether production application adaptation was in scope; -- package ownership and resolved versions; +- package ownership and resolved versions, including the Codebelt xUnit anchor that bounded the `xunit*` versions; - preserved migration invariants; - behavior test added or existing tests retained; -- exact restore/build/test and zero-usage-search results; +- exact restore/build/test results; +- the verdict block from the final `verify-dotnet-test-migration.ps1` run, pasted as it printed. It is the evidence for the migration claim, so a paraphrase or a remembered result from earlier in the session does not stand in for it; - blockers or validation limits. diff --git a/skills/dotnet-test/evals/evals.json b/skills/dotnet-test/evals/evals.json index 3303209..12d5f89 100644 --- a/skills/dotnet-test/evals/evals.json +++ b/skills/dotnet-test/evals/evals.json @@ -53,8 +53,9 @@ "Bootstraps the real Program entry point and does not call WebApplication.CreateBuilder, UseTestServer, new TestServer, or otherwise reconstruct the application pipeline in test source", "Preserves Production environment, in-memory settings, one application per test, and disposes both IHostTest and temporary content through matching synchronous and asynchronous Test hooks", "Keeps CompressionTest and both existing method names unchanged", + "Raises Codebelt.Extensions.Xunit.App above the 11.1.0 managed-fixture floor in the owning props file, without bumping unrelated pinned packages", "Removes Microsoft.AspNetCore.Mvc.Testing when no longer needed", - "The focused inspector postcondition succeeds, search finds no WebApplicationFactory in the selected migration, and restore/build/test succeed" + "verify-dotnet-test-migration.ps1 with -ExpectedWebPattern Focused reports PASSED, and restore/build/test succeed" ], "files": [ "evals/files/focused-web/Directory.Build.props", @@ -134,6 +135,53 @@ "evals/files/worker-functional/test/Acme.QueuePump.FunctionalTests/Acme.QueuePump.FunctionalTests.csproj", "evals/files/worker-functional/test/Acme.QueuePump.FunctionalTests/QueuePumpTest.cs" ] + }, + { + "id": 7, + "prompt": "/dotnet-test", + "expected_output": "The first action is scripts/inspect-dotnet-tests.ps1 against the repository, followed immediately by the migration work its evidence implies. No menu, capability list, questionnaire, or plan-then-stop response is produced.", + "expectations": [ + "Runs inspect-dotnet-tests.ps1 as the first action rather than replying with a question", + "Does not present a numbered menu of modes such as bootstrap versus refactor versus improve coverage", + "Does not ask which project, role, or host ownership to use when the inspector evidence already resolves them", + "Proceeds to classify and migrate using the inspector output without an intervening confirmation round-trip", + "Asks a single scoped question only if the inspector reports a real blocker or its evidence contradicts the request" + ], + "files": [ + "evals/files/focused-web/Directory.Build.props", + "evals/files/focused-web/Directory.Packages.props", + "evals/files/focused-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj", + "evals/files/focused-web/src/Acme.Cdn.Origin/Program.cs", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs" + ] + }, + { + "id": 8, + "prompt": "A previous run reported the attached Acme.Cdn.Origin functional tests as migrated off WebApplicationFactory, but the tests still exercise Microsoft's host. Finish the migration onto the focused Codebelt pattern: CdnOriginTestApplication must stop wrapping WebApplicationFactory, WebApplicationTestFactory.Create with an explicit ManagedWebApplicationFixture must own the host, Microsoft.AspNetCore.Mvc.Testing must go, and the xunit pins must return to the generation the referenced Codebelt release declares. Preserve the Production environment, per-test settings, temporary content ownership, and both existing test names.", + "expected_output": "A focused Test-derived harness owns an IHostTest created through WebApplicationTestFactory with an explicit ManagedWebApplicationFixture, no type derives from WebApplicationFactory, Microsoft.AspNetCore.Mvc.Testing is gone, xunit* is back inside the Codebelt anchor major, and verify-dotnet-test-migration.ps1 reports PASSED.", + "expectations": [ + "Recognizes that the nested private CdnOriginApplicationFactory means the migration never happened, rather than accepting the prior run's report", + "Leaves no type deriving from WebApplicationFactory, including nested, private, and renamed facades", + "Adds WebApplicationTestFactory.Create with an explicit ManagedWebApplicationFixture so Program owns startup", + "Does not keep the static Create factory method as the only change, and does not reconstruct the pipeline with WebApplication.CreateBuilder, UseTestServer, or new TestServer", + "Removes the Microsoft.AspNetCore.Mvc.Testing package reference and its version entry", + "Returns xunit.v3 and xunit.v3.runner.console to the major that Codebelt.Extensions.Xunit.App 11.2.1 declares instead of leaving them on 4.0.0", + "Preserves Production environment, per-test settings, one application per test, temporary content disposal through matching synchronous and asynchronous Test hooks, and both existing method names", + "Runs verify-dotnet-test-migration.ps1 with -ExpectedWebPattern Focused and quotes the PASSED verdict block rather than paraphrasing it" + ], + "files": [ + "evals/files/laundered-web/Directory.Build.props", + "evals/files/laundered-web/Directory.Packages.props", + "evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj", + "evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs" + ] } ] } diff --git a/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props b/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props index bd580fc..e1796e3 100644 --- a/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props +++ b/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props @@ -1,6 +1,7 @@ true + diff --git a/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj b/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj index 32f4530..38670cd 100644 --- a/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj +++ b/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj @@ -1,6 +1,7 @@ Acme.Cdn.Origin + diff --git a/skills/dotnet-test/evals/files/laundered-web/Directory.Build.props b/skills/dotnet-test/evals/files/laundered-web/Directory.Build.props new file mode 100644 index 0000000..380c35c --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/Directory.Build.props @@ -0,0 +1,13 @@ + + + net10.0 + enable + enable + $(MSBuildProjectName.EndsWith('Tests')) + + + Exe + true + + + diff --git a/skills/dotnet-test/evals/files/laundered-web/Directory.Packages.props b/skills/dotnet-test/evals/files/laundered-web/Directory.Packages.props new file mode 100644 index 0000000..47f3deb --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/Directory.Packages.props @@ -0,0 +1,11 @@ + + true + + + + + + + + + diff --git a/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj new file mode 100644 index 0000000..dd2327a --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj @@ -0,0 +1,2 @@ + + diff --git a/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs new file mode 100644 index 0000000..93ef5de --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs @@ -0,0 +1,7 @@ +var builder = WebApplication.CreateBuilder(args); +var app = builder.Build(); +app.MapGet("/compression", (IConfiguration configuration) => configuration.GetValue("Compression:Enabled", false) ? "br" : "identity"); +app.Run(); + +public partial class Program; + diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj new file mode 100644 index 0000000..38670cd --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj @@ -0,0 +1,12 @@ + + Acme.Cdn.Origin + + + + + + + + + + diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs new file mode 100644 index 0000000..abd1051 --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs @@ -0,0 +1,70 @@ +using Codebelt.Extensions.Xunit; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.Hosting; + +namespace Acme.Cdn.Origin; + +/// +/// Hosts the real origin pipeline over an isolated temporary content directory. +/// Implements the Codebelt managed fixture pattern with entrypoint-owned startup. +/// +public sealed class CdnOriginTestApplication : IAsyncDisposable, IDisposable +{ + private readonly WebApplicationFactory _factory; + private readonly TempContent _content; + + private CdnOriginTestApplication(WebApplicationFactory factory, TempContent content) + { + _factory = factory; + _content = content; + } + + public TempContent Content => _content; + + public static CdnOriginTestApplication Create(IDictionary? settings = null) + { + var content = new TempContent(); + var merged = settings is null ? new Dictionary() : new Dictionary(settings); + return new CdnOriginTestApplication(new CdnOriginApplicationFactory(merged), content); + } + + public HttpClient CreateClient() => _factory.CreateClient(); + + public void Dispose() + { + _factory.Dispose(); + _content.Dispose(); + } + + public async ValueTask DisposeAsync() + { + if (_factory is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + _factory.Dispose(); + } + + _content.Dispose(); + } + + private sealed class CdnOriginApplicationFactory : WebApplicationFactory + { + private readonly Dictionary _settings; + + public CdnOriginApplicationFactory(Dictionary settings) + { + _settings = settings; + } + + protected override void ConfigureWebHost(IWebHostBuilder builder) + { + builder.UseEnvironment(Environments.Production); + builder.ConfigureAppConfiguration((_, configuration) => configuration.AddInMemoryCollection(_settings)); + } + } +} diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs new file mode 100644 index 0000000..46e737a --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs @@ -0,0 +1,25 @@ +using Xunit; + +namespace Acme.Cdn.Origin; + +public class CompressionTest +{ + [Fact] + public async Task Get_ShouldNotCompress_WhenCompressionDisabled() + { + await using var application = CdnOriginTestApplication.Create(); + using var client = application.CreateClient(); + + Assert.Equal("identity", await client.GetStringAsync("/compression")); + } + + [Fact] + public async Task Get_ShouldCompress_WhenEnabled() + { + await using var application = CdnOriginTestApplication.Create(new Dictionary { ["Compression:Enabled"] = "true" }); + using var client = application.CreateClient(); + + Assert.Equal("br", await client.GetStringAsync("/compression")); + } +} + diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs new file mode 100644 index 0000000..c70d5fc --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs @@ -0,0 +1,18 @@ +namespace Acme.Cdn.Origin; + +public sealed class TempContent : IDisposable +{ + public TempContent() + { + Root = Path.Combine(Path.GetTempPath(), "acme-cdn-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(Root); + } + + public string Root { get; } + + public void Dispose() + { + if (Directory.Exists(Root)) { Directory.Delete(Root, true); } + } +} + diff --git a/skills/dotnet-test/references/application-functional-tests.md b/skills/dotnet-test/references/application-functional-tests.md index d6c0607..33d2620 100644 --- a/skills/dotnet-test/references/application-functional-tests.md +++ b/skills/dotnet-test/references/application-functional-tests.md @@ -31,7 +31,7 @@ Pass `ManagedApplicationFixture` explicitly for focused tests and use i A repeated focused setup may be encapsulated in a narrow `Test`-derived harness. It must accept `ITestOutputHelper`, retain the `IHostTest`, and dispose that host test plus every owned resource in both the synchronous and asynchronous `Test` disposal hooks. -After migration, run `inspect-dotnet-tests.ps1` with `-ExpectedApplicationPattern Focused` or `-ExpectedApplicationPattern Shared`. A non-zero exit is a migration failure even when restore, build, and tests pass. +After migration, run `verify-dotnet-test-migration.ps1` for the selected project with `-ExpectedApplicationPattern Focused` or `-ExpectedApplicationPattern Shared`. A `FAILED` verdict is a migration failure even when restore, build, and tests all pass, because green tests only prove the host that ran still works, not that it is the one you were asked to move to. ## Host seam gate diff --git a/skills/dotnet-test/references/migration-invariants.md b/skills/dotnet-test/references/migration-invariants.md index 99b9cf4..0b1ff73 100644 --- a/skills/dotnet-test/references/migration-invariants.md +++ b/skills/dotnet-test/references/migration-invariants.md @@ -37,4 +37,4 @@ Prefer focused `WebApplicationTestFactory` ownership when the old test construct For either ownership shape, use the entrypoint-owned `ManagedWebApplicationFixture` and pass it explicitly to factories. Migrate deprecated `BlockingManagedWebApplicationFixture` input; never emit it as a target. -After migration, search the authorized scope. Zero selected `WebApplicationFactory` identifiers is a completion gate, but it is not sufficient by itself: the chosen Codebelt pattern must be present, direct replacement-host construction must be absent, restore/build/test must pass, and lifecycle invariants must still hold. Enforce the web-pattern checks by rerunning `inspect-dotnet-tests.ps1` with `-ExpectedWebPattern Focused` or `-ExpectedWebPattern Shared`. +After migration, search the authorized scope. Zero selected `WebApplicationFactory` identifiers is a completion gate, but it is not sufficient by itself: the chosen Codebelt pattern must be present, direct replacement-host construction must be absent, restore/build/test must pass, and lifecycle invariants must still hold. Enforce all of that with `verify-dotnet-test-migration.ps1` and its `-ExpectedWebPattern` or `-ExpectedApplicationPattern` postcondition; its verdict, not a self-assessment, is what closes the migration. diff --git a/skills/dotnet-test/references/web-functional-tests.md b/skills/dotnet-test/references/web-functional-tests.md index 5e1de76..7340508 100644 --- a/skills/dotnet-test/references/web-functional-tests.md +++ b/skills/dotnet-test/references/web-functional-tests.md @@ -35,6 +35,26 @@ The test must bootstrap the real `Program` entry point. Do not reproduce `Progra When setup is repeated across many focused tests, a narrow `Test`-derived harness may own the factory result and temporary resources. Accept `ITestOutputHelper`, keep one harness per intended isolation scope, and expose a client/host rather than a second composition root. Override both `OnDisposeManagedResources` and `OnDisposeManagedResourcesAsync`: dispose the `IHostTest` and owned resources in each matching path, then call the base hook. Overriding only the synchronous hook is insufficient when callers use `await using`. +`IHostTest` derives from `ITest`, which implements both `IDisposable` and `IAsyncDisposable`, so call `_field.Dispose()` and `await _field.DisposeAsync()` directly on the field: + +```csharp +protected override void OnDisposeManagedResources() +{ + _application.Dispose(); + Content.Dispose(); + base.OnDisposeManagedResources(); +} + +protected override async ValueTask OnDisposeManagedResourcesAsync() +{ + await _application.DisposeAsync().ConfigureAwait(false); + Content.Dispose(); + await base.OnDisposeManagedResourcesAsync().ConfigureAwait(false); +} +``` + +Probing with `if (_application is IAsyncDisposable d)` is dead defensive code — the interface already guarantees it — and it hides the disposal behind a local, which the `inspect-dotnet-tests.ps1` ownership check reads as a harness that never disposes what it owns. + ## Shared xUnit fixture ownership Use `WebApplicationTest` when all tests in a class share one initialized host: @@ -67,4 +87,4 @@ public class HealthTest : WebApplicationTestExe`. - Enable Microsoft Testing Platform: `true`. - Remove `Xunit.Abstractions`; import `Xunit` for `ITestOutputHelper`. diff --git a/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 b/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 index c2863c4..edd7240 100644 --- a/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 +++ b/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 @@ -366,6 +366,19 @@ $reports = foreach ($project in $projects) { } $recommendations = [System.Collections.Generic.List[string]]::new() + # The managed fixtures this skill targets were introduced in Codebelt xUnit 11.1.0. A project pinned below that + # floor restores fine and reports no usage problem, then fails to compile the moment the pattern is written, so + # surface the required bump as evidence during inspection instead of as a build error after the edits. + $managedFixtureFloor = [version]'11.1.0' + foreach ($package in $packages) { + if ($package.id -notmatch '^Codebelt\.Extensions\.Xunit(\.App|\.Hosting(\.AspNetCore)?)?$') { continue } + $normalizedVersion = ([string]$package.version -split '-', 2)[0] + $parsedVersion = $null + if (-not [version]::TryParse($normalizedVersion, [ref]$parsedVersion)) { continue } + if ($parsedVersion -lt $managedFixtureFloor) { + $recommendations.Add("Raise $($package.id) from $($package.version) to at least $managedFixtureFloor in $($package.versionOwner); ManagedWebApplicationFixture and ManagedApplicationFixture do not exist below that version, so the required pattern cannot compile until the version is raised.") + } + } if ($xunitGeneration -eq 'v2') { $recommendations.Add('Modernize the selected project to xUnit v3 and Microsoft Testing Platform while preserving target frameworks and package ownership.') } if ([string]$properties.UseMicrosoftTestingPlatformRunner -ne 'true') { $recommendations.Add('Enable UseMicrosoftTestingPlatformRunner for the selected xUnit v3 test project, preferably in its existing shared test-project property owner.') } if ($webUsages.Count -gt 0) { $recommendations.Add('Replace every selected WebApplicationFactory usage and preserve configuration, start behavior, clients, services, disposal, and isolation.') } diff --git a/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 b/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 index db4c6c8..bc1df02 100644 --- a/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 +++ b/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 @@ -8,6 +8,10 @@ param( [string[]]$PackageId, + [string]$XunitAnchorPackageId, + + [string]$XunitAnchorVersion, + [int]$MaximumCandidates, [string]$CacheDirectory = $env:DOTNET_TEST_RESOLVER_CACHE_DIR, @@ -37,12 +41,20 @@ if ($MaximumCandidates -lt 1 -or $MaximumCandidates -gt 100) { throw "MaximumCandidates must be between 1 and 100. Found '$MaximumCandidates'." } +# xUnit versions its packages independently of this skill: xunit.v3 4.0.0 and xunit.runner.visualstudio 4.0.0 are stable +# on NuGet while Codebelt.Extensions.Xunit still builds against 3.2.2. "Newest stable" would therefore drag a test project +# a whole xUnit generation past the Codebelt API it is meant to use, so every id matching this pattern is anchored to the +# Codebelt package instead of resolved freely. +$xunitPackagePattern = '^xunit(\.|$)' +$stableVersionPattern = '^\d+(?:\.\d+){1,3}$' + function Get-ResolverCacheKey { param( [Parameter(Mandatory = $true)] [string]$RoleName, [Parameter(Mandatory = $true)] [string[]]$Frameworks, [Parameter(Mandatory = $true)] [string[]]$Packages, - [Parameter(Mandatory = $true)] [int]$CandidateLimit + [Parameter(Mandatory = $true)] [int]$CandidateLimit, + [string]$Anchor ) $seed = [ordered]@{ @@ -50,6 +62,7 @@ function Get-ResolverCacheKey { targetFrameworks = @($Frameworks | Sort-Object) packageIds = @($Packages | Sort-Object) maximumCandidates = $CandidateLimit + xunitAnchor = [string]$Anchor } | ConvertTo-Json -Compress return [System.BitConverter]::ToString(([System.Security.Cryptography.SHA256]::HashData($utf8NoBom.GetBytes($seed)))).Replace('-', '').ToLowerInvariant() } @@ -62,7 +75,8 @@ function Write-ResolverTrace { [string[]]$Packages, [int]$CandidateLimit, [bool]$CacheHit, - [double]$DurationSeconds + [double]$DurationSeconds, + [string]$Anchor ) if ([string]::IsNullOrWhiteSpace($TraceFilePath)) { return } @@ -76,6 +90,7 @@ function Write-ResolverTrace { targetFrameworks = @($Frameworks) packageIds = @($Packages) maximumCandidates = $CandidateLimit + xunitAnchor = [string]$Anchor cacheHit = $CacheHit durationSeconds = [math]::Round($DurationSeconds, 3) timestamp = [DateTimeOffset]::UtcNow.ToString('O') @@ -94,6 +109,117 @@ function Get-VersionKey { } } +function Compare-VersionText { + param([string]$Left, [string]$Right) + + $leftKey = Get-VersionKey -Version $Left + $rightKey = Get-VersionKey -Version $Right + foreach ($part in @('major', 'minor', 'patch', 'revision')) { + $leftPart = [int]$leftKey.$part + $rightPart = [int]$rightKey.$part + if ($leftPart -ne $rightPart) { return $leftPart.CompareTo($rightPart) } + } + return 0 +} + +function Get-PackageBaseAddress { + $serviceIndex = Invoke-RestMethod -Uri 'https://api.nuget.org/v3/index.json' + $address = $serviceIndex.resources | + Where-Object { $_.'@type' -eq 'PackageBaseAddress/3.0.0' } | + Select-Object -First 1 -ExpandProperty '@id' + if ([string]::IsNullOrWhiteSpace($address)) { throw 'NuGet service index did not expose PackageBaseAddress/3.0.0.' } + return $address +} + +function Get-StableVersion { + param([string]$BaseAddress, [string]$PackageId) + + $indexUrl = '{0}{1}/index.json' -f $BaseAddress, $PackageId.ToLowerInvariant() + try { $index = Invoke-RestMethod -Uri $indexUrl } catch { throw "NuGet lookup failed for '$PackageId' at '$indexUrl': $($_.Exception.Message)" } + $versions = @($index.versions | + Where-Object { $_ -match $stableVersionPattern } | + ForEach-Object { Get-VersionKey -Version $_ } | + Sort-Object major, minor, patch, revision -Descending) + return [pscustomobject]@{ source = $indexUrl; versions = $versions } +} + +function Resolve-XunitAnchor { + param([string]$BaseAddress, [string]$PackageId, [string]$Version) + + $anchorVersion = $Version + if ([string]::IsNullOrWhiteSpace($anchorVersion)) { + $index = Get-StableVersion -BaseAddress $BaseAddress -PackageId $PackageId + if (@($index.versions).Count -eq 0) { throw "NuGet returned no stable versions for the xUnit anchor package '$PackageId'." } + $anchorVersion = @($index.versions)[0].text + } + if ($anchorVersion -notmatch $stableVersionPattern) { + throw "XunitAnchorVersion must be a stable version such as '11.2.1'. Found '$anchorVersion'." + } + + $nuspecUrl = '{0}{1}/{2}/{1}.nuspec' -f $BaseAddress, $PackageId.ToLowerInvariant(), $anchorVersion.ToLowerInvariant() + try { $nuspec = Invoke-RestMethod -Uri $nuspecUrl } catch { throw "NuGet nuspec lookup failed for '$PackageId' $anchorVersion at '$nuspecUrl': $($_.Exception.Message)" } + + $pins = [ordered]@{} + foreach ($node in @($nuspec.GetElementsByTagName('dependency'))) { + $dependencyId = [string]$node.GetAttribute('id') + if ($dependencyId -notmatch $xunitPackagePattern) { continue } + $declared = (([string]$node.GetAttribute('version')).Trim('[', ']', '(', ')', ' ') -split ',')[0].Trim() + if ($declared -notmatch $stableVersionPattern) { continue } + $key = $dependencyId.ToLowerInvariant() + if ($pins.Contains($key) -and (Compare-VersionText -Left ([string]$pins[$key]) -Right $declared) -ge 0) { continue } + $pins[$key] = $declared + } + if ($pins.Count -eq 0) { + throw "'$PackageId' $anchorVersion declares no stable xunit* dependency, so the xUnit ceiling cannot be anchored to it. Pass -XunitAnchorPackageId with a Codebelt xUnit package that does." + } + + $major = @(@($pins.Values) | ForEach-Object { [int]((Get-VersionKey -Version ([string]$_)).major) } | Sort-Object -Descending)[0] + return [pscustomobject]@{ + packageId = $PackageId + version = $anchorVersion + major = $major + source = $nuspecUrl + pins = $pins + } +} + +function Select-AnchoredCandidate { + param([string]$PackageId, [object[]]$Candidates, [object]$Anchor) + + if ($null -eq $Anchor -or $PackageId -notmatch $xunitPackagePattern) { return @($Candidates) } + + $allowed = @(@($Candidates) | Where-Object { [int]($_.major) -le [int]($Anchor.major) }) + if ($allowed.Count -eq 0) { + throw "No stable '$PackageId' version at or below major $($Anchor.major) exists. That ceiling comes from $($Anchor.packageId) $($Anchor.version); raise the anchor package before raising the xUnit generation." + } + + $key = $PackageId.ToLowerInvariant() + if ($Anchor.pins.Contains($key)) { + $pinned = [string]$Anchor.pins[$key] + $exact = @($allowed | Where-Object { $_.text -eq $pinned }) + if ($exact.Count -gt 0) { + $allowed = @($exact) + @($allowed | Where-Object { $_.text -ne $pinned }) + } + } + return @($allowed) +} + +function Get-AnchorConstraint { + param([string]$PackageId, [string]$Version, [object]$Anchor) + + if ($null -eq $Anchor -or $PackageId -notmatch $xunitPackagePattern) { return 'unanchored' } + + $key = $PackageId.ToLowerInvariant() + if (-not $Anchor.pins.Contains($key)) { + return "capped at major $($Anchor.major) by $($Anchor.packageId) $($Anchor.version)" + } + $pinned = [string]$Anchor.pins[$key] + if ($pinned -eq $Version) { + return "matched 1:1 to the $pinned dependency declared by $($Anchor.packageId) $($Anchor.version)" + } + return "capped at major $($Anchor.major) by $($Anchor.packageId) $($Anchor.version) which declares $pinned" +} + function Test-PackageCompatibility { param([object[]]$Packages, [string[]]$Frameworks, [string]$Workspace) @@ -133,14 +259,24 @@ foreach ($framework in $TargetFramework) { } } +$codebeltPackage = if ($Role -eq 'Unit') { 'Codebelt.Extensions.Xunit' } else { 'Codebelt.Extensions.Xunit.App' } $packageIds = if ($PackageId -and $PackageId.Count -gt 0) { @($PackageId) } else { - $codebeltPackage = if ($Role -eq 'Unit') { 'Codebelt.Extensions.Xunit' } else { 'Codebelt.Extensions.Xunit.App' } @('Microsoft.NET.Test.Sdk', 'xunit.v3', 'xunit.v3.runner.console', 'xunit.runner.visualstudio', $codebeltPackage) } -$cacheKey = if ([string]::IsNullOrWhiteSpace($CacheDirectory)) { $null } else { Get-ResolverCacheKey -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates } +$anchorPackageId = if ([string]::IsNullOrWhiteSpace($XunitAnchorPackageId)) { $codebeltPackage } else { $XunitAnchorPackageId } +$anchoredPackageIds = @($packageIds | Where-Object { $_ -match $xunitPackagePattern }) +$flatContainerAddress = $null +$anchor = $null +if ($anchoredPackageIds.Count -gt 0) { + $flatContainerAddress = Get-PackageBaseAddress + $anchor = Resolve-XunitAnchor -BaseAddress $flatContainerAddress -PackageId $anchorPackageId -Version $XunitAnchorVersion +} +$anchorKey = if ($null -eq $anchor) { '' } else { '{0}/{1}' -f $anchor.packageId, $anchor.version } + +$cacheKey = if ([string]::IsNullOrWhiteSpace($CacheDirectory)) { $null } else { Get-ResolverCacheKey -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -Anchor $anchorKey } $cachePath = if ($null -eq $cacheKey) { $null } else { Join-Path $CacheDirectory ($cacheKey + '.json') } $startedAt = [DateTimeOffset]::UtcNow @@ -157,16 +293,12 @@ if ($null -ne $cachePath -and (Test-Path -LiteralPath $cachePath -PathType Leaf) $cached.cache | Add-Member -NotePropertyName hit -NotePropertyValue $true -Force $cached.cache | Add-Member -NotePropertyName key -NotePropertyValue $cacheKey -Force $cached.timing | Add-Member -NotePropertyName durationSeconds -NotePropertyValue $durationSeconds -Force - Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $true -DurationSeconds $durationSeconds + Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $true -DurationSeconds $durationSeconds -Anchor $anchorKey $cached | ConvertTo-Json -Depth 8 return } -$serviceIndex = Invoke-RestMethod -Uri 'https://api.nuget.org/v3/index.json' -$packageBaseAddress = $serviceIndex.resources | - Where-Object { $_.'@type' -eq 'PackageBaseAddress/3.0.0' } | - Select-Object -First 1 -ExpandProperty '@id' -if ([string]::IsNullOrWhiteSpace($packageBaseAddress)) { throw 'NuGet service index did not expose PackageBaseAddress/3.0.0.' } +if ($null -eq $flatContainerAddress) { $flatContainerAddress = Get-PackageBaseAddress } $workspace = Join-Path ([System.IO.Path]::GetTempPath()) ('dotnet-test-package-resolution-' + [Guid]::NewGuid().ToString('N')) New-Item -ItemType Directory -Path $workspace -Force | Out-Null @@ -174,15 +306,13 @@ New-Item -ItemType Directory -Path $workspace -Force | Out-Null try { $packageCandidates = [System.Collections.Generic.List[object]]::new() foreach ($id in @($packageIds | Sort-Object -Unique)) { - $indexUrl = '{0}{1}/index.json' -f $packageBaseAddress, $id.ToLowerInvariant() - try { $index = Invoke-RestMethod -Uri $indexUrl } catch { throw "NuGet lookup failed for '$id' at '$indexUrl': $($_.Exception.Message)" } - $candidates = @($index.versions | - Where-Object { $_ -match '^\d+(?:\.\d+){1,3}$' } | - ForEach-Object { Get-VersionKey -Version $_ } | - Sort-Object major, minor, patch, revision -Descending | + $index = Get-StableVersion -BaseAddress $flatContainerAddress -PackageId $id + if (@($index.versions).Count -eq 0) { throw "NuGet returned no stable versions for '$id'." } + # Anchor first, trim second: a package whose newest candidates are all above the anchored major would otherwise + # arrive here with nothing left to try. + $candidates = @(Select-AnchoredCandidate -PackageId $id -Candidates @($index.versions) -Anchor $anchor | Select-Object -First $MaximumCandidates) - if ($candidates.Count -eq 0) { throw "NuGet returned no stable versions for '$id'." } - $packageCandidates.Add([pscustomobject]@{ packageId = $id; source = $indexUrl; candidates = $candidates }) + $packageCandidates.Add([pscustomobject]@{ packageId = $id; source = $index.source; candidates = $candidates }) } function Resolve-PackageSet { @@ -239,6 +369,7 @@ try { version = $package.version source = $package.source compatibility = 'combined restore passed' + constraint = Get-AnchorConstraint -PackageId $package.packageId -Version $package.version -Anchor $anchor }) } @@ -247,6 +378,15 @@ try { role = $Role targetFrameworks = @($TargetFramework) maximumCandidates = $MaximumCandidates + xunitAnchor = if ($null -eq $anchor) { $null } else { + [ordered]@{ + packageId = $anchor.packageId + version = $anchor.version + major = $anchor.major + source = $anchor.source + declaredDependencies = $anchor.pins + } + } cache = [ordered]@{ enabled = $null -ne $cachePath hit = $false @@ -266,7 +406,7 @@ try { $result | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $cachePath -Encoding utf8 } - Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $false -DurationSeconds $durationSeconds + Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $false -DurationSeconds $durationSeconds -Anchor $anchorKey $result | ConvertTo-Json -Depth 8 } finally { if (Test-Path -LiteralPath $workspace) { Remove-Item -LiteralPath $workspace -Recurse -Force } diff --git a/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 b/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 index fc96b2b..121053a 100644 --- a/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 +++ b/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 @@ -152,6 +152,21 @@ using Codebelt.Extensions.Xunit.Hosting; public class ConsoleTest : ApplicationT $sharedApplicationReport = $sharedApplicationJson | ConvertFrom-Json if ($sharedApplicationReport.projects[0].sharedApplicationTestUsages.Count -ne 1) { throw 'Expected one shared ApplicationTest usage.' } + Write-File -Path (Join-Path $workspace 'Directory.Packages.props') -Content @' +true +'@ + Write-File -Path (Join-Path $workspace 'test/App.FunctionalTests/App.FunctionalTests.csproj') -Content @' +net10.0true +'@ + $belowFloorReport = & pwsh -NoProfile -File $scriptPath -RepoRoot $workspace -ProjectPath 'test/App.FunctionalTests/App.FunctionalTests.csproj' | ConvertFrom-Json + if (@($belowFloorReport.projects[0].recommendations | Where-Object { $_ -match 'Raise Codebelt\.Extensions\.Xunit\.App from 11\.0\.10 to at least 11\.1\.0' }).Count -ne 1) { throw 'Expected a managed-fixture version-floor recommendation for a below-floor Codebelt package.' } + + Write-File -Path (Join-Path $workspace 'Directory.Packages.props') -Content @' +true +'@ + $atFloorReport = & pwsh -NoProfile -File $scriptPath -RepoRoot $workspace -ProjectPath 'test/App.FunctionalTests/App.FunctionalTests.csproj' | ConvertFrom-Json + if (@($atFloorReport.projects[0].recommendations | Where-Object { $_ -match 'version-floor|to at least 11\.1\.0' }).Count -ne 0) { throw 'Did not expect a version-floor recommendation for a package at or above the managed-fixture floor.' } + Write-Host 'inspect-dotnet-tests.ps1 regression: PASS' } finally { if (Test-Path -LiteralPath $workspace) { Remove-Item -LiteralPath $workspace -Recurse -Force } diff --git a/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 b/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 index 922c998..8845d83 100644 --- a/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 +++ b/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 @@ -10,6 +10,7 @@ $packageBaseAddress = 'https://mock.nuget/flatcontainer/' function Reset-ResolverMock { $global:DotnetTestResolverVersions = @{} + $global:DotnetTestResolverNuspecs = @{} $global:DotnetTestResolverRestoreRequests = [System.Collections.Generic.List[object]]::new() $global:DotnetTestResolverHttpRequests = [System.Collections.Generic.List[string]]::new() $global:DotnetTestResolverFailureMode = 'Success' @@ -31,6 +32,27 @@ function Set-TestPackageVersions { $global:DotnetTestResolverVersions[$Id.ToLowerInvariant()] = @($Versions) } +function Set-TestPackageNuspec { + param( + [Parameter(Mandatory = $true)] + [string]$Id, + + [Parameter(Mandatory = $true)] + [string]$Version, + + [Parameter(Mandatory = $true)] + [hashtable]$Dependencies + ) + + $entries = (@($Dependencies.GetEnumerator() | Sort-Object Key) | ForEach-Object { + '' -f $_.Key, $_.Value + }) -join '' + # Real nuspecs repeat the same dependency once per target-framework group, so the mock does too. + $groups = (@('net10.0', 'net9.0') | ForEach-Object { '{1}' -f $_, $entries }) -join '' + $xml = '{0}{1}{2}' -f $Id, $Version, $groups + $global:DotnetTestResolverNuspecs[('{0}/{1}' -f $Id.ToLowerInvariant(), $Version.ToLowerInvariant())] = $xml +} + function Invoke-RestMethod { param([Parameter(Mandatory = $true)][string]$Uri) @@ -47,6 +69,15 @@ function Invoke-RestMethod { if ($Uri.StartsWith($packageBaseAddress, [System.StringComparison]::Ordinal)) { $segments = $Uri.TrimEnd('/') -split '/' + if ($Uri.EndsWith('.nuspec', [System.StringComparison]::OrdinalIgnoreCase)) { + $nuspecKey = '{0}/{1}' -f $segments[$segments.Count - 3], $segments[$segments.Count - 2] + if (-not $global:DotnetTestResolverNuspecs.ContainsKey($nuspecKey)) { + throw "Unexpected nuspec lookup: $Uri" + } + + return [xml]$global:DotnetTestResolverNuspecs[$nuspecKey] + } + $id = $segments[$segments.Count - 2] if (-not $global:DotnetTestResolverVersions.ContainsKey($id)) { throw "Unexpected package lookup: $Uri" @@ -108,6 +139,8 @@ function Invoke-TestResolver { [string]$CacheDirectory, + [string]$XunitAnchorVersion, + [switch]$UseDefaultCandidateLimit ) @@ -120,6 +153,8 @@ function Invoke-TestResolver { } else { $output = @(& $resolver -TargetFramework net10.0 -Role Unit -PackageId $PackageId -CacheDirectory $CacheDirectory 2>&1) } + } elseif (-not [string]::IsNullOrWhiteSpace($XunitAnchorVersion)) { + $output = @(& $resolver -TargetFramework net10.0 -Role Unit -PackageId $PackageId -MaximumCandidates $MaximumCandidates -XunitAnchorVersion $XunitAnchorVersion 2>&1) } elseif ([string]::IsNullOrWhiteSpace($CacheDirectory)) { $output = @(& $resolver -TargetFramework net10.0 -Role Unit -PackageId $PackageId -MaximumCandidates $MaximumCandidates 2>&1) } else { @@ -291,10 +326,61 @@ try { Remove-Item Env:DOTNET_TEST_MAXIMUM_CANDIDATES -ErrorAction SilentlyContinue } + # xUnit shipped stable 4.0.0 packages while Codebelt.Extensions.Xunit still declared 3.2.2, so "newest stable" + # silently jumped the test project a whole xUnit generation past the Codebelt API it targets. + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('11.2.1', '11.1.0') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2'; 'xunit.v3.extensibility.core' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.v3.assert' -Versions @('4.0.0', '3.3.0', '3.2.2') + Set-TestPackageVersions -Id 'xunit.v3' -Versions @('4.0.0', '3.3.0', '3.2.2') + $anchored = Invoke-TestResolver -PackageId @('Codebelt.Extensions.Xunit', 'xunit.v3', 'xunit.v3.assert') -MaximumCandidates 1 + $anchoredResult = Get-ResolverJsonObject -Text $anchored.text + $anchoredPackages = @($anchoredResult.packages) + Assert-Equal -Actual $anchored.exitCode -Expected 0 -Because 'anchored resolution should succeed' + Assert-Equal -Actual $anchoredResult.xunitAnchor.packageId -Expected 'Codebelt.Extensions.Xunit' -Because 'the unit role must anchor to the Codebelt xUnit package' + Assert-Equal -Actual $anchoredResult.xunitAnchor.major -Expected 3 -Because 'the xUnit ceiling must come from the Codebelt package dependency major' + Assert-Equal -Actual (($anchoredPackages | Where-Object packageId -eq 'xunit.v3.assert').version) -Expected '3.2.2' -Because 'an id the anchor declares must match it 1:1 even when a newer same-major version exists' + Assert-ContainsText -Text (($anchoredPackages | Where-Object packageId -eq 'xunit.v3.assert').constraint) -Expected 'matched 1:1' -Because 'the output must report the 1:1 anchor match' + Assert-Equal -Actual (($anchoredPackages | Where-Object packageId -eq 'xunit.v3').version) -Expected '3.3.0' -Because 'an id the anchor does not declare may take the newest minor or patch below the anchored major' + Assert-Equal -Actual (($anchoredPackages | Where-Object packageId -eq 'Codebelt.Extensions.Xunit').version) -Expected '11.2.1' -Because 'the anchor package itself stays unconstrained' + Assert-True -Condition (@($global:DotnetTestResolverRestoreRequests | Where-Object { @($_.references | Where-Object { $_.version -eq '4.0.0' }).Count -gt 0 }).Count -eq 0) -Because 'no candidate above the anchored major may reach a restore' + + # Anchoring has to precede the candidate trim, otherwise a package whose newest versions are all above the ceiling + # arrives at resolution with nothing left to try. + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('11.2.1') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.v3' -Versions @('4.0.1', '4.0.0', '3.2.2') + $trimmed = Invoke-TestResolver -PackageId @('Codebelt.Extensions.Xunit', 'xunit.v3') -MaximumCandidates 2 + $trimmedResult = Get-ResolverJsonObject -Text $trimmed.text + Assert-Equal -Actual $trimmed.exitCode -Expected 0 -Because 'the candidate limit must apply after the anchored ceiling' + Assert-Equal -Actual ((@($trimmedResult.packages) | Where-Object packageId -eq 'xunit.v3').version) -Expected '3.2.2' -Because 'the newest candidate below the anchored major must survive the candidate trim' + + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('11.2.1') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.runner.visualstudio' -Versions @('4.0.0') + $ceiling = Invoke-TestResolver -PackageId @('Codebelt.Extensions.Xunit', 'xunit.runner.visualstudio') -MaximumCandidates 2 + Assert-True -Condition ($ceiling.exitCode -ne 0) -Because 'an xunit package with no version below the anchored major must fail closed' + Assert-ContainsText -Text $ceiling.text -Expected 'at or below major 3' -Because 'the ceiling failure must name the anchored major' + + # A repository that deliberately pins an older Codebelt xUnit must resolve the xUnit generation that release declared. + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('12.0.0', '11.2.1') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '12.0.0' -Dependencies @{ 'xunit.v3.assert' = '4.0.0' } + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.v3' -Versions @('4.0.0', '3.2.2') + $pinned = Invoke-TestResolver -PackageId @('xunit.v3') -MaximumCandidates 1 -XunitAnchorVersion '11.2.1' + $pinnedResult = Get-ResolverJsonObject -Text $pinned.text + Assert-Equal -Actual $pinned.exitCode -Expected 0 -Because 'an explicit anchor version should resolve' + Assert-Equal -Actual $pinnedResult.xunitAnchor.version -Expected '11.2.1' -Because 'the explicit anchor version must be honored over the newest anchor release' + Assert-Equal -Actual ((@($pinnedResult.packages) | Where-Object packageId -eq 'xunit.v3').version) -Expected '3.2.2' -Because 'the ceiling must follow the pinned anchor release rather than the newest one' + Write-Output 'resolve-test-package-versions.ps1 regression: PASS' } finally { foreach ($name in @( 'DotnetTestResolverVersions', + 'DotnetTestResolverNuspecs', 'DotnetTestResolverRestoreRequests', 'DotnetTestResolverHttpRequests', 'DotnetTestResolverFailureMode', diff --git a/skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1 b/skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1 new file mode 100644 index 0000000..1c46ae0 --- /dev/null +++ b/skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1 @@ -0,0 +1,160 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +$workspace = Join-Path ([System.IO.Path]::GetTempPath()) ('dotnet-test-verify-' + [Guid]::NewGuid().ToString('N')) + +function Write-File { + param([string]$Path, [string]$Content) + $directory = Split-Path -Path $Path -Parent + if (-not (Test-Path -LiteralPath $directory)) { New-Item -ItemType Directory -Path $directory -Force | Out-Null } + [System.IO.File]::WriteAllText($Path, $Content, $utf8NoBom) +} + +$gate = Join-Path $PSScriptRoot 'verify-dotnet-test-migration.ps1' +$projectPath = 'test/App.FunctionalTests/App.FunctionalTests.csproj' + +function Invoke-Gate { + $output = @(& pwsh -NoProfile -File $gate -RepoRoot $workspace -ProjectPath $projectPath -ExpectedWebPattern Focused 2>&1) + return [pscustomobject]@{ + exitCode = $LASTEXITCODE + text = ($output -join [Environment]::NewLine) + } +} + +function Assert-Codes { + param([string]$Scenario, [object]$Run, [int]$ExpectedExit, [string[]]$Expected = @(), [string[]]$Forbidden = @()) + + if ($Run.exitCode -ne $ExpectedExit) { + throw "[$Scenario] expected exit $ExpectedExit, found $($Run.exitCode).`n$($Run.text)" + } + foreach ($code in $Expected) { + if ($Run.text -notmatch [regex]::Escape("[$code]")) { throw "[$Scenario] expected violation $code.`n$($Run.text)" } + } + foreach ($code in $Forbidden) { + if ($Run.text -match [regex]::Escape("[$code]")) { throw "[$Scenario] did not expect $code.`n$($Run.text)" } + } +} + +$packagesPath = Join-Path $workspace 'Directory.Packages.props' +$testProjectPath = Join-Path $workspace 'test/App.FunctionalTests/App.FunctionalTests.csproj' +$harnessPath = Join-Path $workspace 'test/App.FunctionalTests/AppTestApplication.cs' +$assetsPath = Join-Path $workspace 'test/App.FunctionalTests/obj/project.assets.json' + +$anchoredPackages = @' +true +'@ + +$legacyProjectReferences = @' +net10.0true +'@ + +$migratedProjectReferences = @' +net10.0true +'@ + +# The exact shape the failed web-cdn-origin run produced: the legacy factory survives as a private +# nested class behind a renamed facade, so every test file changes while the host never moves. +$launderedHarness = @' +using Microsoft.AspNetCore.Mvc.Testing; + +public sealed class AppTestApplication : IDisposable +{ + private readonly WebApplicationFactory _factory; + + private AppTestApplication(WebApplicationFactory factory) { _factory = factory; } + + public static AppTestApplication Create() => new AppTestApplication(new AppApplicationFactory()); + + public HttpClient CreateClient() => _factory.CreateClient(); + + public void Dispose() { _factory.Dispose(); } + + private sealed class AppApplicationFactory : WebApplicationFactory + { + } +} +'@ + +$migratedHarness = @' +using Codebelt.Extensions.Xunit; +using Codebelt.Extensions.Xunit.Hosting; +using Codebelt.Extensions.Xunit.Hosting.AspNetCore; + +public class HealthTest : Test +{ + private readonly IHostTest _application = WebApplicationTestFactory.Create(hostFixture: new ManagedWebApplicationFixture()); + + protected override void OnDisposeManagedResources() { _application.Dispose(); base.OnDisposeManagedResources(); } + + protected override async ValueTask OnDisposeManagedResourcesAsync() { await _application.DisposeAsync(); await base.OnDisposeManagedResourcesAsync(); } +} +'@ + +function Write-Assets { + param([string]$XunitAssertVersion = '3.2.2') + Write-File -Path $assetsPath -Content ('{"version":3,"targets":{"net10.0":{"Codebelt.Extensions.Xunit.App/11.2.1":{"type":"package","dependencies":{"Codebelt.Extensions.Xunit":"11.2.1","xunit.v3.assert":"' + $XunitAssertVersion + '","xunit.v3.extensibility.core":"' + $XunitAssertVersion + '"}}}},"libraries":{}}') +} + +New-Item -ItemType Directory -Path $workspace -Force | Out-Null +try { + Write-File -Path (Join-Path $workspace 'app/App.csproj') -Content @' +net10.0Exe +'@ + Write-File -Path (Join-Path $workspace 'app/Program.cs') -Content @' +public class Program { public static void Main(string[] args) { var builder = WebApplication.CreateBuilder(args); builder.Build().Run(); } } +'@ + Write-File -Path $packagesPath -Content $anchoredPackages + Write-File -Path $testProjectPath -Content $legacyProjectReferences + Write-File -Path $harnessPath -Content $launderedHarness + Write-Assets + + # A repository with no git history must not crash the churn check; it simply has nothing to compare. + $noGit = Invoke-Gate + Assert-Codes -Scenario 'laundered facade without git' -Run $noGit -ExpectedExit 1 ` + -Expected @('LAUNDERED-FACADE', 'WAF-RETAINED', 'PATTERN-MISSING', 'FIXTURE-MISSING', 'LEGACY-PACKAGE-RETAINED') ` + -Forbidden @('CHURN-WITHOUT-CONVERSION') + + # `git init` alone is enough to make every file report as untracked, which is what the churn + # check reads. No commit, and therefore no identity configuration, is involved. + & git -C $workspace init --quiet 2>&1 | Out-Null + $laundered = Invoke-Gate + Assert-Codes -Scenario 'laundered facade' -Run $laundered -ExpectedExit 1 -Expected @('LAUNDERED-FACADE', 'CHURN-WITHOUT-CONVERSION') + if ($laundered.text -notmatch 'result\s+:\s+FAILED') { throw "[laundered facade] expected a FAILED verdict line.`n$($laundered.text)" } + + # Positive control: a real migration has to pass, otherwise the gate is noise rather than signal. + Remove-Item -LiteralPath $harnessPath -Force + Write-File -Path (Join-Path $workspace 'test/App.FunctionalTests/HealthTest.cs') -Content $migratedHarness + Write-File -Path $testProjectPath -Content $migratedProjectReferences + $migrated = Invoke-Gate + Assert-Codes -Scenario 'completed migration' -Run $migrated -ExpectedExit 0 ` + -Forbidden @('LAUNDERED-FACADE', 'WAF-RETAINED', 'PATTERN-MISSING', 'FIXTURE-MISSING', 'LEGACY-PACKAGE-RETAINED', 'CHURN-WITHOUT-CONVERSION', 'XUNIT-ANCHOR-BREACH') + if ($migrated.text -notmatch 'result\s+:\s+PASSED') { throw "[completed migration] expected a PASSED verdict line.`n$($migrated.text)" } + if ($migrated.text -notmatch 'anchor\s+:\s+Codebelt\.Extensions\.Xunit\.App 11\.2\.1') { throw "[completed migration] expected the resolved anchor in the verdict.`n$($migrated.text)" } + + # Bumping an unanchored xunit id past the anchor major is the version-drift half of the same slop. + Write-File -Path $packagesPath -Content ($anchoredPackages -replace 'Include="xunit.v3" Version="3.2.2"', 'Include="xunit.v3" Version="4.0.0"') + $anchorBreach = Invoke-Gate + Assert-Codes -Scenario 'xunit major breach' -Run $anchorBreach -ExpectedExit 1 -Expected @('XUNIT-ANCHOR-BREACH') + if ($anchorBreach.text -notmatch 'xunit\.v3 is pinned to 4\.0\.0') { throw "[xunit major breach] expected the offending id and version.`n$($anchorBreach.text)" } + + # An id the anchor names itself has to match exactly, not merely stay inside the major. + Write-File -Path $packagesPath -Content ($anchoredPackages -replace 'Include="xunit.v3.assert" Version="3.2.2"', 'Include="xunit.v3.assert" Version="3.1.0"') + $exactBreach = Invoke-Gate + Assert-Codes -Scenario 'anchored id drift' -Run $exactBreach -ExpectedExit 1 -Expected @('XUNIT-ANCHOR-BREACH') + if ($exactBreach.text -notmatch 'declares 3\.2\.2') { throw "[anchored id drift] expected the declared anchor version.`n$($exactBreach.text)" } + + # Without a restored anchor the versions are unproven, which is a warning about missing evidence + # rather than a violation: reporting an unverifiable breach would be a guess. + Write-File -Path $packagesPath -Content $anchoredPackages + Remove-Item -LiteralPath $assetsPath -Force + $unverified = Invoke-Gate + Assert-Codes -Scenario 'unrestored anchor' -Run $unverified -ExpectedExit 0 -Forbidden @('XUNIT-ANCHOR-BREACH') + if ($unverified.text -notmatch 'XUNIT-ANCHOR-UNVERIFIED') { throw "[unrestored anchor] expected the unverified warning.`n$($unverified.text)" } + + $missingExpectation = @(& pwsh -NoProfile -File $gate -RepoRoot $workspace -ProjectPath $projectPath 2>&1) + if ($LASTEXITCODE -ne 2) { throw "Expected a usage error without an expected pattern, found $LASTEXITCODE.`n$($missingExpectation -join [Environment]::NewLine)" } + + Write-Output 'verify-dotnet-test-migration.ps1 checks passed.' +} finally { + if (Test-Path -LiteralPath $workspace) { Remove-Item -LiteralPath $workspace -Recurse -Force -ErrorAction SilentlyContinue } +} diff --git a/skills/dotnet-test/scripts/validate-skill.ps1 b/skills/dotnet-test/scripts/validate-skill.ps1 index b706269..2464bd1 100644 --- a/skills/dotnet-test/scripts/validate-skill.ps1 +++ b/skills/dotnet-test/scripts/validate-skill.ps1 @@ -4,7 +4,8 @@ $skillRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path $required = @( 'SKILL.md', 'FORMS.md', 'evals/evals.json', - 'scripts/inspect-dotnet-tests.ps1', 'scripts/resolve-test-package-versions.ps1', 'scripts/test-inspect-dotnet-tests.ps1', 'scripts/test-resolve-test-package-versions.ps1', + 'scripts/inspect-dotnet-tests.ps1', 'scripts/resolve-test-package-versions.ps1', 'scripts/verify-dotnet-test-migration.ps1', + 'scripts/test-inspect-dotnet-tests.ps1', 'scripts/test-resolve-test-package-versions.ps1', 'scripts/test-verify-dotnet-test-migration.ps1', 'references/unit-tests.md', 'references/web-functional-tests.md', 'references/application-functional-tests.md', 'references/bootstrapper-hosts.md', 'references/xunit-v3-modernization.md', 'references/migration-invariants.md', 'assets/unit/BehaviorTest.cs', 'assets/web/FocusedWebApplicationTest.cs', 'assets/web/SharedWebApplicationTest.cs', @@ -20,20 +21,26 @@ foreach ($relative in $required) { } $skill = [System.IO.File]::ReadAllText((Join-Path $skillRoot 'SKILL.md')) -foreach ($needle in @('WebApplicationTestFactory', 'ApplicationTestFactory', 'ManagedWebApplicationFixture', 'ManagedApplicationFixture', 'zero remaining `WebApplicationFactory`', '-ExpectedWebPattern', '-ExpectedApplicationPattern', 'second composition root')) { +foreach ($needle in @('WebApplicationTestFactory', 'ApplicationTestFactory', 'ManagedWebApplicationFixture', 'ManagedApplicationFixture', 'verify-dotnet-test-migration.ps1', '-ExpectedWebPattern', '-ExpectedApplicationPattern', 'second composition root')) { if (-not $skill.Contains($needle, [System.StringComparison]::Ordinal)) { throw "SKILL.md is missing required contract: $needle" } } if (-not $skill.Contains('An MTP executable run may supplement that gate but never replaces it', [System.StringComparison]::Ordinal)) { throw 'SKILL.md must reject MTP executable substitution for requested dotnet test validation.' } +foreach ($needle in @('What finishing looks like', 'Wrapping the factory', 'Renaming the seam', 'Bumping packages instead')) { + if (-not $skill.Contains($needle, [System.StringComparison]::Ordinal)) { throw "SKILL.md must keep the named laundering failure mode: $needle" } +} & pwsh -NoProfile -File (Join-Path $PSScriptRoot 'test-resolve-test-package-versions.ps1') if ($LASTEXITCODE -ne 0) { throw "Resolver regression failed with exit code $LASTEXITCODE." } $evals = [System.IO.File]::ReadAllText((Join-Path $skillRoot 'evals/evals.json')) -foreach ($needle in @('WebApplicationTestFactory.Create', 'ManagedWebApplicationFixture', 'WebApplication.CreateBuilder', 'focused inspector postcondition')) { - if (-not $evals.Contains($needle, [System.StringComparison]::Ordinal)) { throw "Focused-web eval is missing regression contract: $needle" } +foreach ($needle in @('WebApplicationTestFactory.Create', 'ManagedWebApplicationFixture', 'WebApplication.CreateBuilder', 'verify-dotnet-test-migration.ps1', 'nested private CdnOriginApplicationFactory')) { + if (-not $evals.Contains($needle, [System.StringComparison]::Ordinal)) { throw "Web eval is missing regression contract: $needle" } } & pwsh -NoProfile -File (Join-Path $PSScriptRoot 'test-inspect-dotnet-tests.ps1') if ($LASTEXITCODE -ne 0) { throw "Inspection regression failed with exit code $LASTEXITCODE." } +& pwsh -NoProfile -File (Join-Path $PSScriptRoot 'test-verify-dotnet-test-migration.ps1') +if ($LASTEXITCODE -ne 0) { throw "Migration gate regression failed with exit code $LASTEXITCODE." } + Write-Host 'dotnet-test skill validation: PASS' diff --git a/skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1 b/skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1 new file mode 100644 index 0000000..3d684dc --- /dev/null +++ b/skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1 @@ -0,0 +1,269 @@ +<# +.SYNOPSIS + Blocking completion gate for dotnet-test migrations. + +.DESCRIPTION + Answers one question with evidence instead of narration: did this run actually move the selected + test project onto the Codebelt entrypoint-owned host, or did it only rearrange code around the + host it was supposed to replace? + + The inspector already knows how to recognize the target pattern, so this wraps + inspect-dotnet-tests.ps1 rather than reimplementing its regexes, then adds the checks that only + make sense after the edits exist: the xUnit anchor the resolver established, retained legacy + packages, laundered WebApplicationFactory facades, and edits that produced churn without + conversion. It renders one verdict a reviewer can read without parsing JSON. +#> +param( + [string]$RepoRoot = (Get-Location).Path, + [Parameter(Mandatory)] + [string]$ProjectPath, + [ValidateSet('Focused', 'Shared')] + [string]$ExpectedWebPattern, + [ValidateSet('Focused', 'Shared')] + [string]$ExpectedApplicationPattern, + [switch]$Json +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +[Console]::OutputEncoding = $utf8NoBom +$OutputEncoding = $utf8NoBom + +# Usage problems exit 2 so a caller can tell "the gate could not run" from "the migration failed". +# Write-Error would terminate under the Stop preference above and surface as exit 1, collapsing that +# distinction into the failure code. +function Exit-WithUsageError { + param([string]$Message) + [Console]::Error.WriteLine($Message) + exit 2 +} + +if ([string]::IsNullOrWhiteSpace($ExpectedWebPattern) -and [string]::IsNullOrWhiteSpace($ExpectedApplicationPattern)) { + Exit-WithUsageError 'Specify -ExpectedWebPattern or -ExpectedApplicationPattern. The gate verifies a named target pattern; without one there is nothing to verify.' +} +if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern) -and -not [string]::IsNullOrWhiteSpace($ExpectedApplicationPattern)) { + Exit-WithUsageError 'ExpectedWebPattern and ExpectedApplicationPattern are mutually exclusive.' +} + +$violations = [System.Collections.Generic.List[object]]::new() +$warnings = [System.Collections.Generic.List[object]]::new() + +function Add-Violation { + param([string]$Code, [string]$Message, [string]$Evidence) + $violations.Add([pscustomobject]@{ code = $Code; message = $Message; evidence = $Evidence }) +} + +function Add-Warning { + param([string]$Code, [string]$Message, [string]$Evidence) + $warnings.Add([pscustomobject]@{ code = $Code; message = $Message; evidence = $Evidence }) +} + +function Get-MajorVersion { + param([string]$Version) + if ([string]::IsNullOrWhiteSpace($Version)) { return $null } + $core = ($Version -split '-', 2)[0] + $first = ($core -split '\.')[0] + $parsed = 0 + if ([int]::TryParse($first, [ref]$parsed)) { return $parsed } + return $null +} + +$repoRootPath = (Resolve-Path -LiteralPath $RepoRoot).Path +$scriptRoot = Split-Path -Path $PSCommandPath -Parent +$inspector = Join-Path $scriptRoot 'inspect-dotnet-tests.ps1' +if (-not (Test-Path -LiteralPath $inspector -PathType Leaf)) { + Exit-WithUsageError "The inspector was not found next to this gate: $inspector" +} + +# --- Run the inspector under the expected-pattern postcondition ------------------------------- +$inspectorArguments = @('-NoProfile', '-File', $inspector, '-RepoRoot', $repoRootPath, '-ProjectPath', $ProjectPath) +if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { $inspectorArguments += @('-ExpectedWebPattern', $ExpectedWebPattern) } +if (-not [string]::IsNullOrWhiteSpace($ExpectedApplicationPattern)) { $inspectorArguments += @('-ExpectedApplicationPattern', $ExpectedApplicationPattern) } + +$inspectorOutput = @(& pwsh @inspectorArguments 2>&1) +$inspectorExit = $LASTEXITCODE +$inspectorText = ($inspectorOutput -join [Environment]::NewLine) +$report = $null +try { + $report = ($inspectorText | ConvertFrom-Json).projects[0] +} catch { + Write-Output '================ DOTNET-TEST MIGRATION VERDICT ================' + Write-Output "project : $ProjectPath" + Write-Output 'result : ERROR - the inspector did not return parseable JSON' + Write-Output '' + Write-Output $inspectorText + Write-Output '===============================================================' + exit 2 +} + +foreach ($blocker in @($report.blockers)) { + $code = switch -Regex ($blocker) { + 'still contains WebApplicationFactory' { 'WAF-RETAINED'; break } + 'constructs a replacement host' { 'REPLACEMENT-HOST'; break } + 'deprecated Blocking' { 'BLOCKING-FIXTURE'; break } + 'must explicitly use Managed' { 'FIXTURE-MISSING'; break } + 'does not dispose it through both' { 'DISPOSAL-INCOMPLETE'; break } + 'postcondition requires' { 'PATTERN-MISSING'; break } + 'cannot be applied because' { 'ROLE-MISMATCH'; break } + default { 'INSPECTOR-BLOCKER' } + } + Add-Violation -Code $code -Message $blocker -Evidence 'inspect-dotnet-tests.ps1' +} + +# --- Laundered facade ------------------------------------------------------------------------- +# Wrapping the legacy factory in a new type - a private nested subclass, a renamed facade, a +# constructor turned into a static Create - keeps the Microsoft host in charge while the diff looks +# like a migration. Name it separately from the generic retained-usage blocker so the report says +# what actually happened rather than leaving the reader to infer it from a line number. +foreach ($declaration in @($report.inheritance)) { + if ($declaration.baseTypes -match '\bWebApplicationFactory\s*<') { + Add-Violation -Code 'LAUNDERED-FACADE' ` + -Message "Type '$($declaration.type)' still derives from WebApplicationFactory. Wrapping, nesting, or renaming the legacy factory keeps Microsoft's host in charge; the deliverable is that the Codebelt abstraction owns the host instead." ` + -Evidence "$($declaration.path):$($declaration.line)" + } +} + +# --- Retained legacy packages ----------------------------------------------------------------- +$legacyWebPackages = @('Microsoft.AspNetCore.Mvc.Testing') +if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { + foreach ($package in @($report.packageOwnership)) { + if ($legacyWebPackages -notcontains $package.id) { continue } + Add-Violation -Code 'LEGACY-PACKAGE-RETAINED' ` + -Message "$($package.id) is still referenced. It exists to supply WebApplicationFactory; keeping it after the migration leaves the replaced host one using directive away from returning." ` + -Evidence "$($package.referenceOwner) (version owner: $($package.versionOwner))" + } +} + +# --- xUnit anchor breach ---------------------------------------------------------------------- +# The resolver anchors xunit* to the Codebelt release in use. Nothing re-checks that after the +# edits, so a well-meant "bump everything to latest" can silently push the project a whole xUnit +# generation past the API it was migrated onto. project.assets.json records the anchor's own +# declared dependencies, which makes this verifiable offline from what actually restored. +$projectFullPath = if ([System.IO.Path]::IsPathRooted($ProjectPath)) { $ProjectPath } else { Join-Path $repoRootPath $ProjectPath } +$assetsPath = Join-Path (Split-Path -Path $projectFullPath -Parent) 'obj/project.assets.json' +$anchorId = $null +$anchorVersion = $null +$anchorMajor = $null +$anchorDeclared = @{} +if (Test-Path -LiteralPath $assetsPath -PathType Leaf) { + try { + $assets = [System.IO.File]::ReadAllText($assetsPath, $utf8NoBom) | ConvertFrom-Json + foreach ($targetProperty in $assets.targets.PSObject.Properties) { + foreach ($libraryProperty in $targetProperty.Value.PSObject.Properties) { + if ($libraryProperty.Name -notmatch '^Codebelt\.Extensions\.Xunit(?:\.App)?/(?.+)$') { continue } + $library = $libraryProperty.Value + if ($null -eq $library.PSObject.Properties['dependencies']) { continue } + foreach ($dependency in $library.dependencies.PSObject.Properties) { + if ($dependency.Name -notlike 'xunit*') { continue } + $anchorId = ($libraryProperty.Name -split '/')[0] + $anchorVersion = $Matches['version'] + $anchorDeclared[$dependency.Name] = [string]$dependency.Value + } + } + } + } catch { + Add-Warning -Code 'XUNIT-ANCHOR-UNREADABLE' -Message "project.assets.json could not be parsed, so the xUnit anchor was not verified: $($_.Exception.Message)" -Evidence $assetsPath + } +} + +if ($anchorDeclared.Count -gt 0) { + $anchorMajor = Get-MajorVersion -Version (@($anchorDeclared.Values)[0]) + foreach ($package in @($report.packageOwnership)) { + if ($package.id -notlike 'xunit*') { continue } + $major = Get-MajorVersion -Version $package.version + if ($null -eq $major) { continue } + if ($anchorDeclared.ContainsKey($package.id)) { + $expected = $anchorDeclared[$package.id] + if ($package.version -ne $expected) { + Add-Violation -Code 'XUNIT-ANCHOR-BREACH' ` + -Message "$($package.id) is pinned to $($package.version) but $anchorId $anchorVersion declares $expected. An id the anchor names resolves 1:1 to the version it declares." ` + -Evidence $package.versionOwner + } + } elseif ($null -ne $anchorMajor -and $major -gt $anchorMajor) { + Add-Violation -Code 'XUNIT-ANCHOR-BREACH' ` + -Message "$($package.id) is pinned to $($package.version), past major $anchorMajor of the $anchorId $anchorVersion anchor. Newest-on-NuGet is not the ceiling; the Codebelt package has to move to the next xUnit generation first." ` + -Evidence $package.versionOwner + } + } +} else { + Add-Warning -Code 'XUNIT-ANCHOR-UNVERIFIED' ` + -Message 'No restored Codebelt.Extensions.Xunit anchor was found, so xunit* versions were not bounded. Restore the project and rerun this gate to verify them.' ` + -Evidence $assetsPath +} + +# --- Churn without conversion ----------------------------------------------------------------- +# The most honest single signal that a run produced motion instead of migration: files under the +# selected project changed, yet not one line of the target pattern exists. Renaming a constructor +# to a static factory method reads as progress in a summary and as nothing at all in a diff. +$patternUsageCount = if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { + @($report.focusedWebApplicationTestFactoryUsages).Count + @($report.sharedWebApplicationTestUsages).Count +} else { + @($report.focusedApplicationTestFactoryUsages).Count + @($report.sharedApplicationTestUsages).Count +} +$projectDirectoryRelative = Split-Path -Path $report.project -Parent +if ($patternUsageCount -eq 0 -and -not [string]::IsNullOrWhiteSpace($projectDirectoryRelative)) { + $changed = @() + try { + $changed = @(& git -C $repoRootPath status --porcelain -- $projectDirectoryRelative 2>$null | Where-Object { $_ -notmatch '[\\/](bin|obj)[\\/]' }) + } catch { + $changed = @() + } + if ($changed.Count -gt 0) { + Add-Violation -Code 'CHURN-WITHOUT-CONVERSION' ` + -Message "$($changed.Count) file(s) under the selected project changed, yet the target pattern appears zero times. Edits that rename, wrap, or reformat the existing host produce a reviewable diff without performing the migration." ` + -Evidence (($changed | Select-Object -First 8) -join '; ') + } +} + +# --- Verdict ---------------------------------------------------------------------------------- +$result = if ($violations.Count -eq 0) { 'PASSED' } else { 'FAILED' } +$expectation = if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { + "$ExpectedWebPattern ASP.NET Core web pattern" +} else { + "$ExpectedApplicationPattern console/worker application pattern" +} + +Write-Output '================ DOTNET-TEST MIGRATION VERDICT ================' +Write-Output "project : $($report.project)" +Write-Output "role : $($report.role)" +Write-Output "expected : $expectation" +if ($null -ne $anchorId) { Write-Output "anchor : $anchorId $anchorVersion (xunit major $anchorMajor)" } +Write-Output "result : $result ($($violations.Count) violation(s), $($warnings.Count) warning(s))" +if ($violations.Count -gt 0) { + Write-Output '' + Write-Output 'VIOLATIONS' + $index = 1 + foreach ($violation in $violations) { + Write-Output (" {0}. [{1}] {2}" -f $index, $violation.code, $violation.message) + Write-Output (" evidence: {0}" -f $violation.evidence) + $index++ + } +} +if ($warnings.Count -gt 0) { + Write-Output '' + Write-Output 'WARNINGS' + $index = 1 + foreach ($warning in $warnings) { + Write-Output (" {0}. [{1}] {2}" -f $index, $warning.code, $warning.message) + Write-Output (" evidence: {0}" -f $warning.evidence) + $index++ + } +} +Write-Output '===============================================================' + +if ($Json) { + Write-Output ([ordered]@{ + project = $report.project + role = $report.role + expected = $expectation + result = $result + inspectorExitCode = $inspectorExit + xunitAnchor = [ordered]@{ id = $anchorId; version = $anchorVersion; major = $anchorMajor; declared = $anchorDeclared } + violations = @($violations) + warnings = @($warnings) + } | ConvertTo-Json -Depth 6) +} + +if ($violations.Count -gt 0) { exit 1 } +exit 0 diff --git a/skills/git-visual-squash-summary/SKILL.md b/skills/git-visual-squash-summary/SKILL.md index 8e2fdd1..71c5ddf 100644 --- a/skills/git-visual-squash-summary/SKILL.md +++ b/skills/git-visual-squash-summary/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-squash-summary description: > - Turn many commits into a curated grouped squash summary compatible with the opinionated wording style of git-visual-commits. Use when the user asks to squash a branch into a concise summary, write a squash-and-merge summary, summarize this branch, summarize a commit range or PR as grouped lines, clean up noisy commit history, or asks for a curated summary without committing. For normal squash-and-merge requests, default to the full current feature branch from merge-base to HEAD against the base branch instead of a same-named tracking remote, include commits from all authors unless the user explicitly narrows by author, and do not ask for yolo because the skill is read-only. Returns grouped lines only, resolves the cumulative base-to-HEAD diff first so reverted churn disappears, preserves identifiers, merges overlap, drops noise, and avoids changelog wording. + Turn many commits into a curated grouped squash summary for squash-and-merge contexts. Use when the user asks to squash a branch, summarize PR commits, or clean up history. Defaults to full feature branch against base (not tracking remote), includes all authors unless narrowed, and acts immediately—the skill is read-only with no permission-seeking. Returns grouped lines only, resolving the cumulative diff to drop reverted churn, preserving identifiers and overlap, and avoiding changelog wording. A bare invocation is a complete request: run git commands immediately and return summary lines, never an instruction recap or permission question. --- # Git Visual Squash Summary @@ -16,6 +16,26 @@ This skill has one job: produce a ready-to-paste squash-and-merge summary for th This skill answers one question: **What would this branch effectively do if it were squashed into one commit now?** +## Start Here: The First Response Is the Summary + +Invoking this skill is the request. Nothing needs confirming, because the skill mutates nothing and the scope is derivable on your own: the current branch against its base branch. A confirmation round-trip costs the user a turn and returns no information you could not have resolved yourself with `git`. + +So the first thing to do after loading this skill is run the read-only commands in Step 1 — not compose a reply. The first thing the user sees is the finished grouped summary. + +A response from this skill is one of exactly three things: + +1. The grouped summary lines. This is the normal case and covers nearly every invocation. +2. `No branch changes to summarize.` when every safe base-branch comparison is genuinely empty. +3. One direct question naming the missing base branch or range — only after the Step 1 fallbacks have all been tried and failed. + +Everything else is a failed invocation, including: + +- Reciting these instructions back as "I understand the instructions" or a list of "I will ..." promises. Quoting the rules is not evidence of following them; running the commands is, and the user cannot act on a restatement of your own prompt. +- Offering to do the thing already asked for: "Would you like me to generate a squash summary of your current branch now?" +- Announcing a plan and stopping before any `git` command has run. + +If a sentence you are drafting starts with "I will" or "Would you like", delete it and run `git` instead. The summary is the acknowledgment. + ## Deterministic Reduction Model Use this model for every resolved scope: @@ -74,6 +94,7 @@ Do not classify commit 1, then commit 2, then commit 3 and merge duplicate prose - A bare invocation such as `git-visual-squash-summary` or `/git-visual-squash-summary` is itself a complete request: resolve the current branch against the base branch, then return the grouped summary directly. - Never require, infer, or ask for `yolo` / `auto`. Those modes approve mutating workflows; this skill is read-only and should act directly. - Do not collect commit-set parameters through follow-up questions, widgets, or choice UIs for ordinary squash-and-merge requests. +- Do not answer an invocation with an acknowledgment, a restatement of these rules, or an offer to proceed. Run the commands and return the summary. - Do not ask the user to choose between earlier branch commits and later branch commits such as changelog, version-bump, or release-finalization follow-ups. They are part of the branch unless the user explicitly narrows scope. - Do not stop after comparing `HEAD` to a same-named tracking branch such as `origin/`. That only proves local sync with the remote copy of the feature branch, not that there is nothing to summarize. @@ -93,7 +114,7 @@ Resolve the commit set in this order: Never turn steps 2 or 3 into a user-facing choice. Resolve them automatically and continue. Never add `--author`, `--committer`, current-user, current-email, current-contributor, or identity-mode filters while resolving ordinary branch-level squash summaries. Author metadata may help understand ownership, but it must not narrow the default commit set. Do not stop to ask whether the latest branch commit "should count". If it is on the branch, it is in scope by default. -Do not open with "What would you like me to summarize?" when the user invoked this skill directly or otherwise already asked for a squash summary. +Do not open with "What would you like me to summarize?" or "Would you like me to generate it now?" when the user invoked this skill directly or otherwise already asked for a squash summary. Both questions ask the user to repeat a request they already made. If every safe base-branch comparison is genuinely empty, say `No branch changes to summarize.` and stop. Do not ask for a hypothetical range or demo. Helpful read-only commands: @@ -237,6 +258,7 @@ Output the finished grouped summary lines and stop. Do not run `git commit`, `gi - Chronological narration of each commit in order. - Dumping raw commit subjects line by line. - Preserving reverted dependency or version churn just because it happened in history. +- Restating the skill's own rules as an "I understand the instructions" preamble, then asking permission to start. - Asking the user to choose among commits that are all on the current feature branch when they asked for a squash summary of that branch. - Presenting commit-selection widgets or multiple-choice prompts for ordinary branch-level squash requests. - Filtering the branch to the current user's or current contributor's commits, or treating "my changes" as the default scope. diff --git a/skills/git-visual-squash-summary/evals/evals.json b/skills/git-visual-squash-summary/evals/evals.json index 5d12c1a..9422bc0 100644 --- a/skills/git-visual-squash-summary/evals/evals.json +++ b/skills/git-visual-squash-summary/evals/evals.json @@ -211,6 +211,18 @@ "Produces a small number of high-signal lines independent of commit count", "Does not emit one line per fixup or implementation swap" ] + }, + { + "id": 19, + "prompt": "/git-visual-squash-summary", + "expected_output": "The very first response contains the grouped summary lines for the current branch, with no instruction recap and no request for permission to begin.", + "expectations": [ + "Runs the read-only git resolution commands before composing any user-facing reply", + "Does not restate the skill's rules as an `I understand the instructions` or `I will ...` preamble", + "Does not ask `Would you like me to generate a squash summary of your current branch now?` or any equivalent offer to proceed", + "Does not announce a plan and stop before any git command has run", + "Returns one of only three permitted response shapes: the grouped lines, `No branch changes to summarize.`, or a single base-branch question after every Step 1 fallback failed" + ] } ] }