diff --git a/AGENTS.md b/AGENTS.md index cefaa21..94efa77 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,16 +23,16 @@ An executing harness stays inside its package. Building, testing, or writing any ## AI/LLM Evaluation Automation Prohibition -Repository scripts, CI jobs, skill runners, graders, optimizers, and custom executor hooks must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. +Repository-owned preparation, validation, CI jobs, hooks, deterministic tests, package generation, automatic completion gates, and automatic agent fan-out must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as repository test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. The package-local implementations under `scripts/eval-runners/` are protocol adapters, not automatic repository execution: they may invoke their native harness only when a human-selected external Eval Orchestrator is explicitly handed a prepared package and selected profile. -- Do not create, restore, recommend, or run generic automation that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. +- Do not create, restore, recommend, or run generic automation or automatic fan-out that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. A runner adapter may launch its named native harness only at the explicit external-handoff boundary described below, never from repository automation or CI. - A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. - Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. - Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. - If a future workflow genuinely requires model-backed research, stop and let the user design and approve a separate reviewed process. Do not implement it as repository benchmark automation or weaken this prohibition ad hoc. -This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking. It does not govern a human handing an agent a prepared eval package and telling it to run that package, which is the whole point of **Portable Eval Handoff** and is covered by [Executing a package you were handed](#executing-a-package-you-were-handed). +This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking. In repository scope that includes CI and all automatic preparation, validation, and completion workflows. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for the exact prepared package it was handed. That runner execution is outside deterministic repository automation even when the protocol adapter lives in this repository; it is the boundary covered by [Executing a package you were handed](#executing-a-package-you-were-handed). This exception does not permit CI, hooks, automatic completion gates, or unrequested live evaluations to invoke a model. This rule is Priority 1. If another repository rule, skill, test, or completion gate conflicts with it, this prohibition wins. @@ -40,27 +40,29 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The repository agent does not execute the prepared prompts. The user picks the harness, provider, and model, then hands `RUN-THIS.prompt.md` to that external evaluator. The external evaluator runs both configurations, grades the completed results, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished first-party `report.html` plus the exact upstream `skill-creator-report.html` in the same run. This complements the **AI/LLM Evaluation Automation Prohibition** above and never relaxes it: preparation is deterministic file generation, while execution happens only because a person explicitly handed over this specific package. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It preflights and invokes one fresh runner process per arm, bridges raw `execution-result.json` evidence into the existing result shape, grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval `eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. -Run the script immediately when asked. Do not reply with a plan, a menu of options, or a question about which harness or model the user wants; the harness and model are chosen after the package exists, by the user, outside this repository. +Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. Cline and OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot -Model claude-haiku-4.5 ``` `eval` with no skill named, or `eval changed`, means the whole changed set: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` +Use `-CodebeltReference` only when the script should perform the dynamic Copilot catalog check itself and fail if `claude-haiku-4.5` is no longer present. For noninteractive direct script use, omitting both `-Runner/-Model` and `-CodebeltReference` is an error whenever a package would be generated. + ### Handing the package over -Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected agent the evaluator, grader, and report producer. That agent creates a separate isolated worker for every `with_skill` and `without_skill` run, gives each worker only its prompt and required inputs, records the results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. +Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner, preflights it, invokes it once for every blind `with_skill` and `without_skill` arm, records normalized results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. @@ -70,17 +72,17 @@ The normal path ends in the external evaluator: after all workers finish, it rea ### Prepare, do not execute -Generate the package with the repository script rather than by hand: +Generate the package with the repository script rather than by hand after resolving the Harness + Model choice: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two hermetic run directories a worker actually sees: +It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two isolated run directories a worker actually sees: - `eval-metadata.json` — eval id and name, original prompt, expected output, assertions, required fixtures, fixture and skill hashes, and the assumptions needed to reproduce the run. This is the grading key and lives outside every run directory. - `results/` — one prefilled result stub per configuration, also outside the run directories. -- `with_skill/` — a hermetic run directory that is the worker's sandbox root. It holds `prompt.md` (the task with the effective skill instructions inlined, plus the same input context and response contract as the baseline), `run.json` (a harness-neutral contract naming only paths inside the run directory), `repo/` (the fixtures materialized as real files, which is the worker's working directory), an isolated empty `home/`, and `skill//` (the exact candidate skill revision, so nothing falls back to a globally installed copy). +- `with_skill/` — an isolated run directory that is the worker's staged root. It holds `prompt.md` (the task with the effective skill instructions inlined, plus the same input context and response contract as the baseline), `run.json` (a harness-neutral contract naming only paths inside the run directory), `repo/` (the fixtures materialized as real files, which is the worker's working directory), an isolated empty `home/`, and `skill//` (the exact candidate skill revision, so nothing falls back to a globally installed copy). - `without_skill/` — the same run directory without any `skill/` directory and with no skill instructions or mention of the skill under test. Its `repo/` is byte-identical to the with_skill one. At the iteration root it also writes `manifest.json` and `RUN-THIS.prompt.md`, the single prompt that hands the whole package to an agent of the user's choosing. The one-file path requires a harness that can create isolated workers or sessions, each launched from its run directory with `repo/` as the working directory and `home/` as an isolated profile. A plain single-context client runs one prompt file directly per fresh session instead. @@ -96,7 +98,7 @@ Adding or modifying any repo-managed skill triggers this workflow. It is not som After the final skill edit is in place, run: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ``` It resolves every repo-managed skill this branch changed, uncommitted work included, and prepares a package for each. With no skill changed it says so and exits clean, which satisfies the gate. @@ -134,13 +136,13 @@ Four things still hold while you execute: An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. -The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, and the prompt files needed to dispatch work, but it must not execute an eval prompt itself. In phase one, for every case it creates one new isolated worker for `with_skill` and another for `without_skill`, launching each from its own run directory with `repo/` as the working directory, `home/` as an isolated profile, and filesystem access confined to the run directory. It sends each worker only the matching `prompt.md` and the files already staged in that run directory. Workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after collection, the executor reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and invokes `execute` exactly once for every case's fresh `with_skill` and `without_skill` arm. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs may execute concurrently when the selected harness and the user's token budget allow it, but every run still gets a distinct context and no shared mutable workspace. -`RUN-THIS.prompt.md` requires a harness that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. Partial packages still grade and report what exists; missing arms remain visibly missing. +`RUN-THIS.prompt.md` requires a selected Eval Runner that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. A selected runner that cannot satisfy a required guarantee is `incompatible`; there is no generic fallback or runner substitution. Partial packages still grade and report what exists; missing arms remain visibly missing. An `output` is the model's own message in full, including questions, caveats, explanations, or a refusal. Where a run invoked a tool, that tool's stdout is evidence rather than a replacement for the response. Record the full worker transcript, duration, token usage, and tool-call count when the harness exposes them; omit unavailable metrics rather than estimating them. @@ -154,7 +156,7 @@ A meaningful A/B result requires both configurations to run on the same model, t ### Result handoff -An externally produced result comes back identified by eval id, configuration (`with_skill` or `without_skill`), model and provider, and the produced output. It may also carry the transcript, duration, total tokens, tool-call count, output files, and notes. The user can hand it over as filled-in `results/*.result.json` files, or state it in chat and let the agent fill them in. +An externally produced result comes back identified by eval id, configuration (`with_skill` or `without_skill`), runner-native model, harness, and the produced output. It may also carry the transcript, duration, total tokens, tool-call count, output files, and notes. The user can hand it over as filled-in `results/*.result.json` files, or state it in chat and let the agent fill them in. Which artifact transfer happens depends on where the harness ran, and `RUN-THIS.prompt.md` tells it to close either way. A harness sharing a disk with the package writes the result files, grading, `benchmark.json`, `benchmark.md`, the first-party `report.html`, and the exact upstream `skill-creator-report.html` itself and reports the first-party report path. A harness that does not - a different product, a browser, or a sandbox - ends with one paste-ready block carrying the package path and every completed result object, including grading, plus the reports as file artifacts when supported. A repository session can use `-CollectResults` only as a fallback for transferred results that lack the report artifacts. "Bring the results back" means those artifacts, never a prose recap of how the runs went. @@ -185,7 +187,7 @@ Every repo-managed skill must include its own `evals/evals.json` file at `skills - To compare a skill against a baseline, prepare a package with **Portable Eval Handoff** and hand `RUN-THIS.prompt.md` to the user; the repository agent never runs the prompts, while the user-directed external executor runs, grades, and reports the paired comparison - Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them -If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. +If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` before considering the work complete. Use `-CodebeltReference` instead only after its dynamic Copilot model check passes. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. ## Git Identity @@ -315,7 +317,7 @@ Before any completion message, reread the skill instructions and the current con For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. -Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). +Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` (or `-CodebeltReference` after dynamic availability verification) runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). ## User Input UX @@ -353,126 +355,106 @@ Interim progress updates should describe user-relevant progress, evidence, block - Mention tool/runtime failures only when they block progress, require approval, or change the planned validation - Prefer concise phrasing such as "The first read attempt failed before returning file content; I'm retrying and will report only if that changes the result" -## Anthropic Skill Authoring Reference +## Skill Authoring -Essential conventions from [The Complete Guide to Building Skills for Claude](https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf) (Anthropic, Jan 2026). All skills in this repo must follow these rules. +Skills MUST follow the Agent Skills specification and remain compatible with Anthropic's skill guidance. Repository conventions below may intentionally be stricter than the specification. -### File Structure +### Structure -``` +```text skill-name/ -├── SKILL.md # Required — exact spelling, case-sensitive -├── scripts/ # Optional — executable code (Python, Bash, etc.) -├── references/ # Optional — documentation loaded as needed -└── assets/ # Optional — templates, fonts, icons used in output +├── SKILL.md # Required +├── scripts/ # Optional executable automation +├── references/ # Optional supporting documentation +└── assets/ # Optional templates and output resources ``` -- **No `README.md`** inside the skill folder — all documentation goes in `SKILL.md` or `references/` -- Folder name must be **kebab-case** (no spaces, no underscores, no capitals) -- Folder name must match the `name:` field in YAML frontmatter +- `SKILL.md` MUST use that exact case-sensitive name. +- Skill folders MUST use kebab-case. +- Frontmatter `name` MUST match the folder name. +- Do NOT place `README.md` inside a skill folder. Put skill documentation in `SKILL.md` or `references/`. +- Keep `SKILL.md` focused. Move detailed or rarely needed material to `references/`. +- Keep `SKILL.md` below 5,000 words. -### Progressive Disclosure (Three Levels) +### Progressive Disclosure -| Level | When loaded | Token cost | Content | -|-------|------------|------------|---------| -| **Level 1: Metadata** | Always (at startup) | ~100 tokens | `name` and `description` from YAML frontmatter | -| **Level 2: Instructions** | When skill is triggered | Under 5k tokens | SKILL.md body — workflows, steps, guidance | -| **Level 3: Resources** | As needed | Effectively unlimited | Linked files: scripts, references, assets, FORMS.md | +Design every skill around three levels of context: -Keep SKILL.md under **500 lines / 5,000 words**. Move detailed content to `references/`. Keep references **one level deep** from SKILL.md — nested references cause partial reads. +1. **Metadata:** `name` and `description` are available before activation. +2. **Instructions:** the `SKILL.md` body is loaded when the skill is selected. +3. **Resources:** scripts, references, and assets are loaded or used only when needed. -### YAML Frontmatter +Minimize content at earlier levels. Do not put instructions into metadata merely to advertise skill capabilities. -Required fields: +### Frontmatter ```yaml --- -name: kebab-case-name # max 64 chars, lowercase + numbers + hyphens only -description: > # max 1024 chars, must include WHAT + WHEN + triggers - What it does. Use when user asks to [specific phrases]. +name: skill-name +description: Use when ... --- ``` -Optional fields: - -```yaml -license: MIT # for open-source skills -compatibility: > # max 500 chars — environment requirements - Requires network access and Python 3.10+ -metadata: # custom key-value pairs - author: Company Name - version: 1.0.0 - mcp-server: server-name -``` +Required: -**Forbidden**: XML angle brackets (`< >`), names containing "claude" or "anthropic" (reserved). +- `name`: 1-64 characters, lowercase alphanumeric characters and hyphens only; MUST match the skill directory. +- `description`: 1-1024 characters and MUST communicate both what the skill is for and when it should activate. -### Description Field — The Most Important Part +Optional fields such as `license`, `compatibility`, `metadata`, and supported tool restrictions MAY be used when they provide meaningful runtime or distribution information. -Structure: `[What it does] + [When to use it] + [Key capabilities]` +### Description Is a Trigger -```yaml -# ✅ Good — specific, actionable, includes triggers -description: > - Manages Linear project workflows including sprint planning, - task creation, and status tracking. Use when user mentions - "sprint", "Linear tasks", "project planning", or asks to - "create tickets". - -# ❌ Bad — too vague, no triggers -description: Helps with projects. -``` +Treat `description` as **activation metadata, not documentation**. -- Include trigger phrases users would actually say -- Mention file types if relevant -- Add negative triggers to prevent over-triggering: `Do NOT use for simple data exploration` +- SHOULD begin with trigger-oriented language such as `Use when...`. +- SHOULD describe **user intent**, not the skill's implementation. +- SHOULD stay at or below a **300-character soft ceiling**. +- MAY be shorter than 150 characters when that is sufficient. Never pad a description to meet a minimum length. +- MAY exceed 300 characters only when additional wording materially improves trigger precision or recall. +- MUST remain within the 1024-character specification limit. +- SHOULD include distinctive tasks, artifacts, technologies, file types, or domain terms that help discriminate the skill from others. +- SHOULD cover natural paraphrases conceptually rather than stuffing exact trigger phrases or keywords. +- SHOULD add exclusions only when needed to prevent realistic near-miss or overlapping skills from triggering. +- MUST NOT summarize workflows, scripts, implementation details, references, rationale, or every capability of the skill. +- MUST NOT broaden the description merely to advertise functionality. -### Writing Instructions +Prefer: -- Be **specific and actionable** — `Run scripts/validate.py --input {filename}` not `Validate the data` -- Include **error handling** — common errors, causes, and solutions -- Use **feedback loops** — run validator → fix errors → repeat -- Put **critical instructions at the top** — use `## Critical` or `## Important` headers -- For critical validations, **use scripts over language instructions** — code is deterministic -- Prefer **dynamic defaults over hardcoded values** when the source data is available from the repo, environment, or an official machine-readable feed +```yaml +description: Use when creating or refactoring .NET tests that require deterministic remote or containerized execution across supported test harnesses. +``` -### Skill Categories +Avoid: -| Category | Purpose | Example | -|----------|---------|---------| -| **Document & Asset Creation** | Consistent, high-quality output (docs, code, designs) | `frontend-design`, `docx`, `xlsx` | -| **Workflow Automation** | Multi-step processes with validation gates | `skill-creator`, scaffolding skills | -| **MCP Enhancement** | Workflow guidance layered on top of MCP tool access | `sentry-code-review` | +```yaml +description: Provides comprehensive guidance, scripts, configuration options, troubleshooting procedures, and best practices for running .NET tests remotely using containers and multiple supported test harnesses. +``` -### Common Patterns +Optimize for **trigger precision and recall per character**, not descriptive completeness. -1. **Sequential workflow** — explicit step ordering with dependencies and rollback -2. **Multi-MCP coordination** — phase separation, data passing between services -3. **Iterative refinement** — draft → validate → fix → repeat until quality threshold -4. **Context-aware selection** — decision trees for choosing the right tool/approach -5. **Domain-specific intelligence** — compliance checks, governance, audit trails +### Instructions -### Testing Checklist +Inside `SKILL.md`: -Before shipping a skill, verify: +- Make instructions specific, actionable, and ordered where sequencing matters. +- Put critical constraints near the top. +- Include error handling where failures are predictable. +- Use validation and refinement loops where output quality benefits from iteration. +- Prefer deterministic scripts for repeatable or critical validation rather than lengthy natural-language procedures. +- Prefer values discoverable from the repository, environment, or authoritative machine-readable sources over hardcoded defaults. +- Keep detailed reference material out of the main instruction path. -- [ ] Triggers on obvious tasks -- [ ] Triggers on paraphrased requests -- [ ] Does **not** trigger on unrelated topics -- [ ] Functional tests pass (correct outputs, error handling, edge cases) -- [ ] Performance improves over baseline (fewer messages, fewer errors, fewer tokens) +### Validation -Debug triggering: ask Claude `"When would you use the [skill name] skill?"` — it will quote the description back. +Before shipping or materially changing a skill, verify that it: -### Troubleshooting Quick Reference +- triggers for obvious relevant requests; +- triggers for realistic paraphrases and implicit intent; +- does not trigger for realistic near-miss requests; +- behaves correctly after activation; +- improves the intended outcome compared with not using the skill. -| Symptom | Likely cause | Fix | -|---------|-------------|-----| -| Skill won't upload | `SKILL.md` misspelled or YAML invalid | Exact case `SKILL.md`, check `---` delimiters | -| Skill never triggers | Description too vague | Add trigger phrases, mention file types | -| Skill triggers too often | Description too broad | Add negative triggers, narrow scope | -| Instructions not followed | Too verbose or ambiguous | Shorten, use bullets, move detail to `references/` | -| Slow / degraded responses | Too much content loaded | Keep SKILL.md under 5k words, use progressive disclosure | +When optimizing a description, test both **should-trigger** and **should-not-trigger** cases. Prefer measured trigger behavior over arbitrary description length, and avoid tailoring descriptions to individual evaluation phrases. ## Karpathy Rules diff --git a/CHANGELOG.md b/CHANGELOG.md index bb96609..1a7da4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,32 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.9.1] - 2026-08-22 + +This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, Cline, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. + +### Added + +- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, GitHub Copilot CLI adapter, Cline adapter, OpenCode adapter, runner resolution, artifact/hash validation, and bridge into the existing `eval-result/2` result shape, +- GitHub Copilot CLI as a supported Eval Runner with authentication handling (GitHub tokens, OS keychain, CLI fallback), JSONL-based event output parsing, stdin-based prompt delivery for byte fidelity, repository instruction visibility, and conformance tests covering token management and authentication source detection, +- deterministic fake-runner conformance coverage for fresh paired sessions, prompt fidelity, isolation boundaries, candidate-skill exposure, status normalization, unavailable telemetry, event warnings, artifact references, and report compatibility, +- runner-aware package preparation that reuses `run.json`, keeps runner selection outside `evals/evals.json`, and preserves Anthropic-compatible benchmark/report artifacts. + +### Changed + +- Eval preparation now resolves Harness + Model before writing `RUN-THIS.prompt.md`, removes the redundant portable `provider` field from `execution-profile.json` and result reporting, treats model selectors as runner-native opaque strings, and adds `scripts/Get-HarnessModels.ps1` for current model discovery with Codebelt Reference verification and free-only Cline/OpenCode filtering, +- `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` now distinguish the Eval Runner, Eval Orchestrator, Grader, and Human Reviewer and clarify that runner execution is an explicit external-handoff boundary rather than repository automation, +- All 21 repo-managed skill descriptions refactored to lean, trigger-oriented activation metadata following progressive-disclosure principles and specification compliance, +- `AGENTS.md` Skill Authoring section restructured for clarity, brevity, and progressive disclosure of form fields, asset handling, and dynamic defaults, +- report timing output omits unavailable duration and token telemetry instead of writing zero placeholders, +- Cline and GitHub Copilot runners added to the eval runner lineup alongside existing Codex and OpenCode support, +- Runner conformance tests enhanced with additional event fixtures and isolation capability assessment. + +### Fixed + +- Codex runner now ensures the evidence directory is created before writing output files, preventing file-not-found errors, +- Test runner conformance validation now requires the output parent directory to exist upfront with explicit error reporting when the directory structure is misconfigured. + ## [0.9.0] - 2026-08-20 This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets` — replaces the repository's model-backed eval benchmark workflow with deterministic, local-only validation, and finalizes the portable eval handoff. The selected external evaluator now runs the paired workers, grades their results, and invokes Anthropic's skill-creator aggregator and eval viewer without sending the user back for a second collection command. `dotnet-test` bootstraps and modernizes xUnit test projects against Codebelt conventions with role-aware fixtures; `dotnet-remote-testing` runs .NET tests inside official Microsoft SDK containers using either an existing `testenvironments.json` or zero-config, offline-safe release discovery; and `dotnet-segregated-assets` migrates ASP.NET Core applications to an artifact-first topology where `wwwroot` stays the authoring root while deployed static content is served by a separate hardened origin. Alongside those, `git-keep-a-changelog` and `git-nuget-release-notes` gained deterministic release-entity classification, and `git-visual-commits` gained an invocation routing lock. No published skill was removed or renamed, so adopting this release is non-breaking for existing installs. @@ -612,6 +638,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 [0.8.1]: https://github.com/codebeltnet/agentic/compare/v0.8.0...v0.8.1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 901ed93..3d1dc3d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -89,28 +89,28 @@ Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and c Evals are prepared, not executed, from this repository. Adding or modifying a repo-managed skill requires preparing the packages for every skill the branch touched, which is a completion gate rather than an optional extra: ```console -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, which stays last. For a single skill on demand, use: ```console -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two hermetic run directories, `with_skill/` and `without_skill/`. A run directory is the worker's sandbox root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the agent of your choice. That agent starts immediately, creates one isolated worker for every run, launches it from its run directory with `repo/` as the working directory and `home/` as an isolated profile, gives each worker only its `prompt.md` and staged files, writes the results back, grades after collection using the packaged grader guidance, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; Cline and OpenCode are free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, invokes its common `describe`/`preflight`/`execute` surface once per fresh blind arm, stores raw `execution-result.json`, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. -Repository scripts, CI jobs, and the agent that prepares a package never run those prompts. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A user-selected harness handed a specific package is the executor, not the preparer; its current context orchestrates fresh workers while the workers run the prompt files. +Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. Run both configurations on the same model, same version, and same configuration. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. -Record each external result in the matching `results/*.result.json`: `model`, `provider`, `harness`, and the complete `output`; include `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, and `tool_calls` when the harness exposes them, and the `isolation` flags the harness confirmed. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. The normal external evaluator writes `grading[].passed` and evidence, then generates the report before handing the package back. If the results were transferred without those report artifacts, validate and compare with: +Record each external runner result in the matching `execution-result.json` first. It must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: ```console pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external evaluator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. +That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and Cline are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. @@ -147,7 +147,7 @@ pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD - [ ] At least one eval in `evals/evals.json` - [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name - [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder -- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` was run after the last skill edit, and the prepared prompt paths were reported +- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` or `-CodebeltReference` was run after the last skill edit, and the prepared prompt paths were reported - [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only when transferred results need the repository-side fallback - [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior - [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too diff --git a/README.md b/README.md index dabf8f4..7ee721e 100644 --- a/README.md +++ b/README.md @@ -12,17 +12,19 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. -Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository scripts, CI jobs, runners, graders, optimizers, and custom hooks from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. Model-backed candidate/baseline fan-out is not a completion gate. +Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` workflow and replaces only its execution transport. The repository prepares the paired candidate and baseline inputs as a portable package and stops. The agent chosen by the user later executes the package, grades the completed results with the packaged grader guidance, and invokes the packaged Anthropic aggregator and eval viewer. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The user-facing preparation flow asks for Harness + Model before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, OpenCode, and Cline, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and its availability is verified against the current Copilot model catalog before automatic selection. Cline and OpenCode discovery lists only currently free models; GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: ```powershell -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` -That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. +That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved; direct noninteractive use without `-Runner/-Model` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness. Use `-CodebeltReference` only when you want the script to verify the current Copilot catalog and select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two hermetic run directories. `with_skill/` is a self-contained sandbox root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the evaluator, grader, and report producer: it creates one fresh isolated worker per run, launches it from the run directory with `repo/` as the working directory and `home/` as the profile, records the complete response plus transcript, duration, token usage, optional turns/token buckets/cost, tool calls, and isolation guarantees, grades after collection with the packaged `skill-creator` guidance, then invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. The package guarantees identical repositories, skill-only-in-with_skill staging, and an isolated home; the harness must supply the runtime sandbox that keeps global skills, global config, and the source repository out of reach. A harness that can create isolated workers handles the complete run from that one file. `-CollectResults ` remains a fallback for transferred results without report artifacts; it validates the arms and invokes the same packaged tools. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports. Packages land in gitignored `.bot/` storage by default and are not committed. + +The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. @@ -121,6 +123,8 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith ## Available Skills +Each `SKILL.md` description is lean activation metadata. The catalog below explains what happens after a skill is selected. + | Skill | Description | |-------|-------------| | [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping unless narrowed, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | diff --git a/scripts/Get-HarnessModels.ps1 b/scripts/Get-HarnessModels.ps1 new file mode 100644 index 0000000..93a5f7a --- /dev/null +++ b/scripts/Get-HarnessModels.ps1 @@ -0,0 +1,498 @@ +<# +.SYNOPSIS + Lists current model selectors for a supported eval harness. + +.DESCRIPTION + Discovers runner-native model selectors without executing model requests. GitHub Copilot and Codex return every + model the harness exposes. Cline and OpenCode return only models whose current catalog metadata proves free + availability. Discovery failures are local to the selected harness and never fall back to stale hardcoded catalogs. + +.PARAMETER Runner + Internal Eval Runner id: github-copilot, codex, opencode, or cline. + +.PARAMETER CatalogPath + Optional deterministic catalog fixture used by tests. When supplied, no harness command is invoked. + +.PARAMETER RequireModel + Optional model selector that must exist in the discovered list. + +.PARAMETER Refresh + For harnesses that support explicit catalog refresh, request a refresh before listing. This never executes a model + prompt. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [ValidateSet('github-copilot', 'codex', 'opencode', 'cline')] + [string]$Runner, + + [string]$CatalogPath, + + [string]$RequireModel, + + [switch]$Refresh +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerCommon = Join-Path $PSScriptRoot 'eval-runners/runner-common.ps1' +. $runnerCommon + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Get-HarnessDisplayName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + switch ($RunnerName) { + 'github-copilot' { return 'GitHub Copilot CLI' } + 'codex' { return 'Codex CLI' } + 'opencode' { return 'OpenCode' } + 'cline' { return 'Cline' } + default { return $RunnerName } + } +} + +function Get-PolicyName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + if ($RunnerName -in @('cline', 'opencode')) { + return 'free' + } + return 'all' +} + +function Read-CatalogJson { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Catalog fixture '$Path' does not exist." + } + return [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, $utf8NoBom) | ConvertFrom-Json +} + +function Get-FirstPropertyValue { + param( + [object]$Object, + [string[]]$Names + ) + + foreach ($name in $Names) { + $value = Get-JsonProperty -Object $Object -Name $name -Default $null + if ($null -ne $value -and -not [string]::IsNullOrWhiteSpace([string]$value)) { + return [string]$value + } + } + return $null +} + +function Get-NumericPropertyValues { + param([object]$Object) + + $values = [System.Collections.Generic.List[double]]::new() + if ($null -eq $Object) { + return @() + } + + if ($Object -is [System.Collections.IEnumerable] -and -not ($Object -is [string]) -and -not ($Object -is [System.Collections.IDictionary]) -and -not ($Object -is [pscustomobject])) { + foreach ($item in $Object) { + foreach ($child in @(Get-NumericPropertyValues -Object $item)) { $values.Add([double]$child) } + } + return @($values) + } + + if ($Object -is [System.Collections.IDictionary]) { + $propertyNames = @($Object.Keys) + foreach ($name in $propertyNames) { + $value = $Object[$name] + if ($value -is [int] -or $value -is [long] -or $value -is [double] -or $value -is [decimal]) { + $values.Add([double]$value) + } elseif ($null -ne $value -and -not ($value -is [string]) -and -not ($value -is [System.ValueType]) -and ($value -is [System.Collections.IDictionary] -or $value -is [pscustomobject] -or $value -is [System.Collections.IEnumerable])) { + foreach ($child in @(Get-NumericPropertyValues -Object $value)) { $values.Add([double]$child) } + } + } + return @($values) + } + + foreach ($property in @($Object.PSObject.Properties)) { + $value = $property.Value + if ($value -is [int] -or $value -is [long] -or $value -is [double] -or $value -is [decimal]) { + $values.Add([double]$value) + } elseif ($null -ne $value -and -not ($value -is [string]) -and -not ($value -is [System.ValueType]) -and ($value -is [System.Collections.IDictionary] -or $value -is [pscustomobject] -or $value -is [System.Collections.IEnumerable])) { + foreach ($child in @(Get-NumericPropertyValues -Object $value)) { $values.Add([double]$child) } + } + } + return @($values) +} + +function Get-ModelAvailability { + param([object]$Model) + + $explicit = Get-FirstPropertyValue -Object $Model -Names @('availability', 'billing', 'usageCostDisplay') + if (-not [string]::IsNullOrWhiteSpace($explicit)) { + $normalized = $explicit.ToLowerInvariant() + if ($normalized -eq 'free') { return 'free' } + if ($normalized -in @('paid', 'subscription', 'metered')) { return 'paid' } + } + + foreach ($name in @('free', 'isFree')) { + $value = Get-JsonProperty -Object $Model -Name $name -Default $null + if ($null -eq $value) { + continue + } + if ([bool]$value) { return 'free' } + return 'paid' + } + + foreach ($propertyName in @('pricing', 'cost')) { + $price = Get-JsonProperty -Object $Model -Name $propertyName -Default $null + $numbers = @(Get-NumericPropertyValues -Object $price) + if ($numbers.Count -gt 0) { + if (@($numbers | Where-Object { [double]$_ -gt 0 }).Count -gt 0) { + return 'paid' + } + return 'free' + } + } + + return 'unknown' +} + +function Test-TextModel { + param([object]$Model) + + $operation = [string](Get-JsonProperty -Object $Model -Name 'operation' -Default '') + if (-not [string]::IsNullOrWhiteSpace($operation) -and $operation -notin @('language', 'chat', 'completion')) { + return $false + } + return $true +} + +function ConvertTo-ModelChoice { + param( + [Parameter(Mandatory = $true)][object]$Model, + [Parameter(Mandatory = $true)][string]$RunnerName, + [string]$Source, + [string]$ExplicitSelector + ) + + if (-not (Test-TextModel -Model $Model)) { + return $null + } + + $id = if ([string]::IsNullOrWhiteSpace($ExplicitSelector)) { + Get-FirstPropertyValue -Object $Model -Names @('id', 'slug', 'model', 'modelId', 'model_id') + } else { + $ExplicitSelector + } + if ([string]::IsNullOrWhiteSpace($id)) { + return $null + } + + $provider = Get-FirstPropertyValue -Object $Model -Names @('providerID', 'providerId', 'provider') + if ($RunnerName -in @('cline', 'opencode') -and $id -notmatch '/' -and -not [string]::IsNullOrWhiteSpace($provider)) { + $id = "$provider/$id" + } + + $displayName = Get-FirstPropertyValue -Object $Model -Names @('display_name', 'displayName', 'name') + if ([string]::IsNullOrWhiteSpace($displayName)) { + $displayName = $id + } + + return [pscustomobject][ordered]@{ + id = $id + display_name = $displayName + availability = Get-ModelAvailability -Model $Model + source = $Source + } +} + +function ConvertTo-ModelChoices { + param( + [Parameter(Mandatory = $true)][object]$Catalog, + [Parameter(Mandatory = $true)][string]$RunnerName, + [Parameter(Mandatory = $true)][string]$Source + ) + + $rawModels = [System.Collections.Generic.List[object]]::new() + if ($Catalog -is [array]) { + foreach ($model in $Catalog) { $rawModels.Add($model) } + } elseif (Test-JsonProperty -Object $Catalog -Name 'models') { + foreach ($model in @($Catalog.models)) { $rawModels.Add($model) } + } else { + foreach ($propertyName in @(Get-JsonPropertyNames -Object $Catalog)) { + $value = Get-JsonProperty -Object $Catalog -Name $propertyName -Default $null + if ($null -ne $value -and @($value.PSObject.Properties).Count -gt 0) { + if (-not (Test-JsonProperty -Object $value -Name 'id')) { + $value | Add-Member -NotePropertyName id -NotePropertyValue $propertyName -Force + } + $rawModels.Add($value) + } + } + } + + $choices = [System.Collections.Generic.List[object]]::new() + foreach ($model in $rawModels) { + $choice = ConvertTo-ModelChoice -Model $model -RunnerName $RunnerName -Source $Source + if ($null -ne $choice) { + $choices.Add($choice) + } + } + $seen = @{} + $deduped = [System.Collections.Generic.List[object]]::new() + foreach ($choice in @($choices | Sort-Object id)) { + if (-not $seen.ContainsKey([string]$choice.id)) { + $seen[[string]$choice.id] = $true + $deduped.Add($choice) + } + } + return @($deduped) +} + +function Select-ModelsByPolicy { + param( + [object[]]$Models, + [Parameter(Mandatory = $true)][string]$RunnerName + ) + + $policy = Get-PolicyName -RunnerName $RunnerName + if ($policy -eq 'free') { + $freeModels = @($Models | Where-Object { [string]$_.availability -eq 'free' }) + if ($freeModels.Count -eq 0) { + throw "No free $((Get-HarnessDisplayName -RunnerName $RunnerName)) models are currently available from discovery. Choose another harness or update the harness catalog." + } + return @($freeModels) + } + + return @($Models) +} + +function Invoke-JsonCommand { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [int]$TimeoutSeconds = 60 + ) + + $work = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-model-discovery-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $work -Force | Out-Null + try { + $environment = New-RunnerProbeEnvironment + foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @($Arguments)) -WorkingDirectory $work -Environment $environment -TimeoutSeconds $TimeoutSeconds + if ($process.TimedOut) { + throw "Command '$($CommandInfo.Source)' timed out during model discovery." + } + if ($process.ExitCode -ne 0) { + $detail = [string]::Join("`n", @($process.Stdout, $process.Stderr)).Trim() + throw "Command '$($CommandInfo.Source) $($Arguments -join ' ')' failed during model discovery with exit code $($process.ExitCode). $detail" + } + return [pscustomobject]@{ Stdout = $process.Stdout; Stderr = $process.Stderr } + } finally { + if (Test-Path -LiteralPath $work) { + Remove-Item -LiteralPath $work -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function ConvertFrom-OpenCodeTextCatalog { + param([Parameter(Mandatory = $true)][string]$Text) + + $models = [System.Collections.Generic.List[object]]::new() + $lines = $Text -split "`r?`n" + $currentSelector = $null + $buffer = [System.Collections.Generic.List[string]]::new() + $depth = 0 + foreach ($line in $lines) { + $trimmed = $line.Trim() + if ([string]::IsNullOrWhiteSpace($trimmed)) { + continue + } + if ($depth -eq 0 -and $trimmed -match '^[^\s/]+/.+$') { + $currentSelector = $trimmed + continue + } + if ($trimmed.StartsWith('{') -or $depth -gt 0) { + $buffer.Add($line) + $depth += ([regex]::Matches($line, '\{')).Count + $depth -= ([regex]::Matches($line, '\}')).Count + if ($depth -le 0 -and $buffer.Count -gt 0) { + $json = [string]::Join("`n", @($buffer)) + $object = $json | ConvertFrom-Json + $choice = ConvertTo-ModelChoice -Model $object -RunnerName 'opencode' -Source 'opencode models --verbose' -ExplicitSelector $currentSelector + if ($null -ne $choice) { $models.Add($choice) } + $buffer.Clear() + $depth = 0 + $currentSelector = $null + } + } + } + return @($models) +} + +function Resolve-ClineModelsModulePath { + $command = Resolve-ExternalCommand -Name 'cline' + if ($null -eq $command) { + throw 'Cline CLI executable is not available on PATH.' + } + + $source = [string]$command.Source + $directory = Split-Path -Parent $source + $candidates = @( + (Join-Path $directory 'node_modules/cline/node_modules/@cline/llms/dist/models.js'), + (Join-Path $directory '../lib/node_modules/cline/node_modules/@cline/llms/dist/models.js'), + (Join-Path $directory '../node_modules/cline/node_modules/@cline/llms/dist/models.js') + ) + foreach ($candidate in $candidates) { + $full = [System.IO.Path]::GetFullPath($candidate) + if (Test-Path -LiteralPath $full -PathType Leaf) { + return $full + } + } + + throw 'Cline model registry module was not found under the installed CLI package.' +} + +function Resolve-CopilotSdkPath { + $command = Resolve-ExternalCommand -Name 'copilot' + if ($null -eq $command) { + throw 'GitHub Copilot CLI executable is not available on PATH.' + } + + $source = [string]$command.Source + $directory = Split-Path -Parent $source + $candidates = [System.Collections.Generic.List[string]]::new() + $optionalRoot = Join-Path $directory 'node_modules/@github/copilot/node_modules/@github' + if (Test-Path -LiteralPath $optionalRoot -PathType Container) { + foreach ($package in @(Get-ChildItem -LiteralPath $optionalRoot -Directory -Filter 'copilot-*' -Force)) { + $candidates.Add((Join-Path $package.FullName 'sdk/index.js')) + } + } + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-win32-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-win32-arm64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-linux-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-linux-arm64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-darwin-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-darwin-arm64/sdk/index.js')) + foreach ($candidate in $candidates) { + $full = [System.IO.Path]::GetFullPath($candidate) + if (Test-Path -LiteralPath $full -PathType Leaf) { + return $full + } + } + + throw 'GitHub Copilot CLI does not expose a package-local SDK model listing surface in this installation.' +} + +function Get-CodexModels { + $command = Resolve-ExternalCommand -Name 'codex' + if ($null -eq $command) { + throw 'Codex CLI executable is not available on PATH.' + } + $result = Invoke-JsonCommand -CommandInfo $command -Arguments @('debug', 'models') -TimeoutSeconds 60 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'codex' -Source 'codex debug models' +} + +function Get-OpenCodeModels { + $command = Resolve-ExternalCommand -Name 'opencode' + if ($null -eq $command) { + throw 'OpenCode CLI executable is not available on PATH.' + } + $arguments = @('models', 'opencode', '--verbose') + if ($Refresh) { $arguments += '--refresh' } + $result = Invoke-JsonCommand -CommandInfo $command -Arguments $arguments -TimeoutSeconds 180 + return ConvertFrom-OpenCodeTextCatalog -Text $result.Stdout +} + +function Get-ClineModels { + $modulePath = Resolve-ClineModelsModulePath + $node = Resolve-ExternalCommand -Name 'node' + if ($null -eq $node) { + throw 'Node.js is required to read the Cline model registry.' + } + + $script = @' +import { pathToFileURL } from "node:url"; +const modulePath = process.argv[1]; +const mod = await import(pathToFileURL(modulePath).href); +const providerIds = await mod.getProviderIds(); +const models = []; +for (const providerId of providerIds) { + const providerModels = await mod.getModelsForProvider(providerId); + for (const [key, model] of Object.entries(providerModels || {})) { + models.push({ + id: model.id || key, + providerID: providerId, + name: model.name || model.id || key, + pricing: model.pricing, + operation: model.operation, + capabilities: model.capabilities + }); + } +} +console.log(JSON.stringify({ models })); +'@ + $result = Invoke-JsonCommand -CommandInfo $node -Arguments @('--input-type=module', '-e', $script, $modulePath) -TimeoutSeconds 60 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'cline' -Source '@cline/llms model registry' +} + +function Get-CopilotModels { + $sdkPath = Resolve-CopilotSdkPath + $node = Resolve-ExternalCommand -Name 'node' + if ($null -eq $node) { + throw 'Node.js is required to read the GitHub Copilot CLI model registry.' + } + + $script = @' +import { pathToFileURL } from "node:url"; +const sdkPath = process.argv[1]; +const mod = await import(pathToFileURL(sdkPath).href); +const ids = Array.isArray(mod.HELP_VISIBLE_MODELS) + ? mod.HELP_VISIBLE_MODELS + : Object.keys(mod.SUPPORTED_MODELS || {}); +console.log(JSON.stringify({ models: ids.map((id) => ({ id, name: id, operation: "language" })) })); +'@ + $result = Invoke-JsonCommand -CommandInfo $node -Arguments @('--input-type=module', '-e', $script, $sdkPath) -TimeoutSeconds 90 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'github-copilot' -Source 'GitHub Copilot CLI help-visible model catalog' +} + +try { + $rawModels = if (-not [string]::IsNullOrWhiteSpace($CatalogPath)) { + ConvertTo-ModelChoices -Catalog (Read-CatalogJson -Path $CatalogPath) -RunnerName $Runner -Source $CatalogPath + } else { + switch ($Runner) { + 'github-copilot' { Get-CopilotModels } + 'codex' { Get-CodexModels } + 'opencode' { Get-OpenCodeModels } + 'cline' { Get-ClineModels } + } + } + $models = @(Select-ModelsByPolicy -Models @($rawModels) -RunnerName $Runner) + if ($models.Count -eq 0) { + throw "No models were returned for $((Get-HarnessDisplayName -RunnerName $Runner))." + } + + if (-not [string]::IsNullOrWhiteSpace($RequireModel) -and @($models | Where-Object { [string]$_.id -eq $RequireModel }).Count -eq 0) { + $available = [string]::Join(', ', @($models | Select-Object -ExpandProperty id)) + throw "Required model '$RequireModel' was not returned by current $((Get-HarnessDisplayName -RunnerName $Runner)) discovery. Available models: $available" + } + + [ordered]@{ + schema = 'codebeltnet/agentic/harness-models/1' + runner = $Runner + harness = Get-HarnessDisplayName -RunnerName $Runner + policy = Get-PolicyName -RunnerName $Runner + models = @($models) + } | ConvertTo-Json -Depth 20 +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-report-template.html b/scripts/eval-report-template.html index e5b6588..4881def 100644 --- a/scripts/eval-report-template.html +++ b/scripts/eval-report-template.html @@ -165,7 +165,7 @@ if (!run) return '
' + runLabel(config) + '
Run not recorded.
'; const summary = gradeSummary(run); const rate = passRate(run); - const meta = [run.model, run.provider, run.harness].filter(Boolean).join(' · ') || 'Runtime details not recorded'; + const meta = [run.model, run.harness].filter(Boolean).join(' · ') || 'Runtime details not recorded'; let html = '
' + runLabel(config) + '' + esc(meta) + '
'; html += '
'; html += metric('Turns', run.metrics.turns, config); @@ -303,7 +303,7 @@ document.getElementById('eval-select').addEventListener('change', e => { current = Number(e.target.value); renderOutputs(); }); document.addEventListener('keydown', e => { if (e.target.tagName === 'TEXTAREA') return; if (e.key === 'ArrowLeft') document.getElementById('prev').click(); if (e.key === 'ArrowRight') document.getElementById('next').click(); }); document.getElementById('skill-name').textContent = DATA.skill_name || 'skill'; - document.getElementById('run-meta').textContent = [DATA.metadata && DATA.metadata.model, DATA.metadata && DATA.metadata.provider, DATA.metadata && DATA.metadata.completed_runs + '/' + DATA.metadata.expected_runs + ' runs'].filter(Boolean).join(' · '); + document.getElementById('run-meta').textContent = [DATA.metadata && DATA.metadata.model, DATA.metadata && DATA.metadata.completed_runs + '/' + DATA.metadata.expected_runs + ' runs'].filter(Boolean).join(' · '); for (let i = 0; i < (DATA.evals || []).length; i++) { const option = document.createElement('option'); option.value = String(i); option.textContent = (i + 1) + '. ' + DATA.evals[i].name; document.getElementById('eval-select').appendChild(option); } renderOutputs(); renderBenchmark(); renderSkill(); diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md new file mode 100644 index 0000000..2559817 --- /dev/null +++ b/scripts/eval-runners/README.md @@ -0,0 +1,100 @@ +# Eval Runner protocol + +This directory contains the package-local implementation of the v0.9.1 Eval +Runner protocol. It is copied into prepared packages so the external Eval +Orchestrator can use the same runner implementation that was validated with the +package. It is not a model executor used by repository automation. + +The boundary has three documents: + +```text +run.json + execution-profile.json -> runner -> execution-result.json +``` + +`run.json` is the existing portable one-arm contract. It owns the prompt, +working directory, isolated home, staged candidate skill, and required +experimental controls. Its `filesystemIsolationRequired` and +`mustNotReadOutsideSandbox` fields describe the staged worker-facing package +boundary; they do not claim that the host has a hard OS filesystem sandbox. +`execution-profile.json` selects the runner, runner-native model selector, and +execution configuration. The model string is opaque to the portable layer: a +runner may pass it through unchanged or split it internally when its native CLI +requires separate provider/model arguments. The profile contains no credentials, +secrets, or portable provider field. +`execution-result.json` normalizes one blind execution and keeps grading +separate from raw evidence. + +Every runner exposes the same process surface: + +```text +runner.ps1 describe +runner.ps1 preflight -Run -Profile +runner.ps1 execute -Run -Profile +``` + +The commands emit one JSON document. `describe` and `preflight` do not consume +model tokens. `execute` runs exactly one arm, never resumes a session, never +grades or retries for answer quality, and returns a normalized result even for +refusals, timeouts, failures, and incompatibility. + +The package resolver selects a named child directory under this directory. It +does not guess a runner and does not fall back to an improvised worker. A +selected runner that cannot satisfy the required contract returns +`incompatible`. Hard OS-level filesystem confinement is a confidence signal, +not a universal prerequisite: a run with all mandatory experimental controls +proven reports `strict` isolation when hard confinement is proven and +`pragmatic` isolation when it is not. A missing fresh context, controlled skill +boundary, prompt fidelity, result capture, or other mandatory control remains +incompatible. + +The fake runner is deterministic and is the conformance reference. GitHub +Copilot, Codex, OpenCode, and Cline are thin harness-specific adapters. Their +native CLI flags, environment setup, event parsing, authentication injection, +and isolation checks stay inside their own directories. Windows is supported in +pragmatic mode when the native CLI satisfies the mandatory controls. + +`github-copilot` with `claude-haiku-4.5` is the Codebelt Reference evaluation +configuration: a stable, economical pairing for routine skill comparison. It is +a repository convention, not an Anthropic default, and preparation verifies that +the model still appears in the current Copilot catalog before selecting it. +Cross-runner and cross-model numbers are never blended into one score; a paired +`with_skill` versus `without_skill` comparison is only meaningful within one +identical runner, model, and configuration stratum. + +GitHub Copilot uses `copilot -C --model +--output-format json --allow-all-tools --no-ask-user --disable-builtin-mcps +--no-color --log-level none --no-auto-update +--secret-env-vars=COPILOT_GITHUB_TOKEN,GH_TOKEN,GITHUB_TOKEN` with the exact +prepared prompt bytes delivered once through stdin. It passes no `--prompt`/`-p`, +`--resume`, `--continue`, `--session-id`, or `--connect`, and it does not use +the blanket `--yolo`, `--allow-all`, `--allow-all-paths`, or `--allow-all-urls` +switches. `--allow-all-tools` is a broad tool-approval grant required for +noninteractive execution; it does not disable path or URL verification. +Repository-owned custom instructions remain enabled and are staged identically +in both paired arms. Personal Copilot configuration is excluded by run-local +`COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; +the runner does not copy the normal `.copilot` directory. Authentication prefers +explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; when none is +present, the trusted runner may resolve `gh auth token` outside the worker and +inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` +is never forwarded into the evaluated worker. `--secret-env-vars` removes every +listed token variable from shell and MCP child environments. Preflight does not +make a model request and therefore reports native keychain/service readiness as +conditional rather than claiming successful remote authentication. Codex uses +`--ask-for-approval never` with `exec --sandbox +workspace-write`; it does not combine explicit sandbox selection with +`--approve-for-me`. OpenCode uses `run --format json --auto --model +` with isolated global/config roots and preserves +repository-owned project configuration; it does not depend on +`OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline expects a +`provider/model` selector in the profile and derives its native `--provider` +and `--model` arguments inside the adapter. Each captures an exact observable +CLI version and passes only documented environment credentials when the selected +runner supports them. None copies a global skill directory, memory store, plugin +set, or normal agent profile into a run. + +Model discovery lives in `scripts/Get-HarnessModels.ps1`. It uses the current local harness catalog where available: Copilot through the installed CLI SDK help-visible model list, Codex through `codex debug models`, OpenCode through `opencode models opencode --verbose`, and Cline through the installed `@cline/llms` registry. Cline and OpenCode discovery returns only models with current metadata proving free availability; zero free models is a clear local failure, not a fallback to paid models. + +Freebuff is currently documented as planned/blocked. Its supported CLI remains +TUI-oriented and does not provide the required one-prompt, noninteractive, +machine-readable fresh-session transport, so no Freebuff runner is advertised. diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 new file mode 100644 index 0000000..8774dd3 --- /dev/null +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -0,0 +1,246 @@ +<#! +.SYNOPSIS + Bridges one normalized execution-result.json into the existing eval-result/2 shape. + +.DESCRIPTION + This bridge runs after execution and before grading. It reads no expected + output or assertions, preserves any existing grading array, validates raw + artifact provenance, and deliberately leaves unavailable telemetry null. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$Run, + + [Parameter(Mandatory = $true)] + [string]$ExecutionResult, + + [Parameter(Mandatory = $true)] + [string]$Result +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot 'runner-common.ps1') + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Write-BridgeJson { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][object]$Value + ) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + $serializable = if ($Value -is [System.Collections.IDictionary]) { + $copy = [ordered]@{} + foreach ($key in $Value.Keys) { $copy[[string]$key] = $Value[$key] } + [pscustomobject]$copy + } else { + $Value + } + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $serializable -Depth 100) + [Environment]::NewLine), $utf8NoBom) +} + +function Get-CapabilityBoolean { + param([object]$Value) + + if ([string]$Value -in @('supported', 'verified', 'true')) { return $true } + if ([string]$Value -in @('unsupported', 'excluded', 'false')) { return $false } + return $null +} + +function Get-MetricValue { + param( + [Parameter(Mandatory = $true)][object]$Result, + [Parameter(Mandatory = $true)][string]$Name + ) + + $metric = Get-JsonProperty -Object $Result.telemetry -Name $Name -Default $null + if ($null -eq $metric -or [string](Get-JsonProperty -Object $metric -Name 'status' -Default '') -ne 'available') { + return $null + } + return Get-JsonProperty -Object $metric -Name 'value' -Default $null +} + +function Get-ArtifactPath { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Artifact + ) + + $path = [string](Get-JsonProperty -Object $Artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $Artifact -Name 'scope' -Default '') + if ($scope -eq 'run') { + return Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $path -FieldName 'execution-result artifact.path' -Kind File + } + if ($scope -eq 'package') { + return Resolve-ContainedPath -BasePath $IterationDirectory -RelativePath $path -FieldName 'execution-result package artifact.path' -Kind File + } + throw "Unsupported execution-result artifact scope '$scope'." +} + +function Get-ResultRelativeArtifactPath { + param( + [Parameter(Mandatory = $true)][string]$EvalDirectory, + [Parameter(Mandatory = $true)][string]$FullPath + ) + + $relative = [System.IO.Path]::GetRelativePath($EvalDirectory, $FullPath).Replace('\', '/') + Assert-SafeRelativePath -RelativePath $relative -FieldName 'result.output_files' + return $relative +} + +function Get-ExistingGrading { + param([Parameter(Mandatory = $true)][string]$ResultPath) + + if (-not (Test-Path -LiteralPath $ResultPath -PathType Leaf)) { + return @() + } + try { + $existing = Read-RunnerJson -Path $ResultPath + return @(Get-JsonProperty -Object $existing -Name 'grading' -Default @()) + } catch { + return @() + } +} + +try { + $runData = Resolve-RunContract -RunPath $Run + $runPath = $runData.RunPath + $runDirectory = $runData.RunRoot + $evalDirectory = Split-Path -Parent $runDirectory + $iterationDirectory = Split-Path -Parent $evalDirectory + $executionPath = (Resolve-Path -LiteralPath $ExecutionResult -ErrorAction Stop).Path + $resultPath = [System.IO.Path]::GetFullPath($Result, (Get-Location).Path) + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $executionPath)) { + throw 'execution-result.json must remain inside the prepared iteration package.' + } + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $resultPath)) { + throw 'eval-result output must remain inside the prepared iteration package.' + } + + $raw = Read-RunnerJson -Path $executionPath + [void](Assert-ExecutionResult -Result $raw) + if ([string]$raw.input.prompt_sha256 -ne $runData.PromptHash) { + throw 'execution-result input.prompt_sha256 does not match prompt.md.' + } + if ([string]$raw.input.run_json_sha256 -ne (Get-Sha256HexFromFile -Path $runPath)) { + throw 'execution-result input.run_json_sha256 does not match run.json.' + } + $profilePath = Join-Path $iterationDirectory 'execution-profile.json' + if (-not (Test-Path -LiteralPath $profilePath -PathType Leaf)) { + throw 'Runner-aware package is missing execution-profile.json.' + } + $profile = Resolve-ExecutionProfile -ProfilePath $profilePath + if ([string]$raw.input.profile_sha256 -ne $profile.Hash) { + throw 'execution-result input.profile_sha256 does not match execution-profile.json.' + } + if ([int]$raw.run.eval_id -ne $runData.EvalId -or [string]$raw.run.configuration -ne $runData.Mode) { + throw 'execution-result run identity does not match run.json.' + } + + $artifactPaths = [System.Collections.Generic.List[string]]::new() + foreach ($artifact in @($raw.artifacts)) { + $full = Get-ArtifactPath -RunData $runData -IterationDirectory $iterationDirectory -Artifact $artifact + $expectedHash = [string](Get-JsonProperty -Object $artifact -Name 'sha256' -Default '') + if ((Get-Sha256HexFromFile -Path $full) -ne $expectedHash) { + throw "Artifact '$($artifact.path)' has a hash that does not match the recorded execution evidence." + } + $expectedSize = [int64](Get-JsonProperty -Object $artifact -Name 'size' -Default -1) + if ((Get-Item -LiteralPath $full).Length -ne $expectedSize) { + throw "Artifact '$($artifact.path)' has a size that does not match the recorded execution evidence." + } + $artifactPaths.Add((Get-ResultRelativeArtifactPath -EvalDirectory $evalDirectory -FullPath $full)) + } + + $finalStatus = [string]$raw.final_response.status + $output = if ($finalStatus -eq 'available') { [string]$raw.final_response.text } else { '' } + $transcript = '' + $transcriptMetricObject = Get-JsonProperty -Object $raw.telemetry -Name 'transcript' -Default $null + $transcriptMetric = Get-MetricValue -Result $raw -Name 'transcript' + if ($null -ne $transcriptMetric) { + $transcriptArtifact = [string](Get-JsonProperty -Object $transcriptMetric -Name 'artifact' -Default '') + if (-not [string]::IsNullOrWhiteSpace($transcriptArtifact)) { + $transcript = "artifact: $transcriptArtifact" + } + } + $toolCallsValue = Get-MetricValue -Result $raw -Name 'tool_calls' + $costValue = Get-MetricValue -Result $raw -Name 'cost' + $tokenValue = Get-MetricValue -Result $raw -Name 'tokens' + $evidence = Get-JsonProperty -Object $raw -Name 'evidence' -Default ([ordered]@{}) + $commands = @(Get-JsonProperty -Object $evidence -Name 'commands' -Default @()) + $files = @(Get-JsonProperty -Object $evidence -Name 'files' -Default @()) + $warnings = @((Get-JsonProperty -Object $raw -Name 'warnings' -Default @()) + (Get-JsonProperty -Object $raw -Name 'compatibility_deviations' -Default @())) + $notes = [System.Collections.Generic.List[string]]::new() + $notes.Add("execution_status=$($raw.status)") + if ($finalStatus -eq 'unavailable') { $notes.Add("final_response_unavailable=$($raw.final_response.reason)") } + foreach ($warning in $warnings) { if (-not [string]::IsNullOrWhiteSpace([string]$warning)) { $notes.Add([string]$warning) } } + + $existingGrading = @(Get-ExistingGrading -ResultPath $resultPath) + $caps = Get-JsonProperty -Object $raw.isolation -Name 'capabilities' -Default ([ordered]@{}) + $requestedModel = [string](Get-JsonProperty -Object $raw.requested -Name 'model' -Default '') + $resolvedModelValue = Get-JsonProperty -Object $raw.resolved -Name 'model' -Default $null + $resolvedModel = if ($null -eq $resolvedModelValue) { '' } else { [string]$resolvedModelValue } + $resolutionStatus = [string](Get-JsonProperty -Object $raw.resolved -Name 'status' -Default 'unavailable') + $resolutionReason = [string](Get-JsonProperty -Object $raw.resolved -Name 'reason' -Default '') + $notes.Add("configuration_resolution=$resolutionStatus") + $portableResult = [ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + skill_name = if ($runData.Mode -eq 'with_skill') { [string](Get-JsonProperty -Object $runData.Contract -Name 'skillName' -Default '') } else { '' } + iteration = [int](Get-JsonProperty -Object $runData.Contract -Name 'iteration' -Default 0) + eval_id = $runData.EvalId + eval_name = $runData.EvalName + configuration = $runData.Mode + model = if ([string]::IsNullOrWhiteSpace($resolvedModel)) { $requestedModel } else { $resolvedModel } + requested_model = $requestedModel + resolved_model = $resolvedModel + configuration_resolution_status = $resolutionStatus + configuration_resolution_reason = $resolutionReason + harness = "$(Get-JsonProperty -Object $raw.harness -Name 'name' -Default 'unknown') $(Get-JsonProperty -Object $raw.harness -Name 'version' -Default '')".Trim() + executed_utc = [string]$raw.finished_utc + output = $output + output_files = @($artifactPaths | Sort-Object -Unique) + transcript = $transcript + shell_commands = @($commands) + files_read = @(Get-JsonProperty -Object $evidence -Name 'files_read' -Default @()) + files_written = @(Get-JsonProperty -Object $evidence -Name 'files_written' -Default @()) + stdout = if (@($artifactPaths | Where-Object { $_ -match 'events\.jsonl$' }).Count -gt 0) { 'artifact: events.jsonl' } else { '' } + stderr = if (@($artifactPaths | Where-Object { $_ -match 'stderr\.txt$' }).Count -gt 0) { 'artifact: stderr.txt' } else { '' } + exit_status = Get-JsonProperty -Object $raw.exit -Name 'status' -Default $null + duration_seconds = [double]$raw.duration_seconds + total_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'total_tokens' -Default $null } else { $null } + tool_calls = $toolCallsValue + turns = Get-JsonProperty -Object $evidence -Name 'turns' -Default $null + base_input_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'input_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'input' -Default $null) } else { $null } + output_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'output_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'output' -Default $null) } else { $null } + cache_read_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'cached_input_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'cache_read' -Default $null) } else { $null } + cache_write_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'cache_write_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'cache_write' -Default $null) } else { $null } + cache_write_1h_tokens = $null + estimated_cost_usd = $costValue + model_effort = [string](Get-JsonProperty -Object $raw.resolved -Name 'reasoning_effort' -Default '') + isolation = [ordered]@{ + level = Get-JsonProperty -Object $raw.isolation -Name 'level' -Default 'unsupported' + status = Get-JsonProperty -Object $raw.isolation -Name 'status' -Default 'unverified' + hard_filesystem_confinement = Get-JsonProperty -Object $raw.isolation -Name 'hard_filesystem_confinement' -Default $false + mechanisms = @(Get-JsonProperty -Object $raw.isolation -Name 'mechanisms' -Default @()) + fresh_context = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'fresh_context' -Default $null) + isolated_home = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_home_config' -Default $null) + isolated_cwd = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_working_directory' -Default $null) + filesystem_sandbox = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'filesystem_confinement' -Default $null) + candidate_skill_exposed = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'candidate_skill_exposure' -Default $null) + transcript_captured = if ($null -ne $transcriptMetricObject) { [string](Get-JsonProperty -Object $transcriptMetricObject -Name 'status' -Default '') -eq 'available' } else { $null } + } + execution_status = [string]$raw.status + execution_run_id = [string]$raw.run_id + execution_result_file = [System.IO.Path]::GetRelativePath($evalDirectory, $executionPath).Replace('\', '/') + grading = @($existingGrading) + notes = [string]::Join("`n", @($notes)) + } + Write-BridgeJson -Path $resultPath -Value $portableResult + Write-RunnerJson -Value ([ordered]@{ schema = 'codebeltnet/agentic/eval-result-bridge/1'; result = [System.IO.Path]::GetRelativePath($iterationDirectory, $resultPath).Replace('\', '/'); execution_status = $raw.status }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/cline/runner.ps1 b/scripts/eval-runners/cline/runner.ps1 new file mode 100644 index 0000000..8fb1cdb --- /dev/null +++ b/scripts/eval-runners/cline/runner.ps1 @@ -0,0 +1,552 @@ +<#! +.SYNOPSIS + Cline Eval Runner adapter. + +.DESCRIPTION + The adapter uses Cline's supported headless JSON/NDJSON surface. It starts + one fresh process per arm, supplies the prompt on stdin, uses run-local + data/config and hooks directories, and never supplies a session id. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'cline' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'Cline CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'conditional' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + credential_child_filtering = 'conditional' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-ClineInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Invoke-ClineCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @($Arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-ClineDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'cline' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'Cline CLI'; version = $version } + return $copy +} + +function Resolve-ClineModelSelector { + param([string]$Model) + + if ([string]::IsNullOrWhiteSpace($Model)) { + return [pscustomobject]@{ Provider = $null; Model = $null; Valid = $false } + } + + $parts = $Model.Split([char[]]@('/'), 2, [System.StringSplitOptions]::None) + if ($parts.Count -lt 2 -or [string]::IsNullOrWhiteSpace($parts[0]) -or [string]::IsNullOrWhiteSpace($parts[1])) { + return [pscustomobject]@{ Provider = $null; Model = $Model; Valid = $false } + } + + return [pscustomobject]@{ Provider = $parts[0]; Model = $parts[1]; Valid = $true } +} + +function New-ClineEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $clineRoot = Join-Path $Inputs.Run.HomeDirectoryPath '.cline' + $dataDirectory = Join-Path $clineRoot 'data' + $settingsDirectory = Join-Path $dataDirectory 'settings' + $sandboxDataDirectory = Join-Path $clineRoot 'sandbox-data' + $teamDataDirectory = Join-Path $dataDirectory 'teams' + $hooksDirectory = Join-Path $clineRoot 'hooks' + foreach ($directory in @($clineRoot, $dataDirectory, $settingsDirectory, $sandboxDataDirectory, $teamDataDirectory, $hooksDirectory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) + return [pscustomobject]@{ + Environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $authVariables -Additional @{ + CLINE_DATA_DIR = $dataDirectory + CLINE_SANDBOX_DATA_DIR = $sandboxDataDirectory + CLINE_HOOKS_DIR = $hooksDirectory + CLINE_SESSION_BACKEND_MODE = 'local' + } + Root = $clineRoot + DataDirectory = $dataDirectory + SettingsDirectory = $settingsDirectory + SandboxDataDirectory = $sandboxDataDirectory + TeamDataDirectory = $teamDataDirectory + HooksDirectory = $hooksDirectory + ConfigPath = $clineRoot + } +} + +function Get-ClineInsideEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$EnvironmentData + ) + + $inside = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + CLINE_DATA_DIR = '/run/home/.cline/data' + CLINE_SANDBOX_DATA_DIR = '/run/home/.cline/sandbox-data' + CLINE_HOOKS_DIR = '/run/home/.cline/hooks' + CLINE_SESSION_BACKEND_MODE = 'local' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) + foreach ($name in $authVariables) { + if ($EnvironmentData.Environment.Contains($name) -and -not [string]::IsNullOrWhiteSpace([string]$EnvironmentData.Environment[$name])) { + $inside[$name] = [string]$EnvironmentData.Environment[$name] + } + } + return $inside +} + +function New-ClineCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$EnvironmentData, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $workingDirectory = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $configPath = Get-SandboxVisiblePath -HostPath $EnvironmentData.ConfigPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $dataRoot = Get-SandboxVisiblePath -HostPath $EnvironmentData.DataDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $hooksDirectory = Get-SandboxVisiblePath -HostPath $EnvironmentData.HooksDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--json', '--auto-approve', 'true', '--cwd', $workingDirectory, '--config', $configPath, '--data-dir', $dataRoot, '--hooks-dir', $hooksDirectory, '--provider', $selector.Provider, '--model', $selector.Model, '--retries', '0', '--timeout', [string]$Inputs.Profile.TimeoutSeconds)) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('--thinking') + $arguments.Add([string]$Inputs.Profile.ReasoningEffort) + } + return @($arguments) +} + +function Get-ClineCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + +function Get-ClinePreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $platform = Get-PlatformName + $commandInfo = Resolve-ExternalCommand -Name 'cline' + $sandboxInfo = if ($platform -eq 'linux') { Resolve-ExternalCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-ExternalCommand -Name 'sandbox-exec' } else { $null } + $versionObservation = $null + + if ($profile.Runner -ne 'cline') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than cline.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is cline.')) + } + $selector = Resolve-ClineModelSelector -Model ([string]$profile.Model) + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('Cline requires a model in execution-profile.json.') + } elseif (-not $selector.Valid) { + $reasons.Add("Cline requires a runner-native model selector in 'provider/model' form; received '$($profile.Model)'.") + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + $checks.Add((New-PreflightCheck -Name 'native_model_selector' -Status passed -Detail "--provider $($selector.Provider); --model $($selector.Model)")) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by cline.") } + if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by cline.") } + + $environmentData = New-ClineEnvironment -Inputs $Inputs + if ($null -eq $commandInfo) { + $reasons.Add('The Cline CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment $environmentData.Environment -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The Cline CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'cline --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + $help = Invoke-ClineCli -CommandInfo $commandInfo -Arguments @('--retries', '0', '--help') -Inputs $Inputs -Environment $environmentData.Environment -TimeoutSeconds 30 + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("Cline --retries 0 --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--json', '--auto-approve', '--cwd', '--config', '--data-dir', '--hooks-dir', '--provider', '--model', '--thinking', '--timeout', '--retries')) { + if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed Cline CLI does not advertise required flag '$flag'.") } + } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-ClineCliArguments -Inputs $Inputs -EnvironmentData $environmentData -VisiblePlatform $visiblePlatform + foreach ($forbidden in @('--id', '--continue', '--session', '--yolo', '--zen', '--tui')) { + if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Cline invocation must not use resume or interactive option '$forbidden'.") } + } + $retryIndex = [Array]::IndexOf([string[]]$constructed, '--retries') + if ($retryIndex -lt 0 -or $constructed[$retryIndex + 1] -ne '0') { $reasons.Add('The constructed Cline invocation must set --retries 0.') } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Cline advertises JSON/NDJSON output, isolated directories, provider/model selection, timeout, thinking, auto-approval, and zero internal retries.')) + } + } + } catch { + $reasons.Add("Could not inspect Cline CLI capabilities: $($_.Exception.Message)") + } + } + + $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) + $authVariable = $null + foreach ($name in $authVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { $authVariable = $name; break } + } + if ($authVariables.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status not_applicable -Detail 'No runner-known provider API-key environment variable is required for this Cline model selector.')) + } elseif ([string]::IsNullOrWhiteSpace($authVariable)) { + $reasons.Add("No narrow provider authentication environment variable is available for model provider '$($selector.Provider)'; ambient Cline auth profiles are not copied into an eval run.") + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) + } + + if ($platform -notin @('linux', 'macos')) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) confines Cline to the staged run and run-local data/config roots.")) + } + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new Cline process, supplies no --id, and never reuses a session.')) + $checks.Add((New-PreflightCheck -Name 'retry_semantics' -Status passed -Detail '--retries 0 disables Cline consecutive operational retries; the runner still starts exactly one semantic process with attempt_count=1.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'HOME, Cline data/config, hooks, sessions, and plugin roots are run-local and empty; no ambient user profile is copied.')) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) + $warnings.Add('Cline does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every Cline-launched tool.') + + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'Cline CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('cline --json', '--auto-approve true', '--retries 0', 'no --id session resume', 'run-local HOME', 'run-local Cline data/config/hooks directories', 'prompt on stdin')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function Write-ClineCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Invoke-ClineExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-ClinePreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false; session_id_supplied = $false }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'cline' + $environmentData = New-ClineEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-ExternalCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-ExternalCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $arguments = New-ClineCliArguments -Inputs $Inputs -EnvironmentData $environmentData -VisiblePlatform $visiblePlatform + if ($platform -eq 'linux' -and $hardFilesystem) { + $insideEnvironment = Get-ClineInsideEnvironment -Inputs $Inputs -EnvironmentData $environmentData + $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment -ReadOnlyRoots @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt') + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo -ReadOnlyRoots @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db') + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-ClineCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-ClineCapture -RunData $Inputs -RelativePath 'evidence/cline-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-ClineCapture -RunData $Inputs -RelativePath 'evidence/cline-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact); $artifacts.Add($stderrArtifact) + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Cline event parse error: $parseError") } + $eventCounts = @{} + $contentEnd = [System.Collections.Generic.List[string]]::new() + $contentStart = [System.Collections.Generic.List[string]]::new() + $jsonText = [System.Collections.Generic.List[string]]::new() + $jsonPartialText = [System.Collections.Generic.List[string]]::new() + $completionResultText = [System.Collections.Generic.List[string]]::new() + $finalText = $null + $returnedSessionId = $null + $failureMessage = $null + $usage = [ordered]@{} + $toolCalls = [System.Collections.Generic.List[object]]::new() + $commands = [System.Collections.Generic.List[object]]::new() + $files = [System.Collections.Generic.List[object]]::new() + foreach ($event in @($parsed.Events)) { + $topType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($topType)) { + $warnings.Add('Cline emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($topType)) { $eventCounts[$topType]++ } else { $eventCounts[$topType] = 1 } + $inner = Get-JsonProperty -Object $event -Name 'event' -Default $null + $eventType = if ($topType -eq 'agent_event' -and $null -ne $inner) { [string](Get-JsonProperty -Object $inner -Name 'type' -Default '') } else { $topType } + $payload = if ($topType -eq 'agent_event' -and $null -ne $inner) { $inner } else { $event } + + if ($topType -in @('say', 'ask')) { + $subtypeName = if ($topType -eq 'say') { 'say' } else { 'ask' } + $subtype = [string](Get-JsonProperty -Object $event -Name $subtypeName -Default '') + $text = [string](Get-JsonProperty -Object $event -Name 'text' -Default '') + $isPartial = [bool](Get-JsonProperty -Object $event -Name 'partial' -Default $false) + if (-not [string]::IsNullOrWhiteSpace($text)) { + if ($subtype -eq 'completion_result') { $completionResultText.Add($text) } + elseif ($subtype -eq 'text') { + if ($isPartial) { $jsonPartialText.Add($text) } else { $jsonText.Add($text) } + } + } + if ($topType -eq 'ask') { + if ($subtype -eq 'api_req_failed') { + $failureMessage = if ([string]::IsNullOrWhiteSpace($text)) { 'Cline reported an API request failure.' } else { $text } + } elseif ($subtype -in @('followup', 'plan_mode_respond', 'act_mode_respond')) { + $failureMessage = 'Cline requested interactive input during a noninteractive eval run.' + } elseif ($subtype -in @('use_mcp_server', 'command', 'tool')) { + $toolCalls.Add([ordered]@{ type = $subtype; name = $subtype }) + if ($subtype -eq 'command' -and -not [string]::IsNullOrWhiteSpace($text)) { + $commands.Add([ordered]@{ command = $text }) + } + } + } elseif ($subtype -in @('tool', 'command', 'command_output', 'mcp_server_request_started', 'mcp_server_response')) { + $toolName = [string](Get-JsonProperty -Object $event -Name 'name' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '')) + $toolCalls.Add([ordered]@{ type = $subtype; name = $toolName }) + if ($subtype -eq 'command' -and -not [string]::IsNullOrWhiteSpace($text)) { + $commands.Add([ordered]@{ command = $text }) + } + } + if ($subtype -eq 'completion_result' -and -not [string]::IsNullOrWhiteSpace($text)) { + $finalText = $text + } + if ($subtype -eq 'api_req_finished' -and -not [string]::IsNullOrWhiteSpace($text)) { + try { + $finishedUsage = $text | ConvertFrom-Json + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $finishedUsage -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } catch { + $warnings.Add('Cline api_req_finished text was not a usage JSON object; it was retained in the transcript.') + } + } + if ($subtype -notin @('task', 'error', 'api_req_started', 'api_req_finished', 'api_req_retried', 'api_req_retry_delayed', 'api_req_deleted', 'text', 'reasoning', 'completion_result', 'user_feedback', 'user_feedback_diff', 'command_output', 'tool', 'shell_integration_warning', 'browser_action', 'browser_action_result', 'command', 'mcp_server_request_started', 'mcp_server_response', 'new_task_started', 'new_task', 'subtask_result', 'checkpoint_saved', 'rooignore_error', 'diff_error', 'followup', 'plan_mode_respond', 'act_mode_respond', 'api_req_failed', 'use_mcp_server', 'resume_task', 'resume_completed_task', 'mistake_limit_reached', 'finishTask')) { + $warnings.Add("Unknown Cline $topType subtype '$subtype' was preserved as a warning.") + } + continue + } + switch ($eventType) { + 'done' { + $candidate = Get-JsonProperty -Object $payload -Name 'text' -Default '' + if (-not [string]::IsNullOrWhiteSpace([string]$candidate)) { $finalText = [string]$candidate } + $returnedSessionId = [string](Get-JsonProperty -Object $payload -Name 'sessionId' -Default (Get-JsonProperty -Object $payload -Name 'session_id' -Default $returnedSessionId)) + $doneUsage = Get-JsonProperty -Object $payload -Name 'usage' -Default $null + if ($null -ne $doneUsage) { + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $doneUsage -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } + } + 'content_end' { $text = [string](Get-JsonProperty -Object $payload -Name 'text' -Default ''); if (-not [string]::IsNullOrWhiteSpace($text)) { $contentEnd.Add($text) } } + 'content_start' { $text = [string](Get-JsonProperty -Object $payload -Name 'text' -Default ''); if (-not [string]::IsNullOrWhiteSpace($text)) { $contentStart.Add($text) } } + 'usage' { + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $payload -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } + 'tool_call' { $toolCalls.Add([ordered]@{ type = $eventType; name = [string](Get-JsonProperty -Object $payload -Name 'name' -Default (Get-JsonProperty -Object $payload -Name 'tool' -Default '')) }) } + 'tool_use' { $toolCalls.Add([ordered]@{ type = $eventType; name = [string](Get-JsonProperty -Object $payload -Name 'name' -Default (Get-JsonProperty -Object $payload -Name 'tool' -Default '')) }) } + 'command' { $commands.Add([ordered]@{ command = Get-JsonProperty -Object $payload -Name 'command' -Default (Get-JsonProperty -Object $payload -Name 'text' -Default '') }) } + 'file' { $files.Add([ordered]@{ path = Get-JsonProperty -Object $payload -Name 'path' -Default '' }) } + 'error' { $failureMessage = [string](Get-JsonProperty -Object $payload -Name 'message' -Default 'Cline emitted an error.') } + 'hook_event' { } + 'agent_start' { } + 'agent_end' { } + 'iteration_start' { } + 'iteration_end' { } + default { $warnings.Add("Unknown Cline event '$topType/$eventType' was preserved as a warning.") } + } + if ($eventType -eq 'usage') { + $eventUsage = Get-JsonProperty -Object $payload -Name 'usage' -Default $null + if ($null -ne $eventUsage) { + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $eventUsage -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } + } + } + if ([string]::IsNullOrWhiteSpace($finalText)) { + if ($contentEnd.Count -gt 0) { $finalText = [string]::Join('', @($contentEnd)) } + elseif ($contentStart.Count -gt 0) { $finalText = [string]::Join('', @($contentStart)) } + elseif ($completionResultText.Count -gt 0) { $finalText = [string]$completionResultText[$completionResultText.Count - 1] } + elseif ($jsonText.Count -gt 0) { $finalText = [string]$jsonText[$jsonText.Count - 1] } + elseif ($jsonPartialText.Count -gt 0) { $finalText = [string]::Join('', @($jsonPartialText)) } + } + + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out'; $reason = 'cline_timeout'; $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Cline did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $failureMessage) { + $failureDetail = if ([string]::IsNullOrWhiteSpace($failureMessage)) { 'Cline exited unsuccessfully.' } else { $failureMessage } + $status = 'failed'; $reason = 'cline_failure'; $failure = New-ExecutionFailure -Code 'cline_failure' -Message $failureDetail + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $reason = 'cline_did_not_return_final_response'; $warnings.Add('Cline exited successfully without a final response event.') + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/cline-events.jsonl'; complete = $true }) + tokens = if ($usage.Count -eq 0) { New-UnavailableMetric -Reason 'cline_did_not_expose_usage' } else { New-AvailableMetric -Value $usage } + tool_calls = New-AvailableMetric -Value $toolCalls.Count + cost = if ($usage.Contains('cost')) { New-AvailableMetric -Value $usage['cost'] } else { New-UnavailableMetric -Reason 'cline_did_not_expose_cost' } + } + $finished = [DateTime]::UtcNow + $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('cline --json', '--auto-approve true', '--retries 0', 'no --id session resume', 'run-local HOME', 'run-local Cline data/config/hooks directories', 'prompt on stdin')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $nativeSelector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) + $credentialEvidence = [ordered]@{ + model_provider = $nativeSelector.Provider + provider_environment_variables = @(if ($nativeSelector.Valid) { Get-ProviderAuthenticationVariables -Provider $nativeSelector.Provider }) + unrelated_environment_excluded = $true + child_tool_visibility = 'not_exposed_by_runner_environment; Cline child filtering is not independently observable' + value_observed = $false + } + $sessionResultId = if ([string]::IsNullOrWhiteSpace($returnedSessionId)) { $sessionId } else { $returnedSessionId } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Cline accepted the requested runner-native model selector, thinking, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; native_provider = $nativeSelector.Provider; native_model = $nativeSelector.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort; retries = 0 } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; session_id_supplied = $false; retry_argument = 0; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-ClineDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-ClineInputs + Write-RunnerJson -Value (Get-ClinePreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-ClineInputs + $result = Invoke-ClineExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 new file mode 100644 index 0000000..bf102e9 --- /dev/null +++ b/scripts/eval-runners/codex/runner.ps1 @@ -0,0 +1,566 @@ +<#! +.SYNOPSIS + Codex Eval Runner adapter. + +.DESCRIPTION + This is the only place where Codex CLI flags, CODEX_HOME handling, JSONL + event parsing, and Codex isolation limitations are defined. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'codex' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'supported' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + credential_child_filtering = 'supported' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-CodexInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Get-CodexAuthSource { + $authVariables = @(Get-ProviderAuthenticationVariables -Provider 'openai') + foreach ($name in $authVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return [pscustomobject]@{ Kind = 'environment'; Name = $name; Path = $null } + } + } + + $configuredHome = [Environment]::GetEnvironmentVariable('CODEX_HOME') + $codexHome = if ([string]::IsNullOrWhiteSpace($configuredHome)) { + Join-Path ([Environment]::GetFolderPath('UserProfile')) '.codex' + } else { + $configuredHome + } + $authPath = Join-Path $codexHome 'auth.json' + if (Test-Path -LiteralPath $authPath -PathType Leaf) { + return [pscustomobject]@{ Kind = 'file_unsupported'; Name = 'auth.json'; Path = (Resolve-Path -LiteralPath $authPath).Path } + } + + return [pscustomobject]@{ Kind = 'missing'; Name = $null; Path = $null } +} + +function Invoke-CodexCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-CodexHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [string[]]$Arguments = @('--ask-for-approval', 'never', 'exec', '--help') + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-CodexCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-CodexDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $version } + return $copy +} + +function New-CodexCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][string]$LastResponsePath, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $outputArgument = Get-SandboxVisiblePath -HostPath $LastResponsePath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--ask-for-approval', 'never', 'exec', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--skip-git-repo-check', '--json', '--color', 'never', '--cd', $directoryArgument, '--model', $Inputs.Profile.Model, '--sandbox', 'workspace-write', '--config', 'shell_environment_policy.inherit=none', '--output-last-message', $outputArgument)) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('-c') + $arguments.Add("model_reasoning_effort=$($Inputs.Profile.ReasoningEffort)") + } + $arguments.Add('-') + return @($arguments) +} + +function Get-CodexCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + +function Get-CodexPreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $platform = Get-PlatformName + $sandboxName = switch ($platform) { + 'linux' { 'bwrap' } + 'macos' { 'sandbox-exec' } + default { $null } + } + $sandboxInfo = if ([string]::IsNullOrWhiteSpace([string]$sandboxName)) { $null } else { Resolve-SandboxCommand -Name $sandboxName } + $versionObservation = $null + + if ($profile.Runner -ne 'codex') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than codex.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is codex.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('Codex requires a model in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by codex.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by codex.") + } + + if ($null -eq $commandInfo) { + $reasons.Add('The Codex CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment (New-RunnerEnvironment -Run $run) -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The Codex CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'codex --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + + $globalHelp = Get-CodexHelpResult -CommandInfo $commandInfo -Inputs $Inputs -Arguments @('--help') + $help = Get-CodexHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($globalHelp.TimedOut -or $globalHelp.ExitCode -ne 0) { + $reasons.Add("Codex --help failed with exit status $($globalHelp.ExitCode).") + } + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("Codex --ask-for-approval never exec --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($globalHelp.Stdout, $globalHelp.Stderr, $help.Stdout, $help.Stderr)) + foreach ($flag in @('--ask-for-approval', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--json', '--output-last-message', '--sandbox', '--cd', '--model', '--config')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed Codex CLI does not advertise required flag '$flag'.") + } + } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-CodexCliArguments -Inputs $Inputs -LastResponsePath (Join-Path $run.RunRoot 'evidence/codex-final.txt') -VisiblePlatform $visiblePlatform + if (@($constructed) -contains '--approve-for-me') { + $reasons.Add('The constructed Codex invocation must not combine --approve-for-me with explicit --sandbox selection.') + } + $sandboxIndex = [Array]::IndexOf([string[]]$constructed, '--sandbox') + $approvalIndex = [Array]::IndexOf([string[]]$constructed, '--ask-for-approval') + $execIndex = [Array]::IndexOf([string[]]$constructed, 'exec') + if ($approvalIndex -lt 0 -or $execIndex -lt 0 -or $approvalIndex -gt $execIndex -or $sandboxIndex -lt 0) { + $reasons.Add('The constructed Codex invocation must set --ask-for-approval never before exec and retain --sandbox workspace-write.') + } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Codex accepts the constructed noninteractive invocation: --ask-for-approval never, exec, --sandbox workspace-write, ephemeral JSON output, and isolated configuration controls.')) + } + } + } catch { + $reasons.Add("Could not inspect Codex CLI capabilities: $($_.Exception.Message)") + } + } + + $auth = Get-CodexAuthSource + if ($auth.Kind -eq 'missing') { + $reasons.Add('No narrow Codex provider API-key environment variable is available.') + } elseif ($auth.Kind -eq 'file_unsupported') { + $reasons.Add('Codex auth.json cannot be copied into the worker HOME: the evaluated agent could read that credential file. Set the provider API-key environment variable instead.') + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the narrow $($auth.Name) environment variable; the child shell policy is set to inherit=none.")) + } + + if ($null -eq $sandboxName) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $sandboxName confines Codex to the staged run and required system runtime paths; Codex sandbox=workspace-write remains enabled inside it.")) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter uses --ephemeral and never supplies a resume, continue, or session identifier.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are never copied into the worker HOME.')) + + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $sandboxName filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function New-CodexEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Auth + ) + + $codexHome = Join-Path $Inputs.Run.HomeDirectoryPath '.codex' + New-Item -ItemType Directory -Path $codexHome -Force | Out-Null + $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider 'openai') -Additional @{ CODEX_HOME = $codexHome } + if ($Auth.Kind -ne 'environment') { + throw 'Codex execution requires a provider environment credential; file credentials are not safe to expose in the worker HOME.' + } + return $environment +} + +function Get-LinuxCodexSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $args = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $args.Add($argument) } + foreach ($path in @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt')) { + if (Test-Path -LiteralPath $path) { + $args.Add('--ro-bind'); $args.Add($path); $args.Add($path) + } + } + $args.Add('--proc'); $args.Add('/proc') + $args.Add('--dev'); $args.Add('/dev') + $args.Add('--tmpfs'); $args.Add('/tmp') + $args.Add('--bind'); $args.Add($Inputs.Run.RunRoot); $args.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $args.Add('--ro-bind'); $args.Add($commandDirectory); $args.Add($commandDirectory) + } + } + $args.Add('--chdir'); $args.Add('/run/repo') + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + CODEX_HOME = '/run/home/.codex' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($authName in @(Get-ProviderAuthenticationVariables -Provider 'openai')) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + foreach ($key in @($insideEnvironment.Keys)) { + $args.Add('--setenv'); $args.Add($key); $args.Add([string]$insideEnvironment[$key]) + } + $args.Add('--') + $args.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $args.Add($prefix) } + return @($args) +} + +function New-CodexMacosSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'codex-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $readRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db', $commandDirectory) + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in $readRoots | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('\', '/').Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Write-CodexCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $parent = Split-Path -Parent $path + New-Item -ItemType Directory -Path $parent -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Invoke-CodexExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-CodexPreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + $failureText = [string]::Join('; ', @($preflight.reasons)) + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $failureText) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $auth = Get-CodexAuthSource + $environment = New-CodexEnvironment -Inputs $Inputs -Auth $auth + $lastResponsePath = 'evidence/codex-final.txt' + New-Item -ItemType Directory -Path (Join-Path $Inputs.Run.RunRoot 'evidence') -Force | Out-Null + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $arguments = New-CodexCliArguments -Inputs $Inputs -LastResponsePath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -VisiblePlatform $visiblePlatform + + if ($platform -eq 'linux' -and $hardFilesystem) { + $sandboxArguments = Get-LinuxCodexSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -Environment $environment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-CodexMacosSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-CodexCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + $stdoutArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Codex event parse error: $parseError") } + $finalText = $null + $threadId = $null + $turnFailure = $null + $usage = $null + $toolCalls = 0 + $commands = [System.Collections.Generic.List[object]]::new() + $files = [System.Collections.Generic.List[object]]::new() + $eventCounts = @{} + foreach ($event in @($parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $warnings.Add('Codex emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + switch ($eventType) { + 'thread.started' { $threadId = [string](Get-JsonProperty -Object $event -Name 'thread_id' -Default '') } + 'item.completed' { + $item = Get-JsonProperty -Object $event -Name 'item' -Default $null + $itemType = [string](Get-JsonProperty -Object $item -Name 'type' -Default '') + if ($itemType -eq 'agent_message') { + $candidate = [string](Get-JsonProperty -Object $item -Name 'text' -Default '') + if (-not [string]::IsNullOrWhiteSpace($candidate)) { $finalText = $candidate } + } elseif ($itemType -in @('command_execution', 'mcp_tool_call', 'file_change')) { + $toolCalls++ + if ($itemType -eq 'command_execution') { + $commands.Add([ordered]@{ type = $itemType; command = Get-JsonProperty -Object $item -Name 'command'; exit_code = Get-JsonProperty -Object $item -Name 'exit_code' }) + } else { + $files.Add([ordered]@{ type = $itemType; item = $itemType }) + } + } + } + 'turn.completed' { + $usage = Get-JsonProperty -Object $event -Name 'usage' -Default $null + } + 'turn.failed' { $turnFailure = Get-JsonProperty -Object $event -Name 'error' -Default 'Codex turn failed.' } + 'error' { $turnFailure = Get-JsonProperty -Object $event -Name 'message' -Default 'Codex emitted an error.' } + { $_ -in @('turn.started', 'item.started', 'item.updated') } { } + default { $warnings.Add("Unknown Codex event '$eventType' was preserved as a warning.") } + } + } + if (Test-Path -LiteralPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -PathType Leaf) { + $lastArtifact = New-ArtifactReference -Run $Inputs.Run -Path $lastResponsePath -Scope run -MediaType 'text/plain; charset=utf-8' + $artifacts.Add($lastArtifact) + if ([string]::IsNullOrWhiteSpace($finalText)) { + $finalText = [System.IO.File]::ReadAllText((Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)), [System.Text.UTF8Encoding]::new($false)) + } + } + + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out' + $reason = 'codex_timeout' + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Codex did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $turnFailure) { + $status = 'failed' + $reason = 'codex_failure' + $failure = New-ExecutionFailure -Code 'codex_failure' -Message ([string]$turnFailure) + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $warnings.Add('Codex exited successfully without a final agent message.') + $reason = 'codex_did_not_return_final_response' + } + + $tokenMetric = if ($null -eq $usage) { + New-UnavailableMetric -Reason 'codex_did_not_expose_turn_usage' + } else { + $usageValue = [ordered]@{} + foreach ($name in @('input_tokens', 'cached_input_tokens', 'output_tokens', 'reasoning_output_tokens')) { + $value = Get-JsonProperty -Object $usage -Name $name -Default $null + if ($null -ne $value) { $usageValue[$name] = $value } + } + if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'codex_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/codex-events.jsonl'; complete = $true }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value $toolCalls + cost = New-UnavailableMetric -Reason 'codex_runner_does_not_estimate_cost' + } + $finished = [DateTime]::UtcNow + $sessionResultId = if ([string]::IsNullOrWhiteSpace($threadId)) { $sessionId } else { $threadId } + $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $credentialEvidence = [ordered]@{ + source = $auth.Kind + provider_environment_variable = $auth.Name + unrelated_environment_excluded = $true + child_tool_visibility = 'codex_shell_environment_policy_inherit_none' + value_observed = $false + } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested model and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; output_last_message_argument = (Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform); credential = $credentialEvidence }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-CodexDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-CodexInputs + Write-RunnerJson -Value (Get-CodexPreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-CodexInputs + $result = Invoke-CodexExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/contracts/execution-profile.schema.json b/scripts/eval-runners/contracts/execution-profile.schema.json new file mode 100644 index 0000000..5fd745d --- /dev/null +++ b/scripts/eval-runners/contracts/execution-profile.schema.json @@ -0,0 +1,27 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-execution-profile/1", + "title": "Codebelt Agentic Eval Execution Profile", + "type": "object", + "additionalProperties": false, + "required": [ + "schema", + "runner", + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds", + "concurrency" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-execution-profile/1" }, + "runner": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" }, + "model": { "type": "string", "minLength": 1 }, + "reasoning_effort": { "type": ["string", "null"], "minLength": 1 }, + "configuration_profile": { "type": "string", "minLength": 1 }, + "tool_profile": { "type": "string", "minLength": 1 }, + "timeout_seconds": { "type": "integer", "minimum": 1, "maximum": 86400 }, + "concurrency": { "type": "integer", "minimum": 1 } + } +} diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json new file mode 100644 index 0000000..161607e --- /dev/null +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -0,0 +1,124 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-execution-result/1", + "title": "Codebelt Agentic Eval Execution Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "run_id", + "session", + "status", + "run", + "final_response", + "runner", + "harness", + "requested", + "resolved", + "started_utc", + "finished_utc", + "duration_seconds", + "exit", + "input", + "isolation", + "telemetry", + "evidence", + "artifacts", + "warnings", + "compatibility_deviations", + "attempt_count" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-execution-result/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "run_id": { "type": "string", "minLength": 1 }, + "session": { + "type": "object", + "required": ["id", "fresh", "resumed"], + "properties": { + "id": { "type": "string", "minLength": 1 }, + "fresh": { "const": true }, + "resumed": { "const": false } + } + }, + "status": { "enum": ["completed", "failed", "timed_out", "cancelled", "incompatible"] }, + "final_response": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "enum": ["available", "unavailable"] }, + "text": { "type": "string" }, + "reason": { "type": "string" } + } + }, + "requested": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" }, + "timeout_seconds": { "type": "integer" } + } + }, + "resolved": { + "type": "object", + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": ["string", "null"] }, + "tool_profile": { "type": ["string", "null"] }, + "status": { "enum": ["unavailable", "accepted_request", "resolved"] }, + "accepted": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" } + } + } + } + }, + "duration_seconds": { "type": "number", "minimum": 0 }, + "input": { + "type": "object", + "required": ["prompt_sha256", "run_json_sha256", "profile_sha256"] + }, + "isolation": { + "type": "object", + "required": ["status", "level", "hard_filesystem_confinement", "capabilities", "mechanisms"], + "properties": { + "status": { "enum": ["verified", "unverified"] }, + "level": { "enum": ["strict", "pragmatic", "unsupported"] }, + "hard_filesystem_confinement": { "type": "boolean" }, + "capabilities": { "type": "object" }, + "mechanisms": { "type": "array", "items": { "type": "string" } }, + "required_controls": { "type": "array", "items": { "type": "string" } }, + "unproven_controls": { "type": "array", "items": { "type": "string" } } + } + }, + "telemetry": { "type": "object" }, + "evidence": { "type": "object" }, + "artifacts": { "type": "array" }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, + "attempt_count": { "const": 1 } + } +} diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json new file mode 100644 index 0000000..b942e9c --- /dev/null +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -0,0 +1,60 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-runner-preflight/1", + "title": "Codebelt Agentic Eval Runner Preflight Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "status", + "runner", + "harness", + "run", + "requested", + "checks", + "resolved_capabilities", + "isolation", + "mechanisms", + "warnings", + "reasons" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-runner-preflight/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "status": { "enum": ["compatible", "incompatible"] }, + "requested": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" }, + "timeout_seconds": { "type": "integer" } + } + }, + "checks": { "type": "array" }, + "resolved_capabilities": { "type": "object" }, + "isolation": { + "type": "object", + "required": ["level", "status", "hard_filesystem_confinement", "unproven_controls"], + "properties": { + "level": { "enum": ["strict", "pragmatic", "unsupported"] }, + "status": { "enum": ["verified", "unverified"] }, + "hard_filesystem_confinement": { "type": "boolean" }, + "unproven_controls": { "type": "array", "items": { "type": "string" } } + } + }, + "mechanisms": { "type": "array", "items": { "type": "string" } }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "reasons": { "type": "array", "items": { "type": "string" } } + } +} diff --git a/scripts/eval-runners/contracts/runner-descriptor.schema.json b/scripts/eval-runners/contracts/runner-descriptor.schema.json new file mode 100644 index 0000000..e4ae3b4 --- /dev/null +++ b/scripts/eval-runners/contracts/runner-descriptor.schema.json @@ -0,0 +1,41 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-runner-descriptor/1", + "title": "Codebelt Agentic Eval Runner Descriptor", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "name", + "version", + "platforms", + "harness", + "capabilities", + "supported_telemetry", + "configuration_profiles", + "tool_profiles" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-runner-descriptor/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 }, + "platforms": { "type": "array", "items": { "type": "string" } }, + "harness": { + "type": "object", + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "capabilities": { + "type": "object", + "additionalProperties": { "enum": ["supported", "conditional", "unsupported"] } + }, + "supported_telemetry": { "type": "array", "items": { "type": "string" } }, + "configuration_profiles": { "type": "array", "items": { "type": "string" } }, + "tool_profiles": { "type": "array", "items": { "type": "string" } } + } +} diff --git a/scripts/eval-runners/fake/runner.ps1 b/scripts/eval-runners/fake/runner.ps1 new file mode 100644 index 0000000..9453c56 --- /dev/null +++ b/scripts/eval-runners/fake/runner.ps1 @@ -0,0 +1,278 @@ +<#! +.SYNOPSIS + Deterministic Eval Runner used for protocol conformance tests. + +.DESCRIPTION + This runner never invokes a model or a provider. It records the same + boundaries as a real runner and emits deterministic outcomes selected by + -Scenario or AGENTIC_FAKE_SCENARIO. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile, + + [ValidateSet('normal', 'refusal', 'timeout', 'failure', 'incompatible', 'escape', 'unknown-event')] + [string]$Scenario +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'fake' + version = '0.9.1-test' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'deterministic-fake'; version = '1' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'unsupported' + cache_token_telemetry = 'unsupported' + tool_call_telemetry = 'supported' + command_evidence = 'supported' + file_evidence = 'supported' + cost_telemetry = 'unsupported' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'tool_call_telemetry', 'command_evidence', 'file_evidence') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Get-FakeScenario { + if (-not [string]::IsNullOrWhiteSpace($Scenario)) { + return $Scenario + } + $fromEnvironment = [Environment]::GetEnvironmentVariable('AGENTIC_FAKE_SCENARIO') + if ([string]::IsNullOrWhiteSpace($fromEnvironment)) { + return 'normal' + } + return $fromEnvironment.ToLowerInvariant() +} + +function Resolve-FakeInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + $resolvedRun = Resolve-RunContract -RunPath $Run + $resolvedProfile = Resolve-ExecutionProfile -ProfilePath $Profile + return [pscustomobject]@{ Run = $resolvedRun; Profile = $resolvedProfile } +} + +function Get-FakePreflight { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [string]$ScenarioValue = 'normal' + ) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + + if ($profile.Runner -ne 'fake') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is fake.')) + } + + if ($profile.ConfigurationProfile -notin @($descriptor.configuration_profiles)) { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is not supported by fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'configuration_profile' -Status passed -Detail $profile.ConfigurationProfile)) + } + if ($profile.ToolProfile -notin @($descriptor.tool_profiles)) { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is not supported by fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'tool_profile' -Status passed -Detail $profile.ToolProfile)) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_process' -Status passed -Detail 'Each fake execute command creates a new process and session id.')) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The prompt bytes are read once and recorded without transformation.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "repo=$($run.WorkingDirectoryPath); home=$($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail 'Fake escape probes are rejected by the contained-path guard.')) + $checks.Add((New-PreflightCheck -Name 'candidate_skill_boundary' -Status passed -Detail "candidate_skill_exposed=$($run.CandidateSkillExposed)")) + if ($ScenarioValue -eq 'incompatible') { + $reasons.Add('The deterministic incompatible scenario was requested.') + } + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName + } + $capabilities['candidate_skill_exposure'] = if ($run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + if ($reasons.Count -gt 0) { + $warnings.Add('No execute process is started for an incompatible preflight.') + } + + return New-PreflightDocument -Descriptor $descriptor -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function Write-FakeEvidence { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][string]$SessionId, + [Parameter(Mandatory = $true)][string]$ScenarioValue + ) + + $evidenceDirectory = Join-Path $Inputs.Run.RunRoot 'evidence' + New-Item -ItemType Directory -Path $evidenceDirectory -Force | Out-Null + $eventsPath = Join-Path $evidenceDirectory 'fake-events.jsonl' + $promptEvidencePath = Join-Path $evidenceDirectory 'prompt-delivery.json' + $boundaryEvidencePath = Join-Path $evidenceDirectory 'boundary-probes.json' + + $events = [System.Collections.Generic.List[string]]::new() + $events.Add((([ordered]@{ type = 'session.started'; session_id = $SessionId } | ConvertTo-Json -Compress))) + $events.Add((([ordered]@{ + type = 'task.input' + ordinal = 1 + prompt_sha256 = $Inputs.Run.PromptHash + byte_length = $Inputs.Run.PromptBytes.Length + first_task_input = $true + } | ConvertTo-Json -Compress))) + $events.Add((([ordered]@{ type = 'response.completed'; status = if ($ScenarioValue -eq 'refusal') { 'refusal' } else { 'completed' } } | ConvertTo-Json -Compress))) + if ($ScenarioValue -eq 'unknown-event') { + $events.Add((([ordered]@{ type = 'future.event.v99'; payload = 'ignored-by-conformance-adapter' } | ConvertTo-Json -Compress))) + } + [System.IO.File]::WriteAllText($eventsPath, ([string]::Join("`n", $events) + "`n"), [System.Text.UTF8Encoding]::new($false)) + + [ordered]@{ + prompt_sha256 = $Inputs.Run.PromptHash + first_task_input_sha256 = $Inputs.Run.PromptHash + first_task_input_bytes = $Inputs.Run.PromptBytes.Length + byte_exact = $true + candidate_skill_exposed = $Inputs.Run.CandidateSkillExposed + working_directory = $Inputs.Run.WorkingDirectoryPath + home_directory = $Inputs.Run.HomeDirectoryPath + global_rules_visible = $false + global_memory_visible = $false + global_plugins_visible = $false + global_same_name_skill_visible = $false + } | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $promptEvidencePath -Encoding utf8NoBOM + + $boundary = [ordered]@{ + read_outside_run = [ordered]@{ attempted = $false; blocked = $true; path = '../eval-metadata.json' } + write_outside_run = [ordered]@{ attempted = $false; blocked = $true; path = '../escape-write.txt' } + } + if ($ScenarioValue -eq 'escape') { + foreach ($probe in @('read_outside_run', 'write_outside_run')) { + $boundary[$probe].attempted = $true + try { + [void](Resolve-ContainedPath -BasePath $Inputs.Run.RunRoot -RelativePath ([string]$boundary[$probe].path) -FieldName $probe) + $boundary[$probe].blocked = $false + } catch { + $boundary[$probe].blocked = $true + $boundary[$probe].error = $_.Exception.Message + } + } + } + $boundary | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $boundaryEvidencePath -Encoding utf8NoBOM + + return @( + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/fake-events.jsonl' -Scope run -MediaType 'application/x-ndjson'), + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/prompt-delivery.json' -Scope run -MediaType 'application/json'), + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/boundary-probes.json' -Scope run -MediaType 'application/json') + ) +} + +function Invoke-FakeExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $scenarioValue = Get-FakeScenario + if ($scenarioValue -notin @('normal', 'refusal', 'timeout', 'failure', 'incompatible', 'escape', 'unknown-event')) { + throw "Unsupported fake scenario '$scenarioValue'." + } + + $preflight = Get-FakePreflight -Inputs $Inputs -ScenarioValue $scenarioValue + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $warnings = [System.Collections.Generic.List[string]]::new() + $deviations = [System.Collections.Generic.List[string]]::new() + + if ($preflight.status -eq 'incompatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; candidate_skill_exposure = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } }) -IsolationMechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -CompatibilityDeviations @($deviations) -Evidence ([ordered]@{ preflight = $preflight }) -AttemptCount 1 + } + + $artifacts = @(Write-FakeEvidence -Inputs $Inputs -SessionId $sessionId -ScenarioValue $scenarioValue) + if ($scenarioValue -eq 'unknown-event') { + $warnings.Add('Unknown fake event future.event.v99 was preserved as an explicit warning.') + } + + $status = 'completed' + $finalResponse = 'Deterministic fake runner completed the blind eval arm.' + $finalReason = $null + $exitStatus = [Nullable[int]]0 + $failure = $null + if ($scenarioValue -eq 'refusal') { + $finalResponse = 'I cannot complete this request.' + $warnings.Add('The harness returned a refusal; it was normalized as a completed response, not retried.') + } elseif ($scenarioValue -eq 'timeout') { + $status = 'timed_out' + $finalResponse = $null + $finalReason = 'fake_timeout' + $exitStatus = $null + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'The deterministic fake exceeded its configured execution window.' + } elseif ($scenarioValue -eq 'failure') { + $status = 'failed' + $finalResponse = $null + $finalReason = 'harness_failure' + $exitStatus = [Nullable[int]]17 + $failure = New-ExecutionFailure -Code 'fake_harness_failure' -Message 'The deterministic fake reported a harness failure.' + } + + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fake-events.jsonl'; complete = $true }) + tokens = New-UnavailableMetric -Reason 'fake_harness_does_not_expose_usage' + tool_calls = New-AvailableMetric -Value 0 + cost = New-UnavailableMetric -Reason 'fake_harness_does_not_expose_cost' + } + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalResponse -FinalResponseReason $finalReason -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' }; prompt_fidelity = 'supported'; model_configuration_lock = 'supported'; response_capture = 'supported' }) -IsolationMechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -Telemetry $telemetry -Artifacts $artifacts -Warnings @($warnings) -CompatibilityDeviations @($deviations) -Evidence ([ordered]@{ scenario = $scenarioValue; prompt_first_input = $true; resume = $false; preflight = $preflight }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { + Write-RunnerJson -Value $descriptor -AsOutput + } + 'preflight' { + $inputs = Resolve-FakeInputs + $preflight = Get-FakePreflight -Inputs $inputs -ScenarioValue (Get-FakeScenario) + Write-RunnerJson -Value $preflight -AsOutput + } + 'execute' { + $inputs = Resolve-FakeInputs + $result = Invoke-FakeExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/freebuff-readiness.md b/scripts/eval-runners/freebuff-readiness.md new file mode 100644 index 0000000..e836068 --- /dev/null +++ b/scripts/eval-runners/freebuff-readiness.md @@ -0,0 +1,34 @@ +# Freebuff runner readiness + +Status: planned and blocked. No Freebuff Eval Runner is shipped or advertised +until the official Freebuff CLI provides a supported noninteractive transport. + +The current upstream [Freebuff README](https://github.com/CodebuffAI/freebuff/blob/main/README.md) +documents the `freebuff` TUI. The current upstream +[headless CLI request](https://github.com/CodebuffAI/freebuff/issues/947) asks +for a print/headless mode with machine-readable output; it is evidence that +the required transport is not currently part of the supported CLI contract. +The locally installed CLI was also inspected without a model request: +`freebuff --version` reported `0.0.150`, and its help exposed `login`, +`--continue`, `--cwd`, and `--version`, but no prompt argument, print mode, +JSON/NDJSON output mode, or supported noninteractive session protocol. + +The missing capability is therefore the complete one-prompt-in, +machine-readable-result-out, fresh-session transport. TUI keystroke +automation, screen scraping, PTY emulation, private transport use, and the +paid Codebuff SDK are not substitutes for that capability. + +Freebuff becomes ready only when the supported official CLI can demonstrate all +of the following without a human TTY: + +1. Accept exactly one task/prompt deterministically. +2. Start a fresh independent session without resume or reuse. +3. Select or identify the requested model and configuration. +4. Return the complete final result and any available structured events. +5. Expose a timeout and an enforceable run-local HOME/config/workspace setup. +6. Permit the common Eval Runner contract to keep grading material and paired + arm data out of the worker. + +At that point a runner may be added under `freebuff/runner.ps1` using the +unchanged `describe`, `preflight`, and `execute` protocol. Until then, the +absence of a runner is intentional. diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 new file mode 100644 index 0000000..b5c21e0 --- /dev/null +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -0,0 +1,682 @@ +<#! +.SYNOPSIS + GitHub Copilot CLI Eval Runner adapter. + +.DESCRIPTION + This is the only place where GitHub Copilot CLI flags, COPILOT_HOME + handling, non-interactive JSONL event parsing, Copilot authentication, + and Copilot isolation limitations are defined. It implements the unchanged + describe/preflight/execute process contract shared by every runner. + + Copilot with claude-haiku-4.5 is the Codebelt reference evaluation + configuration. The reference is a repository convention for economical, + stable comparison; it is not an Anthropic default. The model stays fully + configurable through execution-profile.json, so any Copilot-served model can + be selected. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +# GitHub Copilot checks these token variables before its OS credential store and +# GitHub CLI fallback. The values are forwarded only to the Copilot process; +# --secret-env-vars removes them from shell and MCP child environments. +$copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'github-copilot' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'conditional' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'unsupported' + credential_child_filtering = 'supported' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-CopilotInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Get-CopilotTokenVariable { + foreach ($name in $copilotAuthVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $name + } + } + return $null +} + +function Get-CopilotGhConfigDirectory { + # GH_CONFIG_DIR is an authentication-state exception to the isolated + # Copilot configuration roots. Resolve it from GitHub CLI's documented + # precedence without reading or logging any credential file. + $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if ([string]::IsNullOrWhiteSpace($configured)) { + $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { + $configured = Join-Path $xdgConfig 'gh' + } elseif ((Get-PlatformName) -eq 'windows') { + $applicationData = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) + if (-not [string]::IsNullOrWhiteSpace($applicationData)) { + $configured = Join-Path $applicationData 'GitHub CLI' + } + } else { + $userHome = [Environment]::GetEnvironmentVariable('HOME') + if ([string]::IsNullOrWhiteSpace($userHome)) { + $userHome = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) + } + if (-not [string]::IsNullOrWhiteSpace($userHome)) { + $configured = Join-Path (Join-Path $userHome '.config') 'gh' + } + } + } + + if ([string]::IsNullOrWhiteSpace($configured) -or -not (Test-Path -LiteralPath $configured -PathType Container)) { + return $null + } + return [System.IO.Path]::GetFullPath($configured) +} + +function Get-CopilotGitHubCliToken { + $gh = Resolve-ExternalCommand -Name 'gh' + if ($null -eq $gh) { + return $null + } + + $environment = New-RunnerProbeEnvironment + foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'GH_CONFIG_DIR')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + + $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null + try { + $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { + return $null + } + $token = ([string]$process.Stdout).Trim() + if ([string]::IsNullOrWhiteSpace($token)) { + return $null + } + return $token + } finally { + if (Test-Path -LiteralPath $probeDirectory) { + Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function Resolve-CopilotAuthentication { + $tokenVariable = Get-CopilotTokenVariable + if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { + return [pscustomobject]@{ + Source = 'environment' + TokenVariable = $tokenVariable + TokenValue = $null + GitHubCliTokenResolved = $false + } + } + + $githubCliToken = Get-CopilotGitHubCliToken + if (-not [string]::IsNullOrWhiteSpace($githubCliToken)) { + return [pscustomobject]@{ + Source = 'github_cli_token' + TokenVariable = 'GH_TOKEN' + TokenValue = $githubCliToken + GitHubCliTokenResolved = $true + } + } + + return [pscustomobject]@{ + Source = 'copilot_os_keychain_or_github_cli_unverified' + TokenVariable = $null + TokenValue = $null + GitHubCliTokenResolved = $false + } +} + +function Invoke-CopilotCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-CopilotHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [string[]]$Arguments = @('--help') + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-CopilotCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-CopilotDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $version } + return $copy +} + +function New-CopilotCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $secretList = ($copilotAuthVariables -join ',') + $arguments = [System.Collections.Generic.List[string]]::new() + # Noninteractive one-shot JSONL run. --allow-all-tools is a broad + # tool-approval grant required for programmatic execution; it does not + # include --allow-all-paths or --allow-all-urls, so normal path and URL + # verification remains active. --no-ask-user keeps the agent from pausing + # for questions. Repository-owned custom instructions remain enabled so + # both paired arms see the staged repository exactly as supplied. Ambient + # Copilot state is excluded by the run-local COPILOT_HOME and environment + # roots. --disable-builtin-mcps drops the built-in GitHub MCP server. + # --secret-env-vars removes the listed credentials from shell and MCP + # child environments. + foreach ($argument in @( + '-C', $directoryArgument, + '--model', $Inputs.Profile.Model, + '--output-format', 'json', + '--allow-all-tools', + '--no-ask-user', + '--disable-builtin-mcps', + '--no-color', + '--log-level', 'none', + '--no-auto-update', + ('--secret-env-vars=' + $secretList) + )) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('--reasoning-effort') + $arguments.Add([string]$Inputs.Profile.ReasoningEffort) + } + return @($arguments) +} + +function Get-CopilotCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + +function Get-CopilotPreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $versionObservation = $null + + if ($profile.Runner -ne 'github-copilot') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than github-copilot.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is github-copilot.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('GitHub Copilot requires a model in execution-profile.json (claude-haiku-4.5 is the Codebelt reference).') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by github-copilot.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by github-copilot.") + } + + if ($null -eq $commandInfo) { + $reasons.Add('The GitHub Copilot CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment (New-RunnerEnvironment -Run $run) -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The GitHub Copilot CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'copilot --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + $help = Get-CopilotHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("Copilot --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--output-format', '--model', '--allow-all-tools', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed Copilot CLI does not advertise required flag '$flag'.") + } + } + $visibleForConstruction = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-CopilotCliArguments -Inputs $Inputs -VisiblePlatform $visibleForConstruction + foreach ($forbidden in @('--resume', '-r', '--continue', '--session-id', '--connect', '--yolo', '--allow-all', '--allow-all-paths', '--allow-all-urls')) { + if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Copilot invocation must not use session-continuation or over-broad permission option '$forbidden'.") } + } + foreach ($required in @('--output-format', '--allow-all-tools', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { + $present = @($constructed) -contains $required + if ($required -eq '--secret-env-vars') { + $present = $present -or (@($constructed | Where-Object { $_ -like '--secret-env-vars=*' }).Count -gt 0) + } + if (-not $present) { $reasons.Add("The constructed Copilot invocation must include '$required'.") } + } + $promptOptionCount = @($constructed | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count + if ($promptOptionCount -ne 0) { $reasons.Add('The constructed Copilot invocation must not place the prompt in argv; prompt delivery uses stdin.') } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Copilot accepts the constructed noninteractive invocation: stdin prompt delivery, --output-format json, --model, broad --allow-all-tools approval, --no-ask-user, --disable-builtin-mcps, --secret-env-vars, and no session continuation.')) + } + } + } catch { + $reasons.Add("Could not inspect Copilot CLI capabilities: $($_.Exception.Message)") + } + } + + $authState = Resolve-CopilotAuthentication + if ($authState.Source -eq 'environment') { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) + } elseif ($authState.Source -eq 'github_cli_token') { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'GitHub CLI fallback resolved a token in the trusted runner; only a protected token environment variable will be passed to Copilot.')) + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail 'No explicit token is present and GitHub CLI fallback did not yield a token; native Copilot OS-keychain lookup is delegated to the installed CLI. This preflight does not contact the Copilot service.')) + $warnings.Add('Authentication readiness beyond explicit environment tokens and the observable GitHub CLI fallback cannot be proven without a live Copilot request; preflight remains conditional and does not reject a tokenless native OAuth/keychain configuration.') + } + + if ($platform -notin @('linux', 'macos')) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) sandbox confines Copilot to the staged run and required system runtime paths.")) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new Copilot process, supplies one stdin prompt, and passes no --resume, --continue, --session-id, or --connect.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter points COPILOT_HOME, Copilot cache, HOME, USERPROFILE, and XDG roots at the run''s isolated home, disables built-in MCP servers, and preserves only staged repository-owned custom instructions; personal Copilot skills, plugins, MCP config, sessions, memories, and instructions are not imported.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "-C $($run.WorkingDirectoryPath); COPILOT_HOME under $($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The prepared UTF-8 prompt bytes are supplied once through stdin; the execution fake proves the received bytes match the staged prompt.')) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only supported authentication state is made available to Copilot; --secret-env-vars removes every listed token variable from shell and MCP child environments; no Copilot profile or credential file is copied.')) + if ($platform -eq 'macos') { + $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') + } + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function New-CopilotEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $copilotHome = Join-Path $Inputs.Run.HomeDirectoryPath '.copilot' + $copilotCacheHome = Join-Path $Inputs.Run.HomeDirectoryPath '.copilot-cache' + New-Item -ItemType Directory -Path $copilotHome -Force | Out-Null + New-Item -ItemType Directory -Path $copilotCacheHome -Force | Out-Null + $additional = @{ + COPILOT_HOME = $copilotHome + COPILOT_CACHE_HOME = $copilotCacheHome + COPILOT_AUTO_UPDATE = 'false' + } + $tokenVariable = Get-CopilotTokenVariable + $authState = Resolve-CopilotAuthentication + if ($authState.Source -eq 'github_cli_token') { + $additional[$authState.TokenVariable] = $authState.TokenValue + } + return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $copilotAuthVariables -Additional $additional +} + +function New-CopilotInsideEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + COPILOT_HOME = '/run/home/.copilot' + COPILOT_CACHE_HOME = '/run/home/.copilot-cache' + COPILOT_AUTO_UPDATE = 'false' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($authName in $copilotAuthVariables) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + return $insideEnvironment +} + +function Write-CopilotCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Add-NullableInt64 { + param([object]$Current, [object]$Value) + + if ($null -eq $Value) { return $Current } + if ($null -eq $Current) { return [int64]$Value } + return ([int64]$Current + [int64]$Value) +} + +function Read-CopilotEvents { + param( + [Parameter(Mandatory = $true)][object]$Parsed, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Warnings + ) + + $assistantContents = [System.Collections.Generic.List[string]]::new() + $finalText = $null + $observedModel = $null + $usageInput = $null + $usageOutput = $null + $usageCacheRead = $null + $usageCacheWrite = $null + $usageNumToolCalls = 0 + $usageSeen = $false + $toolStarts = 0 + $sessionError = $null + $eventCounts = @{} + + foreach ($event in @($Parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $Warnings.Add('Copilot emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + $data = Get-JsonProperty -Object $event -Name 'data' -Default $null + switch ($eventType) { + 'assistant.message' { + $content = [string](Get-JsonProperty -Object $data -Name 'content' -Default '') + if (-not [string]::IsNullOrWhiteSpace($content)) { + $assistantContents.Add($content) + $finalText = $content + } + $model = [string](Get-JsonProperty -Object $data -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($model)) { $observedModel = $model } + } + 'assistant.usage' { + $usageSeen = $true + $usageInput = Add-NullableInt64 -Current $usageInput -Value (Get-JsonProperty -Object $data -Name 'inputTokens' -Default $null) + $usageOutput = Add-NullableInt64 -Current $usageOutput -Value (Get-JsonProperty -Object $data -Name 'outputTokens' -Default $null) + $usageCacheRead = Add-NullableInt64 -Current $usageCacheRead -Value (Get-JsonProperty -Object $data -Name 'cacheReadTokens' -Default $null) + $usageCacheWrite = Add-NullableInt64 -Current $usageCacheWrite -Value (Get-JsonProperty -Object $data -Name 'cacheWriteTokens' -Default $null) + $numToolCalls = Get-JsonProperty -Object $data -Name 'numToolCalls' -Default $null + if ($null -ne $numToolCalls) { $usageNumToolCalls += [int]$numToolCalls } + $model = [string](Get-JsonProperty -Object $data -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($model)) { $observedModel = $model } + } + 'tool.execution_start' { $toolStarts++ } + 'session.error' { $sessionError = [string](Get-JsonProperty -Object $data -Name 'message' -Default 'Copilot reported a session error.') } + { $_ -in @('session.start', 'session.info', 'session.idle', 'session.shutdown', 'session.task_complete', 'user.message', 'assistant.message_start', 'assistant.message_delta', 'assistant.turn_start', 'assistant.turn_end', 'assistant.reasoning', 'assistant.tool_call_delta', 'tool.execution_progress', 'tool.execution_partial_result', 'tool.execution_complete', 'command.execute', 'command.completed', 'session.usage_info', 'session.usage_checkpoint') } { } + default { $Warnings.Add("Unknown Copilot event '$eventType' was preserved as a warning.") } + } + } + + if ([string]::IsNullOrWhiteSpace($finalText) -and $assistantContents.Count -gt 0) { + $finalText = [string]::Join("`n", $assistantContents) + } + $toolCalls = if ($toolStarts -gt 0) { $toolStarts } else { $usageNumToolCalls } + + return [pscustomobject]@{ + FinalText = $finalText + ObservedModel = $observedModel + UsageSeen = $usageSeen + UsageInput = $usageInput + UsageOutput = $usageOutput + UsageCacheRead = $usageCacheRead + UsageCacheWrite = $usageCacheWrite + ToolCalls = $toolCalls + SessionError = $sessionError + EventCounts = $eventCounts + } +} + +function Invoke-CopilotExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-CopilotPreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $environment = New-CopilotEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $arguments = New-CopilotCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + + if ($platform -eq 'linux' -and $hardFilesystem) { + $insideEnvironment = New-CopilotInsideEnvironment -Inputs $Inputs -Environment $environment + $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-CopilotCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + + $warnings = [System.Collections.Generic.List[string]]::new() + $parsed = if ([string]::IsNullOrEmpty([string]$process.Stdout)) { + [pscustomobject]@{ Events = @(); Errors = @() } + } else { + ConvertFrom-JsonLines -Text $process.Stdout + } + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Copilot event parse error: $parseError") } + $parsedEvents = Read-CopilotEvents -Parsed $parsed -Warnings $warnings + + $finalText = $parsedEvents.FinalText + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out' + $reason = 'copilot_timeout' + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Copilot did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $parsedEvents.SessionError) { + $status = 'failed' + $reason = 'copilot_failure' + $message = if ($null -ne $parsedEvents.SessionError) { [string]$parsedEvents.SessionError } else { "Copilot exited with status $($process.ExitCode)." } + $failure = New-ExecutionFailure -Code 'copilot_failure' -Message $message + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $warnings.Add('Copilot exited successfully without an assistant message; the final response is unavailable.') + $reason = 'copilot_did_not_return_final_response' + } + + $tokenMetric = if (-not $parsedEvents.UsageSeen) { + New-UnavailableMetric -Reason 'copilot_did_not_expose_usage_events' + } else { + $usageValue = [ordered]@{} + if ($null -ne $parsedEvents.UsageInput) { $usageValue['input_tokens'] = [int64]$parsedEvents.UsageInput } + if ($null -ne $parsedEvents.UsageOutput) { $usageValue['output_tokens'] = [int64]$parsedEvents.UsageOutput } + if ($null -ne $parsedEvents.UsageCacheRead) { $usageValue['cache_read_tokens'] = [int64]$parsedEvents.UsageCacheRead } + if ($null -ne $parsedEvents.UsageCacheWrite) { $usageValue['cache_write_tokens'] = [int64]$parsedEvents.UsageCacheWrite } + if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'copilot_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/copilot-events.jsonl'; complete = $true }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value ([int]$parsedEvents.ToolCalls) + cost = New-UnavailableMetric -Reason 'copilot_exposes_a_billing_multiplier_not_a_currency_cost' + } + + $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + + $authState = Resolve-CopilotAuthentication + $credentialEvidence = [ordered]@{ + source = $authState.Source + github_token_variable = $authState.TokenVariable + secret_env_vars = @($copilotAuthVariables) + secret_env_var_scope = @('shell', 'mcp') + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_forwarded = $false + login_profile_copied = $false + auth_file_copied = $false + value_observed = $false + } + $observedModel = if ([string]::IsNullOrWhiteSpace([string]$parsedEvents.ObservedModel)) { $null } else { [string]$parsedEvents.ObservedModel } + $resolvedConfiguration = [ordered]@{ + status = 'accepted_request' + reason = 'Copilot accepted the requested model alias and configuration; it does not expose a distinct backend model snapshot beyond the model it reports in usage events.' + observations = [ordered]@{ + model = $Inputs.Profile.Model + reasoning_effort = $Inputs.Profile.ReasoningEffort + observed_model = $observedModel + } + } + + $finished = [DateTime]::UtcNow + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration $resolvedConfiguration -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $parsedEvents.EventCounts; observed_model = $observedModel; prompt_delivery = 'stdin'; prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-CopilotDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-CopilotInputs + Write-RunnerJson -Value (Get-CopilotPreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-CopilotInputs + $result = Invoke-CopilotExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 new file mode 100644 index 0000000..92adc75 --- /dev/null +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -0,0 +1,532 @@ +<#! +.SYNOPSIS + OpenCode Eval Runner adapter. + +.DESCRIPTION + This is the only place where OpenCode CLI flags, project/global + configuration handling, sandbox process setup, and JSON event parsing are + defined. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'opencode' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'OpenCode CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'supported' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + credential_child_filtering = 'conditional' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-OpenCodeInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Invoke-OpenCodeCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-OpenCodeHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-OpenCodeCli -CommandInfo $CommandInfo -Arguments @('run', '--help') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Get-OpenCodeAuthVariable { + param([Parameter(Mandatory = $true)][string]$Provider) + + $variables = @(Get-ProviderAuthenticationVariables -Provider $Provider) + foreach ($name in $variables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $name + } + } + return $null +} + +function Get-OpenCodeModelProvider { + param([string]$Model) + + if ([string]::IsNullOrWhiteSpace($Model) -or $Model -notmatch '/') { + return $null + } + $parts = $Model.Split([char[]]@('/'), 2, [System.StringSplitOptions]::None) + if ($parts.Count -lt 2 -or [string]::IsNullOrWhiteSpace($parts[0]) -or [string]::IsNullOrWhiteSpace($parts[1])) { + return $null + } + return $parts[0] +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-OpenCodeDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $version } + return $copy +} + +function New-OpenCodeCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('run', '--format', 'json', '--dir', $directoryArgument, '--model', $Inputs.Profile.Model, '--auto')) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('--variant') + $arguments.Add([string]$Inputs.Profile.ReasoningEffort) + } + return @($arguments) +} + +function Get-OpenCodeCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + +function Get-OpenCodePreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $platform = Get-PlatformName + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $versionObservation = $null + + if ($profile.Runner -ne 'opencode') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than opencode.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is opencode.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('OpenCode requires a model in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by opencode.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by opencode.") + } + if ($null -eq $commandInfo) { + $reasons.Add('The OpenCode CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment (New-RunnerEnvironment -Run $run) -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The OpenCode CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'opencode --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + $help = Get-OpenCodeHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("OpenCode run --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--format', '--dir', '--model', '--auto')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed OpenCode CLI does not advertise required flag '$flag'.") + } + } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-OpenCodeCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + foreach ($forbidden in @('--pure', '--continue', '--session')) { + if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed OpenCode invocation must not use session or project-suppression option '$forbidden'.") } + } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'OpenCode run advertises noninteractive, model, directory, and structured-output controls; the adapter intentionally does not use --pure.')) + } + } + } catch { + $reasons.Add("Could not inspect OpenCode CLI capabilities: $($_.Exception.Message)") + } + } + + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$profile.Model) + $authVariable = if ([string]::IsNullOrWhiteSpace($modelProvider)) { $null } else { Get-OpenCodeAuthVariable -Provider $modelProvider } + $knownAuthVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + if ($knownAuthVariables.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status not_applicable -Detail 'No runner-known provider API-key environment variable is required for this OpenCode model selector.')) + } elseif ([string]::IsNullOrWhiteSpace($authVariable)) { + $reasons.Add("No narrow provider authentication environment variable is available for model provider '$modelProvider'. OpenCode global auth profiles are not copied into an eval run.") + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) + } + + if ($platform -notin @('linux', 'macos')) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) sandbox confines the process to the staged run and required system runtime paths.")) + } + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new opencode run process and supplies no resume, continue, or session id.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter isolates global/user configuration roots and deliberately preserves repository-owned project configuration; OPENCODE_DISABLE_PROJECT_CONFIG is not used.')) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) + $warnings.Add('OpenCode does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every OpenCode-launched tool.') + + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + if ($platform -eq 'macos') { + $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') + } + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function New-OpenCodeEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $configDirectory = Join-Path $Inputs.Run.HomeDirectoryPath 'opencode-config' + New-Item -ItemType Directory -Path $configDirectory -Force | Out-Null + $configPath = Join-Path $configDirectory 'opencode.json' + [System.IO.File]::WriteAllText($configPath, '{}', [System.Text.UTF8Encoding]::new($false)) + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $authVariables -Additional @{ + OPENCODE_CONFIG_DIR = $configDirectory + OPENCODE_CONFIG = $configPath + OPENCODE_DISABLE_AUTOUPDATE = '1' + } +} + +function Get-LinuxSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $args = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $args.Add($argument) } + foreach ($path in @('/usr', '/bin', '/lib', '/lib64', '/etc', '/opt')) { + if (Test-Path -LiteralPath $path) { + $args.Add('--ro-bind'); $args.Add($path); $args.Add($path) + } + } + $args.Add('--proc'); $args.Add('/proc') + $args.Add('--dev'); $args.Add('/dev') + $args.Add('--tmpfs'); $args.Add('/tmp') + $args.Add('--bind'); $args.Add($Inputs.Run.RunRoot); $args.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $args.Add('--ro-bind'); $args.Add($commandDirectory); $args.Add($commandDirectory) + } + } + $args.Add('--chdir'); $args.Add('/run/repo') + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + OPENCODE_CONFIG_DIR = '/run/home/opencode-config' + OPENCODE_CONFIG = '/run/home/opencode-config/opencode.json' + OPENCODE_DISABLE_AUTOUPDATE = [string]$Environment['OPENCODE_DISABLE_AUTOUPDATE'] + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + foreach ($authName in $authVariables) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + foreach ($key in @($insideEnvironment.Keys)) { + $args.Add('--setenv'); $args.Add($key); $args.Add([string]$insideEnvironment[$key]) + } + $args.Add('--') + $args.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $args.Add($prefix) } + return @($args) +} + +function New-MacosSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'opencode-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $systemReadRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db', $commandDirectory) + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in $systemReadRoots | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Write-OpenCodeCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Invoke-OpenCodeExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-OpenCodePreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $environment = New-OpenCodeEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $model = [string]$Inputs.Profile.Model + $arguments = New-OpenCodeCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + + if ($platform -eq 'linux' -and $hardFilesystem) { + $sandboxArguments = Get-LinuxSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -Environment $environment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + $arguments + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-OpenCodeCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact); $artifacts.Add($stderrArtifact) + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("OpenCode event parse error: $parseError") } + $finalTextParts = [System.Collections.Generic.List[string]]::new() + $eventCounts = @{} + $toolCalls = 0 + $commands = [System.Collections.Generic.List[object]]::new() + $usageBuckets = [ordered]@{} + $failureMessage = $null + foreach ($event in @($parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $warnings.Add('OpenCode emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + $part = Get-JsonProperty -Object $event -Name 'part' -Default $null + switch ($eventType) { + 'text' { + $text = Get-JsonProperty -Object $event -Name 'text' -Default (Get-JsonProperty -Object $part -Name 'text' -Default '') + if (-not [string]::IsNullOrWhiteSpace([string]$text)) { $finalTextParts.Add([string]$text) } + } + 'step_finish' { + $tokens = Get-JsonProperty -Object $part -Name 'tokens' -Default (Get-JsonProperty -Object $event -Name 'tokens' -Default $null) + if ($null -ne $tokens) { + foreach ($name in @('input', 'output', 'reasoning', 'cache_read', 'cache_write')) { + $value = Get-JsonProperty -Object $tokens -Name $name -Default $null + if ($null -ne $value) { $usageBuckets[$name] = $value } + } + } + $costValue = Get-JsonProperty -Object $part -Name 'cost' -Default (Get-JsonProperty -Object $event -Name 'cost' -Default $null) + if ($null -ne $costValue) { $usageBuckets['cost'] = $costValue } + } + 'tool_use' { + $toolCalls++ + $toolName = Get-JsonProperty -Object $part -Name 'tool' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '') + $commands.Add([ordered]@{ tool = [string]$toolName }) + } + 'error' { + $failureMessage = [string](Get-JsonProperty -Object $event -Name 'message' -Default (Get-JsonProperty -Object $part -Name 'message' -Default 'OpenCode emitted an error.')) + } + 'step_start' { } + 'reasoning' { } + default { $warnings.Add("Unknown OpenCode event '$eventType' was preserved as a warning.") } + } + } + $finalText = if ($finalTextParts.Count -gt 0) { [string]::Join('', $finalTextParts) } else { $null } + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out'; $reason = 'opencode_timeout'; $failure = New-ExecutionFailure -Code 'timed_out' -Message 'OpenCode did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $failureMessage) { + $status = 'failed'; $reason = 'opencode_failure'; $failure = New-ExecutionFailure -Code 'opencode_failure' -Message ([string]$failureMessage) + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $reason = 'opencode_did_not_return_final_response'; $warnings.Add('OpenCode exited successfully without a text response.') + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/opencode-events.jsonl'; complete = $true }) + tokens = if ($usageBuckets.Count -eq 0) { New-UnavailableMetric -Reason 'opencode_did_not_expose_usage' } else { New-AvailableMetric -Value $usageBuckets } + tool_calls = New-AvailableMetric -Value $toolCalls + cost = if ($usageBuckets.Contains('cost')) { New-AvailableMetric -Value $usageBuckets['cost'] } else { New-UnavailableMetric -Reason 'opencode_did_not_expose_cost' } + } + $capabilities = Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $credentialNames = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + $credentialEvidence = [ordered]@{ + model_provider = $modelProvider + provider_environment_variables = $credentialNames + unrelated_environment_excluded = $true + child_tool_visibility = 'provider_credential_may_be_visible_to_native_child_tools; no supported child filter is exposed' + value_observed = $false + } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $process.FinishedUtc.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'OpenCode accepted the requested runner-native model selector and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); prompt_first_input = $true; resume = $false; model_argument = $model; sandbox = $sandboxEvidence; project_configuration = 'repository_owned_project_config_preserved'; disable_project_config_environment = $false; credential = $credentialEvidence }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-OpenCodeDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-OpenCodeInputs + Write-RunnerJson -Value (Get-OpenCodePreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-OpenCodeInputs + $result = Invoke-OpenCodeExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/resolve-runner.ps1 b/scripts/eval-runners/resolve-runner.ps1 new file mode 100644 index 0000000..9517dac --- /dev/null +++ b/scripts/eval-runners/resolve-runner.ps1 @@ -0,0 +1,33 @@ +<#! +.SYNOPSIS + Resolves one package-local Eval Runner without guessing or falling back. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [string]$Runner +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$protocol = 'codebeltnet/agentic/eval-runner-protocol/1' +if ($Runner -notmatch '^[a-z0-9][a-z0-9-]*$') { + throw "Runner name '$Runner' is not a safe package-local runner name." +} + +$runnerPath = Join-Path (Join-Path $PSScriptRoot $Runner) 'runner.ps1' +if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { + throw "Selected Eval Runner '$Runner' is unavailable in this package." +} + +$resolved = (Resolve-Path -LiteralPath $runnerPath).Path +$root = (Resolve-Path -LiteralPath $PSScriptRoot).Path +$relative = [System.IO.Path]::GetRelativePath($root, $resolved).Replace('\', '/') + +[ordered]@{ + schema = 'codebeltnet/agentic/eval-runner-resolution/1' + protocol_version = $protocol + runner = $Runner + path = $relative +} | ConvertTo-Json -Depth 10 -Compress diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 new file mode 100644 index 0000000..98d38ad --- /dev/null +++ b/scripts/eval-runners/runner-common.ps1 @@ -0,0 +1,1088 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Get-RunnerSchemaNames { + return [ordered]@{ + Protocol = 'codebeltnet/agentic/eval-runner-protocol/1' + Descriptor = 'codebeltnet/agentic/eval-runner-descriptor/1' + Preflight = 'codebeltnet/agentic/eval-runner-preflight/1' + Profile = 'codebeltnet/agentic/eval-execution-profile/1' + Result = 'codebeltnet/agentic/eval-execution-result/1' + PortableResult = 'codebeltnet/agentic/eval-result/2' + Run = 'codebeltnet/agentic/eval-run/1' + } +} + +function Get-JsonProperty { + param( + [object]$Object, + [string]$Name, + [object]$Default = $null + ) + + if ($null -ne $Object -and $Object -is [System.Collections.IDictionary] -and $Object.Contains($Name)) { + if ($null -ne $Object[$Name]) { + return $Object[$Name] + } + return $Default + } + + if ($null -ne $Object -and $Object.PSObject.Properties.Name -contains $Name -and $null -ne $Object.$Name) { + return $Object.$Name + } + + return $Default +} + +function Get-JsonPropertyNames { + param([object]$Object) + + if ($null -eq $Object) { + return @() + } + if ($Object -is [System.Collections.IDictionary]) { + return @($Object.Keys | ForEach-Object { [string]$_ }) + } + return @($Object.PSObject.Properties.Name) +} + +function Test-JsonProperty { + param( + [object]$Object, + [Parameter(Mandatory = $true)][string]$Name + ) + + return (Get-JsonPropertyNames -Object $Object) -contains $Name +} + +function Read-RunnerJson { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "JSON file '$Path' does not exist." + } + + return [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json +} + +function Write-RunnerJson { + param( + [Parameter(Mandatory = $true)][object]$Value, + [switch]$AsOutput + ) + + $json = ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine) + if ($AsOutput) { + [Console]::Out.Write($json) + return + } + + return $json +} + +function Get-Sha256HexFromBytes { + param([Parameter(Mandatory = $true)][AllowEmptyCollection()][byte[]]$Bytes) + + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + return ([Convert]::ToHexString($sha.ComputeHash($Bytes))).ToLowerInvariant() + } finally { + $sha.Dispose() + } +} + +function Get-Sha256HexFromFile { + param([Parameter(Mandatory = $true)][string]$Path) + + $resolved = (Resolve-Path -LiteralPath $Path -ErrorAction Stop).Path + return Get-Sha256HexFromBytes -Bytes ([System.IO.File]::ReadAllBytes($resolved)) +} + +function Test-Sha256 { + param([string]$Value) + + return -not [string]::IsNullOrWhiteSpace($Value) -and $Value -match '^[0-9a-fA-F]{64}$' +} + +function Test-PathInside { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath + ) + + $base = ([System.IO.Path]::GetFullPath($BasePath)).TrimEnd([char[]]@('\', '/')) + $candidate = ([System.IO.Path]::GetFullPath($CandidatePath)).TrimEnd([char[]]@('\', '/')) + return $candidate -eq $base -or $candidate.StartsWith($base + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::OrdinalIgnoreCase) +} + +function Assert-SafeRelativePath { + param( + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName + ) + + if ([string]::IsNullOrWhiteSpace($RelativePath) -or [System.IO.Path]::IsPathRooted($RelativePath) -or $RelativePath -match '^[A-Za-z]:') { + throw "$FieldName must be a non-empty relative path." + } + + $normalized = $RelativePath.Replace('\', '/') + if (($normalized -split '/') -contains '..') { + throw "$FieldName must not contain a parent-directory segment." + } +} + +function Resolve-ContainedPath { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName, + [ValidateSet('Any', 'File', 'Directory')][string]$Kind = 'Any' + ) + + Assert-SafeRelativePath -RelativePath $RelativePath -FieldName $FieldName + $resolvedBase = (Resolve-Path -LiteralPath $BasePath -ErrorAction Stop).Path + $candidate = [System.IO.Path]::GetFullPath((Join-Path $resolvedBase ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $resolvedBase -CandidatePath $candidate)) { + throw "$FieldName resolves outside the run directory." + } + + $exists = switch ($Kind) { + 'File' { Test-Path -LiteralPath $candidate -PathType Leaf } + 'Directory' { Test-Path -LiteralPath $candidate -PathType Container } + default { Test-Path -LiteralPath $candidate } + } + if (-not $exists) { + throw "$FieldName '$RelativePath' does not exist under '$resolvedBase'." + } + + $resolvedCandidate = (Resolve-Path -LiteralPath $candidate -ErrorAction Stop).Path + if (-not (Test-PathInside -BasePath $resolvedBase -CandidatePath $resolvedCandidate)) { + throw "$FieldName resolves through a link outside the run directory." + } + return $resolvedCandidate +} + +function Get-PlatformName { + if ($IsWindows) { return 'windows' } + if ($IsMacOS) { return 'macos' } + if ($IsLinux) { return 'linux' } + return 'unknown' +} + +function Get-SandboxVisiblePath { + param( + [Parameter(Mandatory = $true)][string]$HostPath, + [Parameter(Mandatory = $true)][string]$RunRoot, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [string]$MountRoot = '/run' + ) + + $fullHostPath = [System.IO.Path]::GetFullPath($HostPath) + if ($Platform -ne 'linux' -or -not (Test-PathInside -BasePath $RunRoot -CandidatePath $fullHostPath)) { + return $fullHostPath + } + + $relative = [System.IO.Path]::GetRelativePath(([System.IO.Path]::GetFullPath($RunRoot)), $fullHostPath).Replace('\', '/') + if ($relative -eq '.') { + return $MountRoot.TrimEnd('/') + } + return $MountRoot.TrimEnd('/') + '/' + $relative.TrimStart('/') +} + +function Get-ObservableVersionFromText { + param([string]$Text) + + foreach ($line in ($Text -split "`r?`n")) { + $trimmed = $line.Trim() + if (-not [string]::IsNullOrWhiteSpace($trimmed)) { + return $trimmed + } + } + return $null +} + +function New-RunnerProbeEnvironment { + $environment = [ordered]@{} + foreach ($name in @('PATH', 'SystemRoot', 'WINDIR', 'ComSpec', 'PATHEXT', 'LANG', 'LC_ALL', 'TZ', 'SSL_CERT_FILE', 'NODE_PATH')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + $environment['CI'] = '1' + $environment['NO_COLOR'] = '1' + return $environment +} + +function Get-ExternalCommandVersion { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [string]$WorkingDirectory = '', + [System.Collections.IDictionary]$Environment = (New-RunnerProbeEnvironment), + [int]$TimeoutSeconds = 30 + ) + + $probeDirectory = $WorkingDirectory + $ownsProbeDirectory = $false + if ([string]::IsNullOrWhiteSpace($probeDirectory)) { + $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-version-probe-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null + $ownsProbeDirectory = $true + } + try { + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @('--version')) -WorkingDirectory $probeDirectory -Environment $Environment -TimeoutSeconds $TimeoutSeconds + $text = [string]::Join("`n", @($process.Stdout, $process.Stderr)) + $version = Get-ObservableVersionFromText -Text $text + return [pscustomobject]@{ + Version = if ($process.TimedOut -or $process.ExitCode -ne 0 -or [string]::IsNullOrWhiteSpace($version)) { 'unavailable' } else { $version } + Available = (-not $process.TimedOut -and $process.ExitCode -eq 0 -and -not [string]::IsNullOrWhiteSpace($version)) + Process = $process + } + } catch { + return [pscustomobject]@{ Version = 'unavailable'; Available = $false; Process = $null; Error = $_.Exception.Message } + } finally { + if ($ownsProbeDirectory -and (Test-Path -LiteralPath $probeDirectory)) { + Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function Get-IsolationCapabilityAssessment { + param([System.Collections.IDictionary]$Capabilities) + + $required = @( + 'fresh_context', + 'isolated_home_config', + 'isolated_working_directory', + 'ambient_candidate_skill_exclusion', + 'candidate_skill_exposure', + 'prompt_fidelity', + 'model_configuration_lock', + 'response_capture' + ) + $unproven = [System.Collections.Generic.List[string]]::new() + foreach ($name in $required) { + $value = if ($null -ne $Capabilities -and $Capabilities.Contains($name)) { [string]$Capabilities[$name] } else { 'unavailable' } + $valid = if ($name -eq 'candidate_skill_exposure') { $value -in @('supported', 'excluded') } else { $value -eq 'supported' } + if (-not $valid) { $unproven.Add($name) } + } + + $filesystemValue = if ($null -ne $Capabilities -and $Capabilities.Contains('filesystem_confinement')) { [string]$Capabilities['filesystem_confinement'] } else { 'unavailable' } + $hardFilesystem = $filesystemValue -eq 'supported' + $mandatoryProven = $unproven.Count -eq 0 + return [pscustomobject]@{ + MandatoryProven = $mandatoryProven + HardFilesystemConfinement = $hardFilesystem + Level = if (-not $mandatoryProven) { 'unsupported' } elseif ($hardFilesystem) { 'strict' } else { 'pragmatic' } + Unproven = $unproven.ToArray() + Required = @($required) + } +} + +function Resolve-RunContract { + param([Parameter(Mandatory = $true)][string]$RunPath) + + $resolvedRunPath = (Resolve-Path -LiteralPath $RunPath -ErrorAction Stop).Path + $runRoot = Split-Path -Parent $resolvedRunPath + $run = Read-RunnerJson -Path $resolvedRunPath + $schemas = Get-RunnerSchemaNames + + if ([string]$run.schema -ne $schemas.Run) { + throw "run.json must declare '$($schemas.Run)'." + } + if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { + throw 'run.json must require a fresh context, a staged filesystem/workspace boundary, and an isolated home.' + } + + $mode = [string]$run.mode + if ($mode -notin @('with_skill', 'without_skill')) { + throw "run.json mode '$mode' is not with_skill or without_skill." + } + + $promptPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.promptFile) -FieldName 'promptFile' -Kind File + $workingPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.workingDirectory) -FieldName 'workingDirectory' -Kind Directory + $homePath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.homeDirectory) -FieldName 'homeDirectory' -Kind Directory + + $skillPath = $null + if ($mode -eq 'with_skill') { + if ([string]::IsNullOrWhiteSpace([string]$run.skillDirectory)) { + throw 'with_skill run.json must declare skillDirectory.' + } + $skillPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.skillDirectory) -FieldName 'skillDirectory' -Kind Directory + if (-not (Test-Path -LiteralPath (Join-Path $skillPath 'SKILL.md') -PathType Leaf)) { + throw 'with_skill skillDirectory must contain SKILL.md.' + } + } else { + if ($null -ne $run.skillDirectory -and -not [string]::IsNullOrWhiteSpace([string]$run.skillDirectory)) { + throw 'without_skill run.json must not declare skillDirectory.' + } + $skillRoot = Join-Path $runRoot 'skill' + if (Test-Path -LiteralPath $skillRoot) { + throw 'without_skill run must not contain a skill directory.' + } + } + + $promptBytes = [System.IO.File]::ReadAllBytes($promptPath) + $fixtureHash = [string](Get-JsonProperty -Object $run -Name 'fixtureHash' -Default '') + if (-not (Test-Sha256 -Value $fixtureHash)) { + throw 'run.json fixtureHash must be a SHA-256 value.' + } + if ($mode -eq 'with_skill' -and -not (Test-Sha256 -Value ([string]$run.skillHash))) { + throw 'with_skill run.json skillHash must be a SHA-256 value.' + } + + return [pscustomobject]@{ + RunPath = $resolvedRunPath + RunRoot = $runRoot + Contract = $run + EvalId = [int]$run.evalId + EvalName = [string]$run.evalName + Mode = $mode + PromptPath = $promptPath + PromptBytes = $promptBytes + PromptHash = Get-Sha256HexFromBytes -Bytes $promptBytes + WorkingDirectoryPath = $workingPath + HomeDirectoryPath = $homePath + SkillDirectoryPath = $skillPath + CandidateSkillExposed = $mode -eq 'with_skill' + FixtureHash = $fixtureHash + SkillHash = if ($mode -eq 'with_skill') { [string]$run.skillHash } else { $null } + } +} + +function Assert-ProfileHasNoSecrets { + param([Parameter(Mandatory = $true)][object]$Profile) + + foreach ($property in @($Profile.PSObject.Properties)) { + if ([string]$property.Name -match '(?i)(secret|token|password|credential|api[_-]?key|private[_-]?key)') { + throw "execution-profile.json must not contain secret-bearing field '$($property.Name)'." + } + } +} + +function Resolve-ExecutionProfile { + param([Parameter(Mandatory = $true)][string]$ProfilePath) + + $resolvedProfilePath = (Resolve-Path -LiteralPath $ProfilePath -ErrorAction Stop).Path + $profile = Read-RunnerJson -Path $resolvedProfilePath + $schemas = Get-RunnerSchemaNames + if ([string]$profile.schema -ne $schemas.Profile) { + throw "execution-profile.json must declare '$($schemas.Profile)'." + } + Assert-ProfileHasNoSecrets -Profile $profile + + $allowedProperties = @('schema', 'runner', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency') + foreach ($propertyName in @(Get-JsonPropertyNames -Object $profile)) { + if ($allowedProperties -notcontains $propertyName) { + throw "execution-profile.json contains unsupported field '$propertyName'." + } + } + + $timeout = [int](Get-JsonProperty -Object $profile -Name 'timeout_seconds' -Default 0) + $concurrency = [int](Get-JsonProperty -Object $profile -Name 'concurrency' -Default 0) + if ($timeout -lt 1 -or $timeout -gt 86400) { + throw 'execution-profile.json timeout_seconds must be between 1 and 86400.' + } + if ($concurrency -lt 1) { + throw 'execution-profile.json concurrency must be at least 1.' + } + if ([string]::IsNullOrWhiteSpace([string]$profile.configuration_profile) -or [string]::IsNullOrWhiteSpace([string]$profile.tool_profile)) { + throw 'execution-profile.json must declare configuration_profile and tool_profile.' + } + + $runnerValue = [string](Get-JsonProperty -Object $profile -Name 'runner' -Default '') + $modelValue = [string](Get-JsonProperty -Object $profile -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($runnerValue) -and $runnerValue -notmatch '^[a-z0-9][a-z0-9-]*$') { + throw 'execution-profile.json runner must be a safe lowercase runner name.' + } + if ([string]::IsNullOrWhiteSpace($runnerValue) -or [string]::IsNullOrWhiteSpace($modelValue)) { + throw 'execution-profile.json must declare non-empty runner and model before a runner can execute.' + } + return [pscustomobject]@{ + Path = $resolvedProfilePath + Profile = $profile + Hash = Get-Sha256HexFromFile -Path $resolvedProfilePath + Runner = if ([string]::IsNullOrWhiteSpace($runnerValue)) { $null } else { $runnerValue } + Model = if ([string]::IsNullOrWhiteSpace($modelValue)) { $null } else { $modelValue } + ReasoningEffort = if ([string]::IsNullOrWhiteSpace([string]$profile.reasoning_effort)) { $null } else { [string]$profile.reasoning_effort } + ConfigurationProfile = [string]$profile.configuration_profile + ToolProfile = [string]$profile.tool_profile + TimeoutSeconds = $timeout + Concurrency = $concurrency + } +} + +function Assert-RunnerDescriptor { + param([Parameter(Mandatory = $true)][object]$Descriptor) + + $schemas = Get-RunnerSchemaNames + if ([string]$Descriptor.schema -ne $schemas.Descriptor) { + throw "Runner descriptor must declare '$($schemas.Descriptor)'." + } + if ([string]$Descriptor.protocol_version -ne $schemas.Protocol) { + throw "Runner descriptor protocol_version must be '$($schemas.Protocol)'." + } + foreach ($field in @('name', 'version', 'platforms', 'harness', 'capabilities', 'configuration_profiles', 'tool_profiles')) { + if (-not (Test-JsonProperty -Object $Descriptor -Name $field)) { + throw "Runner descriptor is missing '$field'." + } + } + if ([string]::IsNullOrWhiteSpace([string]$Descriptor.name) -or [string]::IsNullOrWhiteSpace([string]$Descriptor.version)) { + throw 'Runner descriptor name and version must be non-empty.' + } + if (-not (Test-JsonProperty -Object $Descriptor.harness -Name 'name') -or -not (Test-JsonProperty -Object $Descriptor.harness -Name 'version')) { + throw 'Runner descriptor harness must declare name and version.' + } + + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $Descriptor.capabilities)) { + $capabilityValue = Get-JsonProperty -Object $Descriptor.capabilities -Name $capabilityName + if ([string]$capabilityValue -notin @('supported', 'conditional', 'unsupported')) { + throw "Runner capability '$capabilityName' must be supported, conditional, or unsupported." + } + } + + $required = @( + 'fresh_context', + 'isolated_home_config', + 'isolated_working_directory', + 'filesystem_confinement', + 'ambient_candidate_skill_exclusion', + 'candidate_skill_exposure', + 'prompt_fidelity', + 'model_configuration_lock', + 'response_capture' + ) + foreach ($name in $required) { + if (-not (Test-JsonProperty -Object $Descriptor.capabilities -Name $name)) { + throw "Runner descriptor is missing required capability '$name'." + } + } + + return $true +} + +function New-PreflightCheck { + param( + [Parameter(Mandatory = $true)][string]$Name, + [Parameter(Mandatory = $true)][ValidateSet('passed', 'failed', 'unavailable', 'not_applicable')][string]$Status, + [Parameter(Mandatory = $true)][string]$Detail + ) + + return [ordered]@{ name = $Name; status = $Status; detail = $Detail } +} + +function New-PreflightDocument { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][bool]$Compatible, + [object[]]$Checks = @(), + [string[]]$Mechanisms = @(), + [object]$ResolvedCapabilities = $null, + [string[]]$Warnings = @(), + [string[]]$Reasons = @() + ) + + $schemas = Get-RunnerSchemaNames + $capabilitiesForAssessment = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + $assessment = Get-IsolationCapabilityAssessment -Capabilities $capabilitiesForAssessment + $effectiveCompatible = $Compatible -and $assessment.MandatoryProven + $unprovenControls = [string[]]$assessment.Unproven + if (-not $effectiveCompatible) { $unprovenControls = [string[]](@($assessment.Unproven) + @('preflight')) } + return [ordered]@{ + schema = $schemas.Preflight + protocol_version = $schemas.Protocol + status = if ($effectiveCompatible) { 'compatible' } else { 'incompatible' } + runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } + harness = $Descriptor.harness + run = [ordered]@{ eval_id = $Run.EvalId; eval_name = $Run.EvalName; configuration = $Run.Mode } + requested = [ordered]@{ + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + timeout_seconds = $Profile.TimeoutSeconds + } + checks = @($Checks) + resolved_capabilities = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + isolation = [ordered]@{ + level = if ($effectiveCompatible) { $assessment.Level } else { 'unsupported' } + status = if ($effectiveCompatible) { 'verified' } else { 'unverified' } + hard_filesystem_confinement = if ($effectiveCompatible) { $assessment.HardFilesystemConfinement } else { $false } + unproven_controls = $unprovenControls + } + mechanisms = @($Mechanisms) + warnings = @($Warnings) + reasons = @($Reasons) + } +} + +function New-UnavailableMetric { + param([Parameter(Mandatory = $true)][string]$Reason) + + return [ordered]@{ status = 'unavailable'; reason = $Reason } +} + +function New-AvailableMetric { + param([Parameter(Mandatory = $true)][object]$Value) + + return [ordered]@{ status = 'available'; value = $Value } +} + +function New-ExecutionFailure { + param( + [Parameter(Mandatory = $true)][string]$Code, + [Parameter(Mandatory = $true)][string]$Message + ) + + return [ordered]@{ code = $Code; message = $Message } +} + +function New-ExecutionResult { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][ValidateSet('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')][string]$Status, + [string]$FinalResponse, + [string]$FinalResponseReason, + [string]$StartedUtc, + [string]$FinishedUtc, + [double]$DurationSeconds = 0, + [Nullable[int]]$ExitStatus, + [object]$Failure, + [string]$SessionId, + [System.Collections.IDictionary]$IsolationCapabilities, + [string[]]$IsolationMechanisms = @(), + [object]$ResolvedConfiguration = $null, + [object]$Telemetry = $null, + [object[]]$Artifacts = @(), + [string[]]$Warnings = @(), + [string[]]$CompatibilityDeviations = @(), + [object]$Evidence = $null, + [int]$AttemptCount = 1 + ) + + $schemas = Get-RunnerSchemaNames + $assessment = Get-IsolationCapabilityAssessment -Capabilities $IsolationCapabilities + $effectiveStatus = $Status + $effectiveFinalResponse = $FinalResponse + $effectiveFinalResponseReason = $FinalResponseReason + $effectiveExitStatus = $ExitStatus + $effectiveFailure = $Failure + $effectiveDeviations = [System.Collections.Generic.List[string]]::new() + foreach ($deviation in @($CompatibilityDeviations)) { $effectiveDeviations.Add([string]$deviation) } + if ($Status -ne 'incompatible' -and -not $assessment.MandatoryProven) { + $effectiveStatus = 'incompatible' + $effectiveFinalResponse = $null + $effectiveFinalResponseReason = 'isolation_controls_unproven' + $effectiveExitStatus = $null + $effectiveFailure = New-ExecutionFailure -Code 'isolation_unproven' -Message ("Mandatory isolation controls were not proven: {0}." -f ([string]::Join(', ', @($assessment.Unproven)))) + $effectiveDeviations.Add('execution_rejected_because_mandatory_isolation_controls_were_unproven') + } + $hasResponse = -not [string]::IsNullOrWhiteSpace($effectiveFinalResponse) + $started = if ([string]::IsNullOrWhiteSpace($StartedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($StartedUtc).ToUniversalTime() } + $finished = if ([string]::IsNullOrWhiteSpace($FinishedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($FinishedUtc).ToUniversalTime() } + $isolation = [ordered]@{ + status = if ($effectiveStatus -ne 'incompatible' -and $assessment.MandatoryProven) { 'verified' } else { 'unverified' } + level = if ($effectiveStatus -eq 'incompatible') { 'unsupported' } else { $assessment.Level } + hard_filesystem_confinement = if ($effectiveStatus -eq 'incompatible') { $false } else { $assessment.HardFilesystemConfinement } + capabilities = if ($null -eq $IsolationCapabilities) { [ordered]@{} } else { $IsolationCapabilities } + mechanisms = @($IsolationMechanisms) + required_controls = @($assessment.Required) + unproven_controls = [string[]]$assessment.Unproven + } + + $resolved = [ordered]@{ + model = $null + reasoning_effort = $null + configuration_profile = $null + tool_profile = $null + status = 'unavailable' + reason = 'harness_only_confirmed_the_requested_configuration' + accepted = [ordered]@{ + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + } + } + if ($null -ne $ResolvedConfiguration) { + $resolved.status = [string](Get-JsonProperty -Object $ResolvedConfiguration -Name 'status' -Default 'resolved') + $resolved.reason = Get-JsonProperty -Object $ResolvedConfiguration -Name 'reason' -Default $null + foreach ($name in @('model', 'reasoning_effort', 'configuration_profile', 'tool_profile')) { + $value = Get-JsonProperty -Object $ResolvedConfiguration -Name $name -Default $null + if ($null -ne $value) { $resolved[$name] = $value } + } + $observations = Get-JsonProperty -Object $ResolvedConfiguration -Name 'observations' -Default $null + if ($null -ne $observations) { $resolved.observations = $observations } + } + + return [ordered]@{ + schema = $schemas.Result + protocol_version = $schemas.Protocol + run_id = [Guid]::NewGuid().ToString('D') + session = [ordered]@{ + id = if ([string]::IsNullOrWhiteSpace($SessionId)) { [Guid]::NewGuid().ToString('D') } else { $SessionId } + fresh = $true + resumed = $false + } + status = $effectiveStatus + run = [ordered]@{ + eval_id = $Run.EvalId + eval_name = $Run.EvalName + configuration = $Run.Mode + } + final_response = if ($hasResponse) { + [ordered]@{ status = 'available'; text = $effectiveFinalResponse } + } else { + [ordered]@{ status = 'unavailable'; reason = if ([string]::IsNullOrWhiteSpace($effectiveFinalResponseReason)) { 'harness_did_not_return_a_final_response' } else { $effectiveFinalResponseReason } } + } + runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } + harness = $Descriptor.harness + requested = [ordered]@{ + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + } + resolved = $resolved + started_utc = $started.ToString('yyyy-MM-ddTHH:mm:ss.fffZ') + finished_utc = $finished.ToString('yyyy-MM-ddTHH:mm:ss.fffZ') + duration_seconds = [Math]::Max(0, [Math]::Round($DurationSeconds, 3)) + exit = [ordered]@{ status = $effectiveExitStatus; failure = $effectiveFailure } + input = [ordered]@{ + prompt_sha256 = $Run.PromptHash + run_json_sha256 = Get-Sha256HexFromFile -Path $Run.RunPath + profile_sha256 = $Profile.Hash + } + isolation = $isolation + telemetry = if ($null -eq $Telemetry) { + [ordered]@{ + transcript = New-UnavailableMetric -Reason 'harness_did_not_expose_transcript' + tokens = New-UnavailableMetric -Reason 'harness_did_not_expose_usage' + tool_calls = New-UnavailableMetric -Reason 'harness_did_not_expose_tool_calls' + cost = New-UnavailableMetric -Reason 'harness_did_not_expose_cost' + } + } else { $Telemetry } + evidence = if ($null -eq $Evidence) { [ordered]@{} } else { $Evidence } + artifacts = @($Artifacts) + warnings = @($Warnings) + compatibility_deviations = @($effectiveDeviations) + attempt_count = $AttemptCount + } +} + +function Assert-ExecutionResult { + param([Parameter(Mandatory = $true)][object]$Result) + + $schemas = Get-RunnerSchemaNames + if ([string]$Result.schema -ne $schemas.Result) { + throw "execution-result.json must declare '$($schemas.Result)'." + } + if ([string]$Result.protocol_version -ne $schemas.Protocol) { + throw "execution-result.json protocol_version must be '$($schemas.Protocol)'." + } + if ([string]$Result.status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "execution-result.json status '$($Result.status)' is unsupported." + } + foreach ($field in @('run_id', 'runner', 'harness', 'requested', 'resolved', 'started_utc', 'finished_utc', 'duration_seconds', 'exit', 'final_response', 'input', 'isolation', 'telemetry', 'evidence', 'artifacts', 'warnings')) { + if (-not (Test-JsonProperty -Object $Result -Name $field)) { + throw "execution-result.json is missing '$field'." + } + } + if ([string]::IsNullOrWhiteSpace([string]$Result.run_id)) { + throw 'execution-result.json run_id must be non-empty.' + } + if (-not (Test-JsonProperty -Object $Result.session -Name 'id') -or + -not [bool](Get-JsonProperty -Object $Result.session -Name 'fresh' -Default $false) -or + [bool](Get-JsonProperty -Object $Result.session -Name 'resumed' -Default $true)) { + throw 'execution-result.json must identify a fresh, non-resumed session.' + } + if ([int](Get-JsonProperty -Object $Result -Name 'attempt_count' -Default 0) -ne 1) { + throw 'execution-result.json attempt_count must be exactly 1; quality retries are not allowed.' + } + $isolationStatus = [string](Get-JsonProperty -Object $Result.isolation -Name 'status' -Default '') + $isolationLevel = [string](Get-JsonProperty -Object $Result.isolation -Name 'level' -Default '') + $hardFilesystem = [bool](Get-JsonProperty -Object $Result.isolation -Name 'hard_filesystem_confinement' -Default $false) + if ($isolationStatus -notin @('verified', 'unverified')) { + throw "execution-result.json isolation.status '$isolationStatus' is unsupported." + } + if ($isolationLevel -notin @('strict', 'pragmatic', 'unsupported')) { + throw "execution-result.json isolation.level '$isolationLevel' is unsupported." + } + if ($Result.status -eq 'incompatible') { + if ($isolationStatus -ne 'unverified' -or $isolationLevel -ne 'unsupported') { + throw 'An incompatible execution must report unverified, unsupported isolation.' + } + } else { + if ($isolationStatus -ne 'verified' -or $isolationLevel -eq 'unsupported') { + throw 'A non-incompatible execution must prove the mandatory experimental controls.' + } + if ($isolationLevel -eq 'strict' -and -not $hardFilesystem) { + throw 'Strict isolation must report hard filesystem confinement.' + } + if ($isolationLevel -eq 'pragmatic' -and $hardFilesystem) { + throw 'Pragmatic isolation must not claim hard filesystem confinement.' + } + $requiredControls = @('fresh_context', 'isolated_home_config', 'isolated_working_directory', 'ambient_candidate_skill_exclusion', 'candidate_skill_exposure', 'prompt_fidelity', 'model_configuration_lock', 'response_capture') + foreach ($control in $requiredControls) { + $value = [string](Get-JsonProperty -Object $Result.isolation.capabilities -Name $control -Default 'unavailable') + if ($control -eq 'candidate_skill_exposure') { + if ($value -notin @('supported', 'excluded')) { throw "Mandatory isolation capability '$control' is not proven." } + } elseif ($value -ne 'supported') { + throw "Mandatory isolation capability '$control' is not proven." + } + } + } + $resolvedStatus = [string](Get-JsonProperty -Object $Result.resolved -Name 'status' -Default '') + if ($resolvedStatus -notin @('unavailable', 'accepted_request', 'resolved')) { + throw "execution-result.json resolved.status '$resolvedStatus' is unsupported." + } + if (-not (Test-JsonProperty -Object $Result.resolved -Name 'accepted')) { + throw 'execution-result.json resolved must preserve the requested configuration as accepted evidence.' + } + $hasPortableProvider = + (Test-JsonProperty -Object $Result.requested -Name 'provider') -or + (Test-JsonProperty -Object $Result.resolved -Name 'provider') -or + (Test-JsonProperty -Object $Result.resolved.accepted -Name 'provider') + if ($hasPortableProvider) { + throw 'execution-result.json must not expose provider in portable requested/resolved configuration fields.' + } + foreach ($hashField in @('prompt_sha256', 'run_json_sha256', 'profile_sha256')) { + if (-not (Test-Sha256 -Value ([string]$Result.input.$hashField))) { + throw "execution-result.json input.$hashField must be a SHA-256 value." + } + } + if ([double]$Result.duration_seconds -lt 0) { + throw 'execution-result.json duration_seconds must not be negative.' + } + $responseStatus = [string]$Result.final_response.status + if ($responseStatus -eq 'available') { + if (-not (Test-JsonProperty -Object $Result.final_response -Name 'text')) { + throw 'Available final_response must contain text.' + } + } elseif ($responseStatus -eq 'unavailable') { + if ([string]::IsNullOrWhiteSpace([string]$Result.final_response.reason)) { + throw 'Unavailable final_response must contain a reason.' + } + } else { + throw "final_response status '$responseStatus' is unsupported." + } + + foreach ($metricName in @(Get-JsonPropertyNames -Object $Result.telemetry)) { + $metric = Get-JsonProperty -Object $Result.telemetry -Name $metricName + $status = [string](Get-JsonProperty -Object $metric -Name 'status' -Default '') + if ($status -notin @('available', 'unavailable')) { + throw "Telemetry '$metricName' must declare available or unavailable status." + } + if ($status -eq 'unavailable' -and [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $metric -Name 'reason' -Default ''))) { + throw "Unavailable telemetry '$metricName' must declare a reason." + } + } + + foreach ($artifact in @($Result.artifacts)) { + $path = [string](Get-JsonProperty -Object $artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $artifact -Name 'scope' -Default '') + Assert-SafeRelativePath -RelativePath $path -FieldName 'artifact.path' + if ($scope -notin @('run', 'package')) { + throw "artifact.scope '$scope' must be run or package." + } + if (-not (Test-Sha256 -Value ([string]$artifact.sha256))) { + throw 'artifact.sha256 must be a SHA-256 value.' + } + if ([int64]$artifact.size -lt 0 -or [string]::IsNullOrWhiteSpace([string]$artifact.media_type)) { + throw 'artifact must declare non-negative size and media_type.' + } + } + + return $true +} + +function New-RunnerEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Run, + [string[]]$AuthenticationVariables = @(), + [hashtable]$Additional = @{} + ) + + $environment = [ordered]@{} + foreach ($name in @('PATH', 'SystemRoot', 'WINDIR', 'ComSpec', 'PATHEXT', 'LANG', 'LC_ALL', 'TZ', 'SSL_CERT_FILE', 'NODE_PATH')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + + $tempPath = Join-Path $Run.HomeDirectoryPath 'tmp' + foreach ($directory in @($Run.HomeDirectoryPath, $tempPath, (Join-Path $Run.HomeDirectoryPath '.config'), (Join-Path $Run.HomeDirectoryPath '.local/share'), (Join-Path $Run.HomeDirectoryPath '.cache'))) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + + $environment['HOME'] = $Run.HomeDirectoryPath + $environment['USERPROFILE'] = $Run.HomeDirectoryPath + $environment['XDG_CONFIG_HOME'] = Join-Path $Run.HomeDirectoryPath '.config' + $environment['XDG_DATA_HOME'] = Join-Path $Run.HomeDirectoryPath '.local/share' + $environment['XDG_CACHE_HOME'] = Join-Path $Run.HomeDirectoryPath '.cache' + $environment['TEMP'] = $tempPath + $environment['TMP'] = $tempPath + $environment['CI'] = '1' + $environment['NO_COLOR'] = '1' + + foreach ($name in $AuthenticationVariables | Sort-Object -Unique) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + foreach ($key in $Additional.Keys) { + $environment[$key] = [string]$Additional[$key] + } + + return $environment +} + +function Get-LinuxEvalSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$InsideEnvironment, + [string[]]$ReadOnlyRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt') + ) + + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $arguments.Add($argument) } + foreach ($path in $ReadOnlyRoots) { + if (Test-Path -LiteralPath $path -PathType Container) { + $arguments.Add('--ro-bind'); $arguments.Add($path); $arguments.Add($path) + } + } + $arguments.Add('--proc'); $arguments.Add('/proc') + $arguments.Add('--dev'); $arguments.Add('/dev') + $arguments.Add('--tmpfs'); $arguments.Add('/tmp') + $arguments.Add('--bind'); $arguments.Add($Inputs.Run.RunRoot); $arguments.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $arguments.Add('--ro-bind'); $arguments.Add($commandDirectory); $arguments.Add($commandDirectory) + } + } + $arguments.Add('--chdir'); $arguments.Add('/run/repo') + foreach ($key in @($InsideEnvironment.Keys)) { + $arguments.Add('--setenv'); $arguments.Add([string]$key); $arguments.Add([string]$InsideEnvironment[$key]) + } + $arguments.Add('--') + $arguments.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $arguments.Add($prefix) } + return @($arguments) +} + +function New-MacosEvalSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [string[]]$ReadOnlyRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db') + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'eval-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in @($ReadOnlyRoots + @($commandDirectory)) | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('\', '/').Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Resolve-ExternalCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + $command = Get-Command $Name -ErrorAction SilentlyContinue + if ($null -eq $command) { + foreach ($candidateName in @("$Name.ps1", "$Name.cmd", "$Name.exe")) { + $command = Get-Command $candidateName -ErrorAction SilentlyContinue + if ($null -ne $command) { break } + } + } + if ($null -eq $command) { + return $null + } + + $source = [string]$command.Source + $extension = [System.IO.Path]::GetExtension($source).ToLowerInvariant() + if ($extension -eq '.ps1') { + $pwsh = Get-Command pwsh -ErrorAction SilentlyContinue + if ($null -eq $pwsh) { + return $null + } + return [pscustomobject]@{ FileName = [string]$pwsh.Source; Prefix = @('-NoProfile', '-File', $source); Source = $source } + } + + return [pscustomobject]@{ FileName = $source; Prefix = @(); Source = $source } +} + +function Invoke-RunnerProcess { + param( + [Parameter(Mandatory = $true)][string]$FileName, + [string[]]$ArgumentList = @(), + [Parameter(Mandatory = $true)][string]$WorkingDirectory, + [System.Collections.IDictionary]$Environment = @{}, + [AllowEmptyCollection()][byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 900 + ) + + $start = [DateTime]::UtcNow + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = $FileName + $startInfo.WorkingDirectory = $WorkingDirectory + $startInfo.UseShellExecute = $false + $startInfo.CreateNoWindow = $true + $startInfo.RedirectStandardInput = $true + $startInfo.RedirectStandardOutput = $true + $startInfo.RedirectStandardError = $true + foreach ($argument in @($ArgumentList)) { + [void]$startInfo.ArgumentList.Add([string]$argument) + } + $startInfo.Environment.Clear() + foreach ($key in $Environment.Keys) { + $startInfo.Environment[$key] = [string]$Environment[$key] + } + + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $startInfo + try { + if (-not $process.Start()) { + throw "Could not start '$FileName'." + } + + if ($null -ne $InputBytes -and $InputBytes.Length -gt 0) { + $process.StandardInput.BaseStream.Write($InputBytes, 0, $InputBytes.Length) + } + $process.StandardInput.Close() + $stdoutTask = $process.StandardOutput.ReadToEndAsync() + $stderrTask = $process.StandardError.ReadToEndAsync() + $timeoutMilliseconds = [Math]::Min([int64]::MaxValue, [int64]$TimeoutSeconds * 1000) + $exited = $process.WaitForExit([int]([Math]::Min($timeoutMilliseconds, [int]::MaxValue))) + $timedOut = -not $exited + if ($timedOut) { + try { $process.Kill($true) } catch { } + $process.WaitForExit() + } + $stdout = $stdoutTask.GetAwaiter().GetResult() + $stderr = $stderrTask.GetAwaiter().GetResult() + $finish = [DateTime]::UtcNow + + return [pscustomobject]@{ + ExitCode = if ($timedOut) { $null } else { $process.ExitCode } + TimedOut = $timedOut + Stdout = $stdout + Stderr = $stderr + StartedUtc = $start + FinishedUtc = $finish + DurationSeconds = [Math]::Round(($finish - $start).TotalSeconds, 3) + } + } finally { + $process.Dispose() + } +} + +function Get-ProviderAuthenticationVariables { + param([string]$Provider) + + $normalized = ([string]$Provider).ToLowerInvariant() + switch -Regex ($normalized) { + '^openai$|^chatgpt$' { return @('OPENAI_API_KEY') } + '^anthropic$' { return @('ANTHROPIC_API_KEY') } + '^google$|^google-vertex$|^gemini$' { return @('GOOGLE_API_KEY', 'GEMINI_API_KEY') } + '^openrouter$' { return @('OPENROUTER_API_KEY') } + '^xai$|^x-ai$' { return @('XAI_API_KEY') } + '^mistral$' { return @('MISTRAL_API_KEY') } + '^cline$' { return @('CLINE_API_KEY') } + default { return @() } + } +} + +function Test-EnvironmentVariablePresent { + param([string[]]$Names) + + foreach ($name in @($Names)) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $true + } + } + return $false +} + +function New-ArtifactReference { + param( + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][string]$Path, + [ValidateSet('run', 'package')][string]$Scope = 'run', + [string]$MediaType = 'application/octet-stream' + ) + + Assert-SafeRelativePath -RelativePath $Path -FieldName 'artifact.path' + $base = if ($Scope -eq 'run') { $Run.RunRoot } else { Split-Path -Parent (Split-Path -Parent $Run.RunRoot) } + $full = Resolve-ContainedPath -BasePath $base -RelativePath $Path -FieldName 'artifact.path' -Kind File + return [ordered]@{ + path = $Path.Replace('\', '/') + scope = $Scope + sha256 = Get-Sha256HexFromFile -Path $full + size = (Get-Item -LiteralPath $full).Length + media_type = $MediaType + } +} + +function Get-MediaType { + param([string]$Path) + + switch ([System.IO.Path]::GetExtension($Path).ToLowerInvariant()) { + '.jsonl' { return 'application/x-ndjson' } + '.json' { return 'application/json' } + '.txt' { return 'text/plain; charset=utf-8' } + '.md' { return 'text/markdown; charset=utf-8' } + default { return 'application/octet-stream' } + } +} + +function ConvertFrom-JsonLines { + param([Parameter(Mandatory = $true)][string]$Text) + + $events = [System.Collections.Generic.List[object]]::new() + $errors = [System.Collections.Generic.List[string]]::new() + $lineNumber = 0 + foreach ($line in ($Text -split "`r?`n")) { + $lineNumber++ + if ([string]::IsNullOrWhiteSpace($line)) { continue } + try { + $events.Add(($line | ConvertFrom-Json)) + } catch { + $errors.Add("line ${lineNumber}: $($_.Exception.Message)") + } + } + return [pscustomobject]@{ Events = @($events); Errors = @($errors) } +} + +function Get-OutputTextFromFinalResponse { + param([object]$Result) + + if ([string](Get-JsonProperty -Object $Result.final_response -Name 'status' -Default '') -eq 'available') { + return [string]$Result.final_response.text + } + return $null +} diff --git a/scripts/eval-runners/tests/fixtures/cline-events.jsonl b/scripts/eval-runners/tests/fixtures/cline-events.jsonl new file mode 100644 index 0000000..21d2a65 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/cline-events.jsonl @@ -0,0 +1,11 @@ +{"type":"hook_event","hookEventName":"agent_start","sessionId":"fixture-session"} +{"type":"say","say":"text","text":"2 + 2 = 4.","partial":false} +{"type":"say","say":"tool","name":"read_file","text":"fixture.md","partial":false} +{"type":"say","say":"completion_result","text":"2 + 2 = 4.","partial":false} +{"type":"agent_event","event":{"type":"iteration_start","iteration":1}} +{"type":"agent_event","event":{"type":"content_start","contentType":"text","text":"2 + 2 = "}} +{"type":"agent_event","event":{"type":"content_end","contentType":"text","text":"2 + 2 = 4."}} +{"type":"agent_event","event":{"type":"usage","inputTokens":5022,"outputTokens":28,"totalTokens":5050,"cost":0.01}} +{"type":"agent_event","event":{"type":"done","reason":"completed","text":"2 + 2 = 4.","sessionId":"fixture-session","iterations":1,"usage":{"inputTokens":5022,"outputTokens":28,"totalTokens":5050,"cost":0.01}}} +{"type":"future.event.v99","payload":"preserve-as-warning"} +{"type":"hook_event","hookEventName":"agent_end","sessionId":"fixture-session"} diff --git a/scripts/eval-runners/tests/fixtures/codex-events.jsonl b/scripts/eval-runners/tests/fixtures/codex-events.jsonl new file mode 100644 index 0000000..3a3399f --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/codex-events.jsonl @@ -0,0 +1,5 @@ +{"type":"thread.started","thread_id":"fixture-thread"} +{"type":"turn.started"} +{"type":"item.completed","item":{"type":"agent_message","text":"fixture response"}} +{"type":"turn.completed","usage":{"input_tokens":12,"cached_input_tokens":3,"output_tokens":4}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/fixtures/copilot-events.jsonl b/scripts/eval-runners/tests/fixtures/copilot-events.jsonl new file mode 100644 index 0000000..e3b5a1a --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/copilot-events.jsonl @@ -0,0 +1,9 @@ +{"type":"session.start","id":"e1","parentId":null,"timestamp":"2026-01-01T00:00:00.000Z","data":{"sessionId":"fixture-session"}} +{"type":"user.message","id":"e2","parentId":"e1","timestamp":"2026-01-01T00:00:00.100Z","data":{"content":"fixture prompt"}} +{"type":"assistant.message","id":"e3","parentId":"e2","timestamp":"2026-01-01T00:00:01.000Z","data":{"messageId":"m1","model":"claude-haiku-4.5","content":"fixture progress note"}} +{"type":"tool.execution_start","id":"e4","parentId":"e3","timestamp":"2026-01-01T00:00:01.200Z","data":{"callId":"t1","toolName":"str_replace_editor"}} +{"type":"tool.execution_complete","id":"e5","parentId":"e4","timestamp":"2026-01-01T00:00:01.800Z","data":{"callId":"t1","status":"success"}} +{"type":"assistant.message","id":"e6","parentId":"e5","timestamp":"2026-01-01T00:00:02.000Z","data":{"messageId":"m2","model":"claude-haiku-4.5","content":"fixture final response"}} +{"type":"assistant.usage","id":"e7","parentId":"e6","ephemeral":true,"timestamp":"2026-01-01T00:00:02.100Z","data":{"model":"claude-haiku-4.5","inputTokens":12,"outputTokens":4,"cacheReadTokens":3,"numToolCalls":1,"cost":0.2,"finishReason":"stop"}} +{"type":"session.task_complete","id":"e8","parentId":"e7","timestamp":"2026-01-01T00:00:02.200Z","data":{}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/fixtures/opencode-events.jsonl b/scripts/eval-runners/tests/fixtures/opencode-events.jsonl new file mode 100644 index 0000000..d0a705a --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/opencode-events.jsonl @@ -0,0 +1,5 @@ +{"type":"step_start","part":{"id":"fixture-step"}} +{"type":"text","part":{"text":"fixture response"}} +{"type":"tool_use","part":{"tool":"read"}} +{"type":"step_finish","part":{"tokens":{"input":12,"output":4},"cost":0.01}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 new file mode 100644 index 0000000..d7c0553 --- /dev/null +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -0,0 +1,877 @@ +<#! +.SYNOPSIS + Deterministic conformance suite for the common Eval Runner protocol. + +.DESCRIPTION + Creates ephemeral packages under the system temp directory, invokes the + deterministic fake runner and recorded fake CLI processes, and checks the + contracts and recorded event fixtures. It never invokes a real harness or + a live model. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..\..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Invoke-AdapterJson { + param( + [Parameter(Mandatory = $true)][string]$RunnerPath, + [Parameter(Mandatory = $true)][string]$Command, + [Parameter(Mandatory = $true)][string]$RunPath, + [Parameter(Mandatory = $true)][string]$ProfilePath + ) + + $output = & pwsh -NoProfile -File $RunnerPath $Command -Run $RunPath -Profile $ProfilePath + if ($LASTEXITCODE -ne 0) { throw "Recorded runner '$Command' failed for '$RunnerPath': $([string]::Join(' ', @($output)))" } + $json = [string]::Join([Environment]::NewLine, @($output)) + if ([string]::IsNullOrWhiteSpace($json)) { throw "Recorded runner '$Command' returned no JSON for '$RunnerPath'." } + return $json | ConvertFrom-Json +} + +function Invoke-RecordedRunnerTests { + $recordedRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-recorded-runner-' + [Guid]::NewGuid().ToString('N')) +$recordedOldPath = $env:PATH +$recordedOldOpenAi = $env:OPENAI_API_KEY +$recordedOldCodexHome = $env:CODEX_HOME +$recordedOldGlobalSecret = $env:AGENTIC_GLOBAL_SECRET +$recordedOldProjectDisable = $env:OPENCODE_DISABLE_PROJECT_CONFIG +$recordedOldCopilotToken = $env:COPILOT_GITHUB_TOKEN +$recordedOldGhToken = $env:GH_TOKEN +$recordedOldGithubToken = $env:GITHUB_TOKEN +$recordedOldCopilotHome = $env:COPILOT_HOME +$recordedOldGhConfigDir = $env:GH_CONFIG_DIR +try { + $fakeBin = Join-Path $recordedRoot 'bin' + New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null + $recordedIteration = Join-Path $recordedRoot 'iteration-1' + New-Item -ItemType Directory -Path $recordedIteration -Force | Out-Null + $with = New-TestRun -IterationDirectory $recordedIteration -Configuration with_skill + $without = New-TestRun -IterationDirectory $recordedIteration -Configuration without_skill + [System.IO.File]::WriteAllText((Join-Path $with.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $without.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) + $fakeCli = @' +[CmdletBinding()] +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +$harness = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Path) +$logPath = Join-Path (Get-Location).Path ("{0}-fake-cli-log.jsonl" -f $harness) +$arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) +$authNames = @('OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'GEMINI_API_KEY', 'OPENROUTER_API_KEY', 'XAI_API_KEY', 'MISTRAL_API_KEY', 'CLINE_API_KEY') +$authPresent = @($authNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) +$copilotAuthNames = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') +$copilotAuthPresent = @($copilotAuthNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) +$copilotHome = [Environment]::GetEnvironmentVariable('COPILOT_HOME') +$repositoryAgentsPath = Join-Path (Get-Location).Path 'AGENTS.md' +$repositoryCopilotInstructionsPath = Join-Path (Get-Location).Path '.github\copilot-instructions.md' +$candidateSkillPath = Join-Path (Split-Path -Parent (Get-Location).Path) 'skill' +$repositoryInstructionMarkerVisible = $false +if (Test-Path -LiteralPath $repositoryCopilotInstructionsPath -PathType Leaf) { + $repositoryInstructionMarkerVisible = [IO.File]::ReadAllText($repositoryCopilotInstructionsPath, [Text.UTF8Encoding]::new($false)).Contains('repo-owned-copilot-instruction') +} +$copilotAuthenticationSource = if ($copilotAuthPresent.Count -gt 0) { + 'explicit_environment' +} elseif (-not [string]::IsNullOrWhiteSpace($copilotHome) -and (Test-Path -LiteralPath (Join-Path $copilotHome 'fixture-os-keychain-available') -PathType Leaf)) { + 'os_keychain' +} elseif (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) -and (Test-Path -LiteralPath ([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) -PathType Container)) { + 'github_cli' +} else { + 'unavailable' +} +$record = [ordered]@{ + args = $arguments + working_directory = (Get-Location).Path + home = [Environment]::GetEnvironmentVariable('HOME') + userprofile = [Environment]::GetEnvironmentVariable('USERPROFILE') + auth_names_present = $authPresent + copilot_auth_names_present = $copilotAuthPresent + copilot_authentication_source = $copilotAuthenticationSource + unrelated_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('AGENTIC_GLOBAL_SECRET')) + disable_project_config_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('OPENCODE_DISABLE_PROJECT_CONFIG')) + project_config_visible = Test-Path -LiteralPath (Join-Path (Get-Location).Path 'opencode.json') -PathType Leaf + stdin_received = $false + prompt_via_arg = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count -gt 0 + prompt_arg_count = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count + copilot_home = $copilotHome + copilot_cache_home = [Environment]::GetEnvironmentVariable('COPILOT_CACHE_HOME') + gh_config_dir = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + custom_instructions_disabled = ($arguments -contains '--no-custom-instructions') + builtin_mcps_disabled = ($arguments -contains '--disable-builtin-mcps') + repository_agents_visible = Test-Path -LiteralPath $repositoryAgentsPath -PathType Leaf + repository_copilot_instructions_visible = Test-Path -LiteralPath $repositoryCopilotInstructionsPath -PathType Leaf + repository_instruction_marker_visible = $repositoryInstructionMarkerVisible + candidate_skill_staged = Test-Path -LiteralPath $candidateSkillPath -PathType Container + ambient_copilot_instructions_visible = if ([string]::IsNullOrWhiteSpace($copilotHome)) { $false } else { Test-Path -LiteralPath (Join-Path $copilotHome 'copilot-instructions.md') -PathType Leaf } + secret_env_vars_arg = @($arguments | Where-Object { $_ -like '--secret-env-vars=*' }) +} +if ($arguments -contains '--version') { + $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $version + exit 0 +} +if ($arguments -contains '--help') { + $help = switch ($harness) { + 'codex' { '--ask-for-approval never --ephemeral --ignore-user-config --ignore-rules --json --output-last-message --sandbox --cd --model --config --approve-for-me' } + 'opencode' { '--format --dir --model --auto --pure --continue --session' } + 'copilot' { '--prompt --output-format --model --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all --allow-all-paths --allow-all-urls' } + default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } + } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $help + exit 0 +} +$stdinMemory = [IO.MemoryStream]::new() +[Console]::OpenStandardInput().CopyTo($stdinMemory) +$stdinBytes = $stdinMemory.ToArray() +$expectedPromptPath = Join-Path (Split-Path -Parent (Get-Location).Path) 'prompt.md' +$expectedPromptBytes = if (Test-Path -LiteralPath $expectedPromptPath -PathType Leaf) { [IO.File]::ReadAllBytes($expectedPromptPath) } else { [byte[]]@() } +$stdinExact = $stdinBytes.Length -eq $expectedPromptBytes.Length +if ($stdinExact) { + for ($index = 0; $index -lt $stdinBytes.Length; $index++) { + if ($stdinBytes[$index] -ne $expectedPromptBytes[$index]) { + $stdinExact = $false + break + } + } +} +$stdinHash = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($stdinBytes))).ToLowerInvariant() +$record.stdin_received = $stdinBytes.Length -gt 0 +$record.stdin_delivery_count = if ($stdinBytes.Length -gt 0) { 1 } else { 0 } +$record.stdin_byte_length = $stdinBytes.Length +$record.stdin_sha256 = $stdinHash +$record.stdin_exact = $stdinExact +$record.stdin_expected_sha256 = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($expectedPromptBytes))).ToLowerInvariant() +$record.stdin_utf8_round_trip = ([Text.UTF8Encoding]::new($false, $true).GetString($stdinBytes) -eq [Text.UTF8Encoding]::new($false, $true).GetString($expectedPromptBytes)) +$probeCommand = '$result = [ordered]@{ provider_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENAI_API_KEY'')); copilot_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''COPILOT_GITHUB_TOKEN'')); gh_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''GH_TOKEN'')); github_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''GITHUB_TOKEN'')); auth_file_visible = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable(''HOME'')) ''.codex/auth.json''); global_secret_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''AGENTIC_GLOBAL_SECRET'')); project_disable_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENCODE_DISABLE_PROJECT_CONFIG'')) }; $result | ConvertTo-Json -Compress' +$probeInfo = [Diagnostics.ProcessStartInfo]::new() +$probeInfo.FileName = (Get-Command pwsh).Source +$probeInfo.UseShellExecute = $false +$probeInfo.CreateNoWindow = $true +$probeInfo.RedirectStandardOutput = $true +$probeInfo.RedirectStandardError = $true +$probeInfo.WorkingDirectory = (Get-Location).Path +$probeInfo.ArgumentList.Add('-NoProfile') +$probeInfo.ArgumentList.Add('-Command') +$probeInfo.ArgumentList.Add($probeCommand) +$probeInfo.Environment.Clear() +$probeInfo.Environment['PATH'] = [Environment]::GetEnvironmentVariable('PATH') +$probeInfo.Environment['HOME'] = [Environment]::GetEnvironmentVariable('HOME') +$probeInfo.Environment['USERPROFILE'] = [Environment]::GetEnvironmentVariable('USERPROFILE') +if ($harness -ne 'codex') { $probeInfo.Environment['OPENAI_API_KEY'] = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') } +$probe = [Diagnostics.Process]::new() +$probe.StartInfo = $probeInfo +try { + [void]$probe.Start() + $probeOutput = $probe.StandardOutput.ReadToEnd() + $probeError = $probe.StandardError.ReadToEnd() + $probe.WaitForExit() + if ($probe.ExitCode -ne 0) { throw "worker credential probe failed: $probeError" } + $probeResult = $probeOutput | ConvertFrom-Json + $record.worker_provider_visible = [bool]$probeResult.provider_visible + $record.worker_copilot_token_visible = [bool]$probeResult.copilot_token_visible + $record.worker_gh_token_visible = [bool]$probeResult.gh_token_visible + $record.worker_github_token_visible = [bool]$probeResult.github_token_visible + $record.worker_auth_file_visible = [bool]$probeResult.auth_file_visible + $record.worker_global_secret_visible = [bool]$probeResult.global_secret_visible + $record.worker_project_disable_visible = [bool]$probeResult.project_disable_visible +} finally { + $probe.Dispose() +} +[IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) +if ($harness -eq 'copilot' -and $copilotAuthenticationSource -eq 'unavailable') { + [Console]::Error.WriteLine('deterministic fixture: no Copilot authentication mechanism is available') + exit 17 +} +if ($harness -eq 'codex') { + $outputIndex = [Array]::IndexOf([string[]]$arguments, '--output-last-message') + if ($outputIndex -ge 0 -and $outputIndex + 1 -lt $arguments.Count) { + $outputPath = $arguments[$outputIndex + 1] + $outputParent = Split-Path -Parent $outputPath + if (-not (Test-Path -LiteralPath $outputParent -PathType Container)) { + [Console]::Error.WriteLine("recorded Codex requires the output parent directory to exist: $outputParent") + exit 19 + } + [IO.File]::WriteAllText($outputPath, 'recorded Codex final response', [Text.UTF8Encoding]::new($false)) + } + Write-Output '{"type":"thread.started","thread_id":"recorded-thread"}' + Write-Output '{"type":"item.completed","item":{"type":"agent_message","text":"recorded Codex final response"}}' + Write-Output '{"type":"turn.completed","usage":{"input_tokens":2,"output_tokens":3}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} elseif ($harness -eq 'opencode') { + Write-Output '{"type":"text","text":"recorded OpenCode final response"}' + Write-Output '{"type":"step_finish","part":{"tokens":{"input":2,"output":3},"cost":0.01}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} elseif ($harness -eq 'copilot') { + Write-Output '{"type":"session.start","id":"e1","parentId":null,"data":{"sessionId":"recorded"}}' + Write-Output '{"type":"assistant.message","id":"e2","parentId":"e1","data":{"messageId":"m1","model":"claude-haiku-4.5","content":"recorded Copilot progress"}}' + Write-Output '{"type":"tool.execution_start","id":"e3","parentId":"e2","data":{"callId":"t1","toolName":"str_replace_editor"}}' + Write-Output '{"type":"tool.execution_complete","id":"e4","parentId":"e3","data":{"callId":"t1","status":"success"}}' + Write-Output '{"type":"assistant.message","id":"e5","parentId":"e4","data":{"messageId":"m2","model":"claude-haiku-4.5","content":"recorded Copilot final response"}}' + Write-Output '{"type":"assistant.usage","id":"e6","parentId":"e5","ephemeral":true,"data":{"model":"claude-haiku-4.5","inputTokens":2,"outputTokens":3,"cacheReadTokens":1,"numToolCalls":1,"cost":0.2}}' + Write-Output '{"type":"session.task_complete","id":"e7","parentId":"e6","data":{}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} else { + Write-Output '{"type":"say","say":"text","text":"recorded Cline progress","partial":false}' + Write-Output '{"type":"say","say":"tool","name":"read_file","text":"fixture.md","partial":false}' + Write-Output '{"type":"say","say":"completion_result","text":"recorded Cline final response","partial":false}' + Write-Output '{"type":"say","say":"api_req_finished","text":"{\"inputTokens\":2,\"outputTokens\":3,\"totalTokens\":5}"}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} +'@ + foreach ($harness in @('codex', 'opencode', 'cline', 'copilot')) { + [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) + } + $fakeGh = @' +[CmdletBinding()] +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and $RemainingArguments[1] -eq 'token') { + $config = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if (-not [string]::IsNullOrWhiteSpace($config) -and (Test-Path -LiteralPath (Join-Path $config 'auth-marker.txt') -PathType Leaf)) { + Write-Output 'recorded-gh-fallback-token-not-logged' + exit 0 + } + [Console]::Error.WriteLine('not logged in') + exit 1 +} +[Console]::Error.WriteLine('unsupported gh fixture command') +exit 2 +'@ + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'gh.ps1'), $fakeGh, [System.Text.UTF8Encoding]::new($false)) + $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$recordedOldPath" + $env:OPENAI_API_KEY = 'recorded-canary-not-logged' + $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' + $env:OPENCODE_DISABLE_PROJECT_CONFIG = '1' + $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' + $env:GH_TOKEN = 'recorded-gh-canary-not-logged' + $env:GITHUB_TOKEN = 'recorded-github-canary-not-logged' + $recordedGhConfig = Join-Path $recordedRoot 'github-cli-auth' + New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) + $env:GH_CONFIG_DIR = $recordedGhConfig + $ambientCopilotHome = Join-Path $recordedRoot 'ambient-copilot-home' + New-Item -ItemType Directory -Path $ambientCopilotHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'copilot-instructions.md'), '# ambient-personal-instruction-not-logged', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'config.json'), '{"loggedInUsers":[{"login":"ambient-profile-not-logged"}]}', [System.Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $ambientCopilotHome + $recordedProfiles = [ordered]@{} + foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { + $profilePath = Join-Path $recordedRoot "$runnerName-profile.json" + $profileModel = switch ($runnerName) { + 'copilot' { 'claude-haiku-4.5' } + 'codex' { 'gpt-5.6-luna' } + 'opencode' { 'opencode/muse-spark-1.2-contributor-free' } + 'cline' { 'deepseek/deepseek-v4-flash' } + } + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } + model = $profileModel + reasoning_effort = 'medium' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $recordedProfiles[$runnerName] = $profilePath + } + $resolvedRecordedCodex = Resolve-ExternalCommand -Name 'codex' + Assert-Equal (Join-Path $fakeBin 'codex.ps1') $resolvedRecordedCodex.Source 'recorded Codex command is selected before the installed CLI' + $recordedVersion = Get-ExternalCommandVersion -CommandInfo $resolvedRecordedCodex -WorkingDirectory (Join-Path $with.Root 'repo') + if (-not $recordedVersion.Available) { throw "recorded Codex --version is not observable (exit=$($recordedVersion.Process.ExitCode), timed_out=$($recordedVersion.Process.TimedOut), stdout='$($recordedVersion.Process.Stdout)', stderr='$($recordedVersion.Process.Stderr)')" } + Assert-Equal 'recorded-codex 9.1' $recordedVersion.Version 'recorded Codex exact version helper' + foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { + $runnerDir = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } + $runnerPath = Join-Path $runnerRoot "$runnerDir\runner.ps1" + $description = Invoke-AdapterJson -RunnerPath $runnerPath -Command describe -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + [void](Assert-RunnerDescriptor -Descriptor $description) + $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } + Assert-Equal $expectedVersion $description.harness.version "$runnerName exact describe version" + $preflightWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + $preflightWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $preflightWith.status "$runnerName with_skill pragmatic preflight" + Assert-Equal 'compatible' $preflightWithout.status "$runnerName without_skill pragmatic preflight" + Assert-Equal $expectedVersion $preflightWith.harness.version "$runnerName exact preflight version" + Assert-Equal 'pragmatic' $preflightWith.isolation.level "$runnerName pragmatic preflight level" + if ($runnerName -eq 'copilot') { + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot preflight accepts explicit environment authentication' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq '--allow-all-tools broad tool approval' }).Count -eq 1) 'Copilot preflight describes --allow-all-tools as broad tool approval' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)' }).Count -eq 1) 'Copilot preflight records preserved path and URL verification' + } + if ($runnerName -in @('opencode', 'cline')) { + Assert-True (@($preflightWith.warnings | Where-Object { $_ -match 'child-tool environment filter' }).Count -gt 0) "$runnerName reports the child credential-filter limitation" + } + $resultWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + $resultWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] + foreach ($result in @($resultWith, $resultWithout)) { + [void](Assert-ExecutionResult -Result $result) + Assert-Equal 'completed' $result.status "$runnerName recorded completion" + Assert-Equal $expectedVersion $result.harness.version "$runnerName exact execution version" + Assert-Equal 'accepted_request' $result.resolved.status "$runnerName accepted configuration provenance" + Assert-True ($null -eq $result.resolved.model) "$runnerName does not claim concrete model resolution" + Assert-Equal 'pragmatic' $result.isolation.level "$runnerName pragmatic execution level" + Assert-True (-not $result.isolation.hard_filesystem_confinement) "$runnerName pragmatic execution has no hard confinement" + Assert-Equal 1 $result.attempt_count "$runnerName one semantic attempt" + Assert-Equal 'available' $result.final_response.status "$runnerName captures final response" + $resultRoot = if ($result.run.configuration -eq 'with_skill') { $with.Root } else { $without.Root } + foreach ($artifact in @($result.artifacts)) { + Assert-True ($artifact.path -notmatch '(^|/|\\)\.\.(/|\\|$)') "$runnerName artifact path remains relative" + Assert-True (Test-Path -LiteralPath (Join-Path $resultRoot ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -PathType Leaf) "$runnerName artifact exists inside its run" + } + } + $logPath = Join-Path $with.Root "repo\$runnerName-fake-cli-log.jsonl" + Assert-True (Test-Path -LiteralPath $logPath -PathType Leaf) "$runnerName recorded process log exists" + $records = @(Get-Content -LiteralPath $logPath | ForEach-Object { $_ | ConvertFrom-Json }) + $executionRecords = @($records | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }) + Assert-Equal 1 $executionRecords.Count "$runnerName one execution process per checked arm" + $execution = $executionRecords[0] + Assert-True $execution.stdin_received "$runnerName receives a non-empty stdin prompt" + Assert-Equal 1 $execution.stdin_delivery_count "$runnerName delivers one prompt through stdin" + Assert-True $execution.stdin_exact "$runnerName fake CLI received the exact staged prompt bytes" + Assert-True $execution.stdin_utf8_round_trip "$runnerName preserves arbitrary UTF-8 prompt content" + Assert-True (-not $execution.unrelated_present) "$runnerName does not pass unrelated credential canary" + Assert-True (-not $execution.disable_project_config_present) "$runnerName does not pass ambient project-disable override" + Assert-True (-not $execution.worker_auth_file_visible) "$runnerName worker probe cannot read a copied Codex auth file" + Assert-True (-not $execution.worker_global_secret_visible) "$runnerName worker probe cannot read the parent/global canary" + Assert-True (-not $execution.worker_project_disable_visible) "$runnerName worker probe cannot read the parent project-disable variable" + if ($runnerName -eq 'codex') { + Assert-True (-not $execution.worker_provider_visible) 'Codex shell policy hides the provider API-key variable from the worker probe' + } elseif ($runnerName -eq 'copilot') { + Assert-True (-not $execution.worker_copilot_token_visible) 'Copilot secret COPILOT_GITHUB_TOKEN is unavailable to the worker probe' + Assert-True (-not $execution.worker_gh_token_visible) 'Copilot secret GH_TOKEN is unavailable to the worker probe' + Assert-True (-not $execution.worker_github_token_visible) 'Copilot secret GITHUB_TOKEN is unavailable to the worker probe' + } else { + Assert-True (-not $execution.worker_provider_visible) "$runnerName free-model fixture does not require a provider API key" + } + $args = @($execution.args) + foreach ($forbidden in @('--continue', '--session', '--resume')) { Assert-True ($args -notcontains $forbidden) "$runnerName does not pass '$forbidden'" } + if ($runnerName -eq 'codex') { + Assert-True ($args -contains '--ask-for-approval') 'Codex uses explicit approval policy' + Assert-True ($args -contains 'never') 'Codex approval policy is never' + Assert-True ($args -contains '--sandbox' -and $args -contains 'workspace-write') 'Codex retains workspace-write sandbox' + Assert-True ($args -notcontains '--approve-for-me') 'Codex avoids the conflicting approve-for-me flag' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'gpt-5.6-luna' $args[$modelIndex + 1] 'Codex opaque model selector propagates to the CLI invocation' + $outputIndex = [Array]::IndexOf([string[]]$args, '--output-last-message') + Assert-Equal (Join-Path $with.Root 'evidence\codex-final.txt') $args[$outputIndex + 1] 'Codex output path is host-visible on Windows' + } elseif ($runnerName -eq 'opencode') { + Assert-True ($args -notcontains '--pure') 'OpenCode preserves repository-owned project configuration' + Assert-True ($args -contains '--auto') 'OpenCode is noninteractive' + Assert-True $execution.project_config_visible 'OpenCode paired arm retains repository-owned project configuration' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'opencode/muse-spark-1.2-contributor-free' $args[$modelIndex + 1] 'OpenCode opaque model selector propagates to the CLI invocation' + } elseif ($runnerName -eq 'copilot') { + Assert-True (@($args | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count -eq 0) 'Copilot does not place the prompt in argv' + Assert-Equal 0 $execution.prompt_arg_count 'Copilot has no prompt argument' + Assert-True $execution.stdin_received 'Copilot reads the prompt from stdin' + Assert-True ($args -contains '--output-format' -and $args -contains 'json') 'Copilot uses structured JSONL output' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'claude-haiku-4.5' $args[$modelIndex + 1] 'Copilot reference model claude-haiku-4.5 propagates to the CLI invocation' + Assert-True ($args -contains '--allow-all-tools') 'Copilot grants broad tool approval for noninteractive execution' + Assert-True ($args -contains '--no-ask-user') 'Copilot does not pause for interactive questions' + Assert-True ($args -notcontains '--no-custom-instructions') 'Copilot preserves repository-owned custom instructions' + Assert-True ($args -contains '--disable-builtin-mcps') 'Copilot disables ambient built-in MCP servers' + foreach ($broad in @('--yolo', '--allow-all', '--allow-all-paths', '--allow-all-urls', '--session-id', '--connect', '-r')) { Assert-True ($args -notcontains $broad) "Copilot avoids the over-broad or session option '$broad'" } + Assert-Equal 1 (@($args | Where-Object { $_ -like '--secret-env-vars=*' }).Count) 'Copilot filters protected variables with --secret-env-vars' + Assert-Equal 'COPILOT_GITHUB_TOKEN,GH_TOKEN,GITHUB_TOKEN' ([string]($args | Where-Object { $_ -like '--secret-env-vars=*' }) -replace '^--secret-env-vars=', '') 'Copilot protects every forwarded token variable' + Assert-True (-not $execution.custom_instructions_disabled -and $execution.builtin_mcps_disabled) 'Copilot preserves repository instructions while disabling built-in MCPs' + Assert-True ($execution.repository_agents_visible -and $execution.repository_copilot_instructions_visible -and $execution.repository_instruction_marker_visible) 'Copilot sees staged repository-owned instructions' + Assert-True $execution.candidate_skill_staged 'Copilot with_skill arm retains the staged candidate skill independently of repository instructions' + Assert-True (-not $execution.ambient_copilot_instructions_visible) 'Copilot does not see the ambient personal instruction file' + Assert-Equal 'explicit_environment' $execution.copilot_authentication_source 'Copilot uses explicit environment authentication in the token fixture' + Assert-Equal 3 @($execution.copilot_auth_names_present).Count 'Copilot process receives all protected token variables without logging values' + Assert-True ([string]::IsNullOrWhiteSpace([string]$execution.gh_config_dir)) 'Copilot explicit-token path does not forward host GH_CONFIG_DIR' + Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is run-local' + Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' + Assert-Equal 'stdin' $resultWith.evidence.prompt_delivery 'Copilot result records stdin prompt delivery' + Assert-Equal 'COPILOT_GITHUB_TOKEN' $resultWith.evidence.credential.github_token_variable 'Copilot follows explicit token precedence' + Assert-True (-not $resultWith.evidence.credential.github_cli_config_forwarded) 'Copilot result records that GH_CONFIG_DIR was not forwarded with an explicit token' + Assert-Equal 'supported' $resultWith.isolation.capabilities.credential_child_filtering 'Copilot documents protected child-environment filtering' + Assert-Equal 'shell,mcp' ([string]::Join(',', @($resultWith.evidence.credential.secret_env_var_scope))) 'Copilot evidence names the documented filtering scope' + Assert-Equal 'recorded Copilot final response' $resultWith.final_response.text 'Copilot final response is the last assistant message, not an intermediate one' + Assert-Equal 'claude-haiku-4.5' $resultWith.requested.model 'Copilot requested model is preserved as the Codebelt reference model' + Assert-True ($null -eq $resultWith.resolved.model) 'Copilot does not claim a distinct backend model resolution' + Assert-Equal 'claude-haiku-4.5' $resultWith.evidence.observed_model 'Copilot observed model is captured separately from the requested model' + Assert-Equal 'available' $resultWith.telemetry.tokens.status 'Copilot reports available token telemetry' + Assert-Equal 2 ([int]$resultWith.telemetry.tokens.value.input_tokens) 'Copilot input tokens are parsed from assistant.usage' + Assert-Equal 3 ([int]$resultWith.telemetry.tokens.value.output_tokens) 'Copilot output tokens are parsed from assistant.usage' + Assert-Equal 'available' $resultWith.telemetry.tool_calls.status 'Copilot reports available tool-call telemetry' + Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Copilot parses documented tool.execution events' + Assert-Equal 'unavailable' $resultWith.telemetry.cost.status 'Copilot does not estimate a currency cost' + } else { + $retryIndex = [Array]::IndexOf([string[]]$args, '--retries') + Assert-Equal '0' $args[$retryIndex + 1] 'Cline disables internal retries' + Assert-True ($args -notcontains '--id') 'Cline does not resume a session' + Assert-True ($args -contains '--json') 'Cline uses structured output' + $providerIndex = [Array]::IndexOf([string[]]$args, '--provider') + Assert-Equal 'deepseek' $args[$providerIndex + 1] 'Cline derives its native provider argument from the opaque selector' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'deepseek-v4-flash' $args[$modelIndex + 1] 'Cline derives its native model argument from the opaque selector' + $configIndex = [Array]::IndexOf([string[]]$args, '--config') + Assert-True ($args[$configIndex + 1] -match '(?i)[\\/]\.cline$') 'Cline uses the documented isolated config root' + $dataIndex = [Array]::IndexOf([string[]]$args, '--data-dir') + Assert-True ($args[$dataIndex + 1] -match '(?i)[\\/]\.cline[\\/]data$') 'Cline data-dir is the isolated data root' + Assert-Equal 'available' $resultWith.telemetry.tool_calls.status 'Cline reports available tool-call telemetry' + Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Cline parses documented tool events' + } + $logText = [System.IO.File]::ReadAllText($logPath, [System.Text.UTF8Encoding]::new($false)) + Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName logs do not contain credential values" + Assert-True (($resultWith | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName result evidence does not contain credential values" + $withoutLogPath = Join-Path $without.Root "repo\$runnerName-fake-cli-log.jsonl" + Assert-True (Test-Path -LiteralPath $withoutLogPath -PathType Leaf) "$runnerName baseline process log exists" + $withoutRecords = @(Get-Content -LiteralPath $withoutLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + $withoutExecution = @($withoutRecords | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }) + Assert-Equal 1 $withoutExecution.Count "$runnerName baseline has one execution process" + $withoutLogText = [System.IO.File]::ReadAllText($withoutLogPath, [System.Text.UTF8Encoding]::new($false)) + Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName baseline log does not contain credential values" + Assert-True $withoutExecution[0].stdin_exact "$runnerName baseline receives exact prompt bytes" + if ($runnerName -eq 'copilot') { + Assert-True $withoutExecution[0].repository_agents_visible 'Copilot baseline sees the same staged AGENTS.md instruction' + Assert-True $withoutExecution[0].repository_copilot_instructions_visible 'Copilot baseline sees the same staged repository instruction' + Assert-True (-not $withoutExecution[0].candidate_skill_staged) 'Copilot baseline does not receive the candidate skill directory' + Assert-True (-not $withoutExecution[0].ambient_copilot_instructions_visible) 'Copilot baseline excludes the ambient personal instruction' + Assert-True (($resultWithout | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') 'Copilot baseline result evidence does not contain credential values' + } + } + $staleCli = $fakeCli.Replace("'opencode' { '--format --dir --model --auto --pure --continue --session' }", "'opencode' { '--format --dir --model --pure --continue --session' }") + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $staleCli, [System.Text.UTF8Encoding]::new($false)) + $stalePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['opencode'] + Assert-Equal 'incompatible' $stalePreflight.status 'stale OpenCode help contract is rejected during preflight' + Assert-True (@($stalePreflight.reasons | Where-Object { $_ -match '--auto' }).Count -gt 0) 'stale OpenCode option failure identifies the missing flag' + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $fakeCli, [System.Text.UTF8Encoding]::new($false)) + $fileAuthHome = Join-Path $recordedRoot 'codex-file-auth' + New-Item -ItemType Directory -Path $fileAuthHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'auth.json'), '{"canary":"not-logged"}', [System.Text.UTF8Encoding]::new($false)) + $env:OPENAI_API_KEY = $null + $env:CODEX_HOME = $fileAuthHome + $fileAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $fileAuthPreflight.status 'Codex file-only authentication is fail-closed' + Assert-True (@($fileAuthPreflight.reasons | Where-Object { $_ -match 'auth\.json' }).Count -gt 0) 'Codex file-auth limitation is explicit' + $env:OPENAI_API_KEY = 'recorded-canary-not-logged' + $env:CODEX_HOME = $recordedOldCodexHome + # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and + # no-auth fixtures are all deterministic and contain no credential values. + $env:COPILOT_GITHUB_TOKEN = $null + $env:GH_TOKEN = $null + $env:GITHUB_TOKEN = $null + $missingGhConfig = Join-Path $recordedRoot 'missing-github-cli-auth' + + # The fixture marker is fake-CLI input only; it models a positive OS + # keychain lookup without naming or reading a real credential-store file. + $copilotKeychainHome = Join-Path $recordedRoot 'copilot-keychain-home' + New-Item -ItemType Directory -Path $copilotKeychainHome -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $with.Root 'home\.copilot') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available'), 'fixture marker only', [Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $copilotKeychainHome + $env:GH_CONFIG_DIR = $missingGhConfig + $copilotKeychainPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotKeychainPreflight.status 'Copilot tokenless OS-keychain authentication remains compatible' + Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'unavailable' }).Count -eq 1) 'Copilot preflight leaves native keychain readiness conditional' + Assert-True (@($copilotKeychainPreflight.warnings | Where-Object { $_ -match 'cannot be proven' }).Count -gt 0) 'Copilot preflight explains the unverified keychain/service boundary' + $copilotKeychainResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotKeychainResult.status 'Copilot keychain fixture executes without an exported token' + $keychainRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.copilot_authentication_source -eq 'os_keychain' }) + Assert-Equal 1 $keychainRecords.Count 'Copilot fake observes the simulated OS-keychain path' + Remove-Item -LiteralPath (Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available') -Force + + $copilotGhFallbackHome = Join-Path $recordedRoot 'copilot-gh-fallback-home' + New-Item -ItemType Directory -Path $copilotGhFallbackHome -Force | Out-Null + $copilotGhConfig = Join-Path $recordedRoot 'copilot-gh-config' + New-Item -ItemType Directory -Path $copilotGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $copilotGhFallbackHome + $env:GH_CONFIG_DIR = $copilotGhConfig + $copilotGhPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' + $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' + Assert-True $copilotGhResult.evidence.credential.github_cli_token_resolved 'Copilot records GitHub CLI token fallback without storing the token value' + Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' + $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.stdin_received -eq $true -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) + Assert-Equal 1 $ghRecords.Count 'Copilot fake observes only the protected GH_TOKEN produced by trusted GitHub CLI fallback' + + $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' + New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null + $env:COPILOT_HOME = $copilotNoAuthHome + $env:GH_CONFIG_DIR = $missingGhConfig + $copilotNoAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotNoAuthPreflight.status 'Copilot preflight does not require an exported token when native auth is not observable' + Assert-True (@($copilotNoAuthPreflight.warnings | Where-Object { $_ -match 'conditional' }).Count -gt 0) 'Copilot no-auth preflight is explicitly conditional' + $copilotNoAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'failed' $copilotNoAuthResult.status 'Copilot no-auth execution failure is captured without a model request' + Assert-Equal 'copilot_os_keychain_or_github_cli_unverified' $copilotNoAuthResult.evidence.credential.source 'Copilot no-auth evidence does not claim authentication' + Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + $env:COPILOT_HOME = $recordedOldCopilotHome + Write-Output 'Real runner deterministic adapter conformance: PASS' +} finally { + $env:PATH = $recordedOldPath + $env:OPENAI_API_KEY = $recordedOldOpenAi + $env:CODEX_HOME = $recordedOldCodexHome + $env:AGENTIC_GLOBAL_SECRET = $recordedOldGlobalSecret + $env:OPENCODE_DISABLE_PROJECT_CONFIG = $recordedOldProjectDisable + $env:COPILOT_GITHUB_TOKEN = $recordedOldCopilotToken + $env:GH_TOKEN = $recordedOldGhToken + $env:GITHUB_TOKEN = $recordedOldGithubToken + $env:COPILOT_HOME = $recordedOldCopilotHome + $env:GH_CONFIG_DIR = $recordedOldGhConfigDir + if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } +} +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } +} + +function Assert-Throws { + param([scriptblock]$Action, [string]$Message) + $thrown = $false + try { & $Action } catch { $thrown = $true } + if (-not $thrown) { throw "ASSERT: $Message" } +} + +function Write-TestJson { + param([string]$Path, [object]$Value) + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Invoke-Fake { + param( + [string]$FakePath, + [string]$Command, + [string]$RunPath, + [string]$ProfilePath, + [string]$Scenario = '' + ) + + $arguments = @('-NoProfile', '-File', $FakePath, $Command, '-Run', $RunPath, '-Profile', $ProfilePath) + if (-not [string]::IsNullOrWhiteSpace($Scenario)) { $arguments += @('-Scenario', $Scenario) } + $output = & pwsh @arguments + if ($LASTEXITCODE -ne 0) { throw "Fake runner '$Command' failed: $([string]::Join(' ', @($output)))" } + $json = [string]::Join([Environment]::NewLine, @($output)) + if ([string]::IsNullOrWhiteSpace($json)) { throw "Fake runner '$Command' returned no JSON." } + return $json | ConvertFrom-Json +} + +function New-TestRun { + param( + [string]$IterationDirectory, + [ValidateSet('with_skill', 'without_skill')][string]$Configuration + ) + + $evalDirectory = Join-Path $IterationDirectory 'conformance' + $runRoot = Join-Path $evalDirectory $Configuration + $repo = Join-Path $runRoot 'repo' + $homeDirectory = Join-Path $runRoot 'home' + New-Item -ItemType Directory -Path $repo,$homeDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'README.txt'), 'isolated home', [System.Text.UTF8Encoding]::new($false)) + New-Item -ItemType Directory -Path (Join-Path $repo '.github') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repo 'AGENTS.md'), '# repo-owned-agent-instruction', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $repo '.github\copilot-instructions.md'), '# repo-owned-copilot-instruction', [System.Text.UTF8Encoding]::new($false)) + $prompt = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀.`r`n" + ("large-prompt-line-0123456789`r`n" * 4096) + [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) + if ($Configuration -eq 'with_skill') { + $skill = Join-Path $runRoot 'skill\candidate' + New-Item -ItemType Directory -Path $skill -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $skill 'SKILL.md'), '# candidate', [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = 'skill/candidate' + $skillHash = ('b' * 64) + } else { + $skillDirectory = $null + $skillHash = $null + } + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = 1 + evalName = 'conformance' + skillName = if ($Configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $Configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @() + fixtureHash = ('a' * 64) + skillHash = $skillHash + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + $path = Join-Path $runRoot 'run.json' + Write-TestJson -Path $path -Value $run + return [pscustomobject]@{ Root = $runRoot; Path = $path; Contract = $run } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-runner-conformance-' + [Guid]::NewGuid().ToString('N')) +try { + $iteration = Join-Path $testRoot 'iteration-1' + New-Item -ItemType Directory -Path $iteration -Force | Out-Null + $canaryPath = Join-Path $testRoot 'eval-metadata.json' + [System.IO.File]::WriteAllText($canaryPath, 'grading-key-canary', [System.Text.UTF8Encoding]::new($false)) + $globalRoot = Join-Path $testRoot 'seeded-global-profile' + New-Item -ItemType Directory -Path $globalRoot -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $globalRoot 'same-name-SKILL.md'), 'must remain invisible', [System.Text.UTF8Encoding]::new($false)) + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_GLOBAL_RULES', $globalRoot, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_MEMORY', 'seeded-memory', 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_PLUGINS', 'seeded-plugins', 'Process') + + $profilePath = Join-Path $iteration 'execution-profile.json' + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $unsupportedProfilePath = Join-Path $iteration 'unsupported-profile.json' + Write-TestJson -Path $unsupportedProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'unsupported' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $legacyProviderProfilePath = Join-Path $iteration 'legacy-provider-profile.json' + Write-TestJson -Path $legacyProviderProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + provider = 'fixture-provider' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + + $with = New-TestRun -IterationDirectory $iteration -Configuration with_skill + $without = New-TestRun -IterationDirectory $iteration -Configuration without_skill + $fakePath = Join-Path $runnerRoot 'fake\runner.ps1' + + $descriptor = Invoke-Fake -FakePath $fakePath -Command describe -Run $with.Path -Profile $profilePath + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + Assert-Equal 'fake' $descriptor.name 'descriptor identity' + Assert-Equal (Get-RunnerSchemaNames).Protocol $descriptor.protocol_version 'descriptor protocol' + Assert-Throws { Assert-RunnerDescriptor -Descriptor ([pscustomobject]@{ schema = $descriptor.schema; protocol_version = 'changed'; name = 'fake' }) } 'changed protocol must fail descriptor validation' + Assert-Throws { Resolve-ExecutionProfile -ProfilePath $legacyProviderProfilePath } 'execution profile rejects the removed provider field' + + $unsupported = Invoke-Fake -FakePath $fakePath -Command preflight -Run $with.Path -Profile $unsupportedProfilePath + Assert-Equal 'incompatible' $unsupported.status 'unsupported capability/profile must fail during preflight' + Assert-True (@($unsupported.reasons).Count -gt 0) 'incompatible preflight must explain its reason' + + $withResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath + $withoutResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath + foreach ($result in @($withResult, $withoutResult)) { + [void](Assert-ExecutionResult -Result $result) + Assert-Equal 'completed' $result.status 'normal completion status' + Assert-True $result.session.fresh 'fresh session flag' + Assert-True (-not $result.session.resumed) 'resume must be false' + Assert-Equal 1 $result.attempt_count 'answer-quality retry is forbidden' + Assert-True ($result.requested.PSObject.Properties.Name -notcontains 'provider') 'portable execution result must not expose provider' + Assert-Equal 'fixture-model' $result.requested.model 'model must pass unchanged' + Assert-Equal 'high' $result.requested.reasoning_effort 'reasoning effort must pass unchanged' + Assert-Equal 'isolated-default' $result.requested.configuration_profile 'configuration profile must pass unchanged' + Assert-Equal 'default' $result.requested.tool_profile 'tool profile must pass unchanged' + Assert-Equal 'unavailable' $result.telemetry.tokens.status 'missing token telemetry must be explicit' + Assert-True ($result.telemetry.tokens.PSObject.Properties.Name -notcontains 'value') 'missing token telemetry must not contain a zero placeholder' + foreach ($artifact in @($result.artifacts)) { + Assert-True ($artifact.path -notmatch '(^|/|\\)\.\.(/|\\|$)') 'artifact path must not escape the run' + $artifactPath = Join-Path $($with.Root) ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if ($result.run.configuration -eq 'without_skill') { $artifactPath = Join-Path $($without.Root) ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar) } + Assert-True (Test-Path -LiteralPath $artifactPath -PathType Leaf) 'artifact must exist inside its run' + Assert-Equal $artifact.sha256 ((Get-FileHash -Algorithm SHA256 -LiteralPath $artifactPath).Hash.ToLowerInvariant()) 'artifact hash' + Assert-Equal $artifact.size (Get-Item -LiteralPath $artifactPath).Length 'artifact size' + Assert-True (-not [string]::IsNullOrWhiteSpace($artifact.media_type)) 'artifact media type' + } + } + Assert-True ($withResult.session.id -ne $withoutResult.session.id) 'paired arms must have distinct session ids' + Assert-True ($withResult.run.configuration -ne $withoutResult.run.configuration) 'paired arms must retain distinct configurations' + + $withPromptEvidence = Get-Content (Join-Path $with.Root 'evidence\prompt-delivery.json') -Raw | ConvertFrom-Json + $withoutPromptEvidence = Get-Content (Join-Path $without.Root 'evidence\prompt-delivery.json') -Raw | ConvertFrom-Json + foreach ($pair in @( + [pscustomobject]@{ Run = $with; Evidence = $withPromptEvidence; Result = $withResult } + [pscustomobject]@{ Run = $without; Evidence = $withoutPromptEvidence; Result = $withoutResult } + )) { + $promptBytes = [System.IO.File]::ReadAllBytes((Join-Path $pair.Run.Root 'prompt.md')) + Assert-Equal (Get-Sha256HexFromBytes -Bytes $promptBytes) $pair.Evidence.first_task_input_sha256 'prompt must be the first task input byte-for-byte' + Assert-Equal $promptBytes.Length $pair.Evidence.first_task_input_bytes 'prompt byte length' + Assert-True $pair.Evidence.byte_exact 'prompt fidelity evidence' + Assert-Equal (Join-Path $pair.Run.Root 'repo') $pair.Evidence.working_directory 'working directory' + Assert-Equal (Join-Path $pair.Run.Root 'home') $pair.Evidence.home_directory 'isolated home' + Assert-True (-not $pair.Evidence.global_rules_visible -and -not $pair.Evidence.global_memory_visible -and -not $pair.Evidence.global_plugins_visible -and -not $pair.Evidence.global_same_name_skill_visible) 'seeded ambient rules/memory/plugins/skill must remain invisible' + } + Assert-True $withPromptEvidence.candidate_skill_exposed 'candidate skill is exposed only for with_skill' + Assert-True (-not $withoutPromptEvidence.candidate_skill_exposed) 'candidate skill is excluded for without_skill' + Assert-True (Test-Path -LiteralPath (Join-Path $with.Root 'skill\candidate\SKILL.md')) 'with_skill has staged skill' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $without.Root 'skill'))) 'without_skill has no staged skill' + + $escape = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath -Scenario escape + $escapeEvidence = Get-Content (Join-Path $with.Root 'evidence\boundary-probes.json') -Raw | ConvertFrom-Json + Assert-True $escapeEvidence.read_outside_run.attempted 'read escape probe was exercised' + Assert-True $escapeEvidence.read_outside_run.blocked 'read escape probe was blocked' + Assert-True $escapeEvidence.write_outside_run.attempted 'write escape probe was exercised' + Assert-True $escapeEvidence.write_outside_run.blocked 'write escape probe was blocked' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $testRoot 'escape-write.txt'))) 'escape write did not create a file' + + $refusal = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario refusal + Assert-Equal 'completed' $refusal.status 'refusal is a completed captured response' + Assert-True $refusal.final_response.text.Contains('cannot') 'refusal response is retained' + $timeout = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario timeout + Assert-Equal 'timed_out' $timeout.status 'timeout normalization' + Assert-Equal 'unavailable' $timeout.final_response.status 'timeout has no final response' + Assert-True ($null -eq $timeout.exit.status) 'timeout exit status is unavailable' + $failure = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario failure + Assert-Equal 'failed' $failure.status 'harness failure normalization' + Assert-Equal 17 $failure.exit.status 'harness failure exit status' + $incompatible = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario incompatible + Assert-Equal 'incompatible' $incompatible.status 'incompatible normalization' + $unknown = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario unknown-event + Assert-True (@($unknown.warnings | Where-Object { $_ -match 'future\.event\.v99' }).Count -gt 0) 'unknown events produce explicit warnings' + + $resolvedWith = Resolve-RunContract -RunPath $with.Path + $resolvedProfile = Resolve-ExecutionProfile -ProfilePath $profilePath + $mandatoryCapabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + } + $pragmaticCapabilities = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $pragmaticCapabilities[$name] = $mandatoryCapabilities[$name] } + $pragmaticCapabilities['filesystem_confinement'] = 'unsupported' + $strictCapabilities = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $strictCapabilities[$name] = $mandatoryCapabilities[$name] } + $strictCapabilities['filesystem_confinement'] = 'supported' + $pragmaticResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'pragmatic response' -ExitStatus ([Nullable[int]]0) -IsolationCapabilities $pragmaticCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $pragmaticResult) + Assert-Equal 'completed' $pragmaticResult.status 'pragmatic completed result remains usable' + Assert-Equal 'pragmatic' $pragmaticResult.isolation.level 'missing hard confinement downgrades confidence' + Assert-True (-not $pragmaticResult.isolation.hard_filesystem_confinement) 'pragmatic result does not claim hard confinement' + $strictResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'strict response' -ExitStatus ([Nullable[int]]0) -IsolationCapabilities $strictCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $strictResult) + Assert-Equal 'strict' $strictResult.isolation.level 'proven hard confinement reports strict isolation' + Assert-True $strictResult.isolation.hard_filesystem_confinement 'strict result claims hard confinement' + $failedResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status failed -ExitStatus ([Nullable[int]]17) -Failure (New-ExecutionFailure -Code 'fixture_failure' -Message 'fixture failure') -IsolationCapabilities $pragmaticCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $failedResult) + Assert-Equal 'failed' $failedResult.status 'failed execution keeps proven pragmatic isolation' + Assert-Equal 'verified' $failedResult.isolation.status 'failed execution retains control verification' + $timedOutResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status timed_out -IsolationCapabilities $strictCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $timedOutResult) + Assert-Equal 'timed_out' $timedOutResult.status 'timed out execution keeps proven strict isolation' + $missingCapability = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $missingCapability[$name] = $mandatoryCapabilities[$name] } + $missingCapability.Remove('response_capture') + $rejectedResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'must be rejected' -IsolationCapabilities $missingCapability -AttemptCount 1 + [void](Assert-ExecutionResult -Result $rejectedResult) + Assert-Equal 'incompatible' $rejectedResult.status 'unproven mandatory control rejects completion' + Assert-Equal 'unverified' $rejectedResult.isolation.status 'rejected completion is unverified' + Assert-Equal 'unsupported' $rejectedResult.isolation.level 'rejected completion has unsupported isolation' + $preflightRejected = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status incompatible -FinalResponseReason 'preflight_incompatible' -IsolationCapabilities ([ordered]@{}) -AttemptCount 1 + [void](Assert-ExecutionResult -Result $preflightRejected) + Assert-Equal 'unverified' $preflightRejected.isolation.status 'preflight incompatibility is unverified' + Assert-Equal 'unsupported' $preflightRejected.isolation.level 'preflight incompatibility is unsupported' + $translated = Get-SandboxVisiblePath -HostPath (Join-Path $with.Root 'repo\file.txt') -RunRoot $with.Root -Platform 'linux' + Assert-Equal '/run/repo/file.txt' $translated 'Linux hard sandbox paths use the child namespace' + $macPath = Get-SandboxVisiblePath -HostPath (Join-Path $with.Root 'repo\file.txt') -RunRoot $with.Root -Platform 'macos' + Assert-Equal ([System.IO.Path]::GetFullPath((Join-Path $with.Root 'repo\file.txt'))) $macPath 'macOS sandbox paths remain host-visible' + Assert-Equal 'recorded-cli 1.2.3' (Get-ObservableVersionFromText "`nrecorded-cli 1.2.3`n") 'observable version capture keeps the exact line' + Assert-True ($null -eq (Get-ObservableVersionFromText "`n `n")) 'empty version output has no observable value' + + foreach ($fixture in @('codex-events.jsonl', 'opencode-events.jsonl', 'copilot-events.jsonl')) { + $fixturePath = Join-Path $PSScriptRoot "fixtures\$fixture" + $parsed = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText($fixturePath, [System.Text.UTF8Encoding]::new($false))) + Assert-Equal 0 $parsed.Errors.Count "recorded $fixture has valid JSONL" + Assert-True ($parsed.Events.Count -ge 4) "recorded $fixture has events" + Assert-True (@($parsed.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) "recorded $fixture includes an unknown event" + } + $copilotFixture = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\copilot-events.jsonl'), [System.Text.UTF8Encoding]::new($false))) + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.message' }).Count -ge 1) 'recorded copilot fixture includes documented assistant.message output' + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.usage' }).Count -eq 1) 'recorded copilot fixture includes documented assistant.usage output' + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'tool.execution_start' }).Count -eq 1) 'recorded copilot fixture includes documented tool.execution output' + $clineFixture = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\cline-events.jsonl'), [System.Text.UTF8Encoding]::new($false))) + Assert-Equal 0 $clineFixture.Errors.Count 'recorded cline fixture has valid JSONL' + Assert-True ($clineFixture.Events.Count -ge 9) 'recorded cline fixture has events' + Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'say' -and $_.say -eq 'tool' }).Count -eq 1) 'recorded cline fixture includes documented tool output' + Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'say' -and $_.say -eq 'completion_result' }).Count -eq 1) 'recorded cline fixture includes documented completion_result output' + Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) 'recorded cline fixture includes an unknown event' + + $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) + $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) + $bridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-execution-result.ps1'), [System.Text.UTF8Encoding]::new($false)) + Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' + Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' + Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' + + $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' + $resultPath = Join-Path $iteration 'conformance\results\with-skill.result.json' + New-Item -ItemType Directory -Path (Split-Path -Parent $rawPath) -Force | Out-Null + $bridgeResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath + Write-TestJson -Path $rawPath -Value $bridgeResult + $bridgePath = Join-Path $runnerRoot 'bridge-execution-result.ps1' + $bridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $with.Path -ExecutionResult $rawPath -Result $resultPath + if ($LASTEXITCODE -ne 0) { throw "execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))" } + $portable = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json + Assert-Equal 'codebeltnet/agentic/eval-result/2' $portable.schema 'bridge preserves existing result schema' + Assert-Equal 'completed' $portable.execution_status 'bridge carries execution status' + Assert-Equal 'fixture-model' $portable.model 'bridge carries resolved model' + Assert-True ($portable.PSObject.Properties.Name -notcontains 'provider') 'bridge removes provider from portable result' + Assert-True ($null -eq $portable.total_tokens) 'bridge keeps unavailable total tokens unavailable' + Assert-Equal 0 $portable.tool_calls 'bridge carries available tool-call count' + Assert-True $portable.isolation.transcript_captured 'bridge carries transcript availability' + Assert-Equal 'strict' $portable.isolation.level 'bridge carries isolation confidence level' + Assert-Equal 'verified' $portable.isolation.status 'bridge carries isolation verification status' + Assert-True (@($portable.isolation.mechanisms).Count -gt 0) 'bridge carries isolation mechanisms' + Assert-True (@($portable.output_files).Count -gt 0) 'bridge carries confined evidence paths' + + $acceptedBridgeResult = $bridgeResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $acceptedBridgeResult.resolved.status = 'accepted_request' + $acceptedBridgeResult.resolved.model = $null + $acceptedBridgeResult.resolved.reason = 'fixture accepted the requested alias without exposing backend resolution.' + Write-TestJson -Path $rawPath -Value $acceptedBridgeResult + $acceptedBridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $with.Path -ExecutionResult $rawPath -Result $resultPath + if ($LASTEXITCODE -ne 0) { throw "accepted-configuration bridge failed: $([string]::Join(' ', @($acceptedBridgeOutput)))" } + $acceptedPortable = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json + Assert-Equal 'fixture-model' $acceptedPortable.model 'bridge keeps requested model compatibility label' + Assert-Equal 'fixture-model' $acceptedPortable.requested_model 'bridge records requested model separately' + Assert-True ([string]::IsNullOrWhiteSpace([string]$acceptedPortable.resolved_model)) 'bridge does not invent a resolved model' + Assert-Equal 'accepted_request' $acceptedPortable.configuration_resolution_status 'bridge carries configuration provenance' + Assert-True ([string]$acceptedPortable.notes -match 'configuration_resolution=accepted_request') 'bridge notes configuration provenance' + + Write-Output 'Eval Runner conformance: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_GLOBAL_RULES', $null, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_MEMORY', $null, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_PLUGINS', $null, 'Process') + if (Test-Path -LiteralPath $testRoot) { Remove-Item -LiteralPath $testRoot -Recurse -Force } +} + +Invoke-RecordedRunnerTests diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index 63be59f..c28bfe8 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -400,7 +400,10 @@ function Get-ReportRun { configuration = $Configuration feedback_key = "eval-$EvalId-$Configuration" model = [string](Get-Property -Object $Result -Name 'model' -Default '') - provider = [string](Get-Property -Object $Result -Name 'provider' -Default '') + requested_model = [string](Get-Property -Object $Result -Name 'requested_model' -Default '') + resolved_model = [string](Get-Property -Object $Result -Name 'resolved_model' -Default '') + configuration_resolution_status = [string](Get-Property -Object $Result -Name 'configuration_resolution_status' -Default '') + configuration_resolution_reason = [string](Get-Property -Object $Result -Name 'configuration_resolution_reason' -Default '') harness = [string](Get-Property -Object $Result -Name 'harness' -Default '') executed_utc = [string](Get-Property -Object $Result -Name 'executed_utc' -Default '') output = $output @@ -412,6 +415,9 @@ function Get-ReportRun { stdout = [string](Get-Property -Object $Result -Name 'stdout' -Default '') stderr = [string](Get-Property -Object $Result -Name 'stderr' -Default '') exit_status = Get-Property -Object $Result -Name 'exit_status' -Default $null + execution_status = Get-Property -Object $Result -Name 'execution_status' -Default $null + execution_run_id = Get-Property -Object $Result -Name 'execution_run_id' -Default $null + execution_result_file = Get-Property -Object $Result -Name 'execution_result_file' -Default $null metrics = $metrics isolation = Get-Property -Object $Result -Name 'isolation' -Default $null grades = @($grades) @@ -456,7 +462,6 @@ function Write-FirstPartyReport { $evals = [System.Collections.Generic.List[object]]::new() $allModels = [System.Collections.Generic.List[string]]::new() - $allProviders = [System.Collections.Generic.List[string]]::new() $completedRuns = 0 foreach ($entry in @($Manifest.evals)) { $evalDirectory = Join-Path $IterationPath ([string]$entry.directory) @@ -469,13 +474,11 @@ function Write-FirstPartyReport { if ($null -ne $result) { $run = Get-ReportRun -Result $result -Configuration $configuration -EvalName ([string]$entry.eval_name) -EvalId ([int]$metadata.eval_id) -Assertions $assertions -RunPackageDirectory (Join-Path $evalDirectory $configuration) -EvalDirectory $evalDirectory -IterationPath $IterationPath $runMap[$configuration] = $run - if (-not [string]::IsNullOrWhiteSpace([string]$run.output) -or @($run.output_files).Count -gt 0) { $completedRuns++ } + $executionStatus = [string](Get-Property -Object $result -Name 'execution_status' -Default '') + if (-not [string]::IsNullOrWhiteSpace([string]$run.output) -or @($run.output_files).Count -gt 0 -or ($executionStatus -and $executionStatus -ne 'unrun')) { $completedRuns++ } if (-not [string]::IsNullOrWhiteSpace([string]$run.model) -and -not $allModels.Contains([string]$run.model)) { $allModels.Add([string]$run.model) } - if (-not [string]::IsNullOrWhiteSpace([string]$run.provider) -and -not $allProviders.Contains([string]$run.provider)) { - $allProviders.Add([string]$run.provider) - } } else { $runMap[$configuration] = $null } @@ -492,7 +495,6 @@ function Write-FirstPartyReport { $metadata = [ordered]@{ model = if ($allModels.Count -gt 0) { $allModels -join ', ' } else { $null } - provider = if ($allProviders.Count -gt 0) { $allProviders -join ', ' } else { $null } completed_runs = $completedRuns expected_runs = @($Manifest.evals).Count * 2 generated_utc = [string](Get-Property -Object $Manifest -Name 'generated_utc' -Default '') @@ -544,14 +546,19 @@ function Write-UpstreamGrading { $duration = Get-Property -Object $Result -Name 'duration_seconds' -Default $null $tokens = Get-Property -Object $Result -Name 'total_tokens' -Default $null $toolCalls = Get-Property -Object $Result -Name 'tool_calls' -Default $null + $exitStatus = Get-Property -Object $Result -Name 'exit_status' -Default $null + $errorsEncountered = if ($null -eq $exitStatus -or [string]::IsNullOrWhiteSpace([string]$exitStatus)) { $null } elseif ([int]$exitStatus -eq 0) { 0 } else { 1 } - $durationSeconds = if ($null -eq $duration) { 0.0 } else { [double]$duration } - $totalTokens = if ($null -eq $tokens) { 0 } else { [int64]$tokens } - Write-JsonFile -Path (Join-Path $RunDirectory 'timing.json') -Value ([ordered]@{ - total_tokens = $totalTokens - duration_ms = [math]::Round($durationSeconds * 1000, 0) - total_duration_seconds = $durationSeconds - }) + $timing = [ordered]@{} + if ($null -ne $duration -and -not [string]::IsNullOrWhiteSpace([string]$duration)) { + $durationSeconds = [double]$duration + $timing.duration_ms = [math]::Round($durationSeconds * 1000, 0) + $timing.total_duration_seconds = $durationSeconds + } + if ($null -ne $tokens -and -not [string]::IsNullOrWhiteSpace([string]$tokens)) { + $timing.total_tokens = [int64]$tokens + } + Write-JsonFile -Path (Join-Path $RunDirectory 'timing.json') -Value $timing $gradingDocument = [ordered]@{ expectations = @($expectations) @@ -563,7 +570,7 @@ function Write-UpstreamGrading { } execution_metrics = [ordered]@{ total_tool_calls = $toolCalls - errors_encountered = if ([int](Get-Property -Object $Result -Name 'exit_status' -Default 0) -eq 0) { 0 } else { 1 } + errors_encountered = $errorsEncountered } # The upstream aggregator reads timing.json when grading.json does not claim a duration. Keep the # portable run's timing in that sibling file so both elapsed time and token usage survive aggregation. diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index c73787a..7663f36 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -5,19 +5,21 @@ .DESCRIPTION This script computes and prints. It never executes a prompt, never spawns an agent, and never calls a model. It turns skills//evals/evals.json into a paste-ready evaluation package that a human can run in whatever - harness, provider, and model they choose, then validates the results that come back. + harness and model they choose, then validates the results that come back. Prepare mode writes one directory per eval. The grading key and result stubs stay at the eval-case level, outside - the two hermetic run directories a worker actually sees: + the two isolated run directories a worker actually sees: eval-metadata.json id, name, prompt, expected output, assertions, fixtures, hashes, assumptions results/ one prefilled result stub per configuration - with_skill/ a hermetic run: prompt.md, run.json, repo/ (materialized fixtures), home/, skill// + with_skill/ an isolated run: prompt.md, run.json, repo/ (materialized fixtures), home/, skill// without_skill/ the same run without any skill/ directory and no skill instructions - Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an isolated profile, and skill/ + Each run directory is the worker's staged root: repo/ is the working tree, home/ is an isolated profile, and skill/ (with_skill only) holds the candidate skill revision. run.json is a harness-neutral contract naming only paths inside - the run directory. Preparation validates the isolation invariants and fails early if a package would let a baseline - reach the skill, let a worker reach the source repository, or stage mismatched fixtures. + the run directory. The selected runner enforces the isolation and reports strict confidence when it also proves hard + filesystem confinement or pragmatic confidence when it does not. Preparation validates the isolation invariants and + fails early if a package would let a baseline reach the skill, stage the source repository into a run, or stage + mismatched fixtures. Collect mode reads a prepared package plus whatever result files were filled in, validates them, and invokes the packaged Anthropic skill-creator aggregator and static eval viewer after writing a deterministic comparison. The @@ -42,6 +44,34 @@ .PARAMETER Force Overwrite an existing iteration directory. +.PARAMETER Runner + Required package-local Eval Runner id written to execution-profile.json when -CodebeltReference is not used. + +.PARAMETER Model + Required runner-native model selector written to execution-profile.json when -CodebeltReference is not used. + +.PARAMETER CodebeltReference + Resolve the Codebelt reference configuration by discovering current GitHub Copilot CLI models and selecting + claude-haiku-4.5 only when it is still available. + +.PARAMETER ModelCatalogPath + Optional deterministic catalog JSON used by the model discovery helper. Intended for tests and offline validation. + +.PARAMETER ReasoningEffort + Optional runner-supported reasoning/effort setting written to execution-profile.json. + +.PARAMETER ConfigurationProfile + Runner configuration profile. Defaults to isolated-default. + +.PARAMETER ToolProfile + Runner tool profile. Defaults to default. + +.PARAMETER TimeoutSeconds + Per-arm runner timeout. Defaults to 900 seconds. + +.PARAMETER Concurrency + Requested external-orchestrator concurrency. Defaults to 1. It does not change paired-arm semantics. + .PARAMETER Changed Prepares a package for every repo-managed skill this branch changed, including uncommitted work. This is the form the eval completion gate uses after adding or modifying a skill. @@ -56,10 +86,10 @@ This is the fallback for results that were not finalized by the external evaluator. .EXAMPLE - pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-strong-name-signing + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-strong-name-signing -Runner github-copilot -Model claude-haiku-4.5 .EXAMPLE - pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -CodebeltReference .EXAMPLE pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults $env:TEMP/dotnet-strong-name-signing-workspace/iteration-1 @@ -93,6 +123,44 @@ param( [Parameter(ParameterSetName = 'Changed')] [switch]$Force, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$Runner, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$Model, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [switch]$CodebeltReference, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ModelCatalogPath, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ReasoningEffort, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ConfigurationProfile = 'isolated-default', + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ToolProfile = 'default', + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [ValidateRange(1, 86400)] + [int]$TimeoutSeconds = 900, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [ValidateRange(1, 128)] + [int]$Concurrency = 1, + [Parameter(ParameterSetName = 'Collect', Mandatory = $true)] [string]$CollectResults ) @@ -112,12 +180,16 @@ $packageSchema = 'codebeltnet/agentic/eval-package/2' $metadataSchema = 'codebeltnet/agentic/eval-metadata/2' $resultSchema = 'codebeltnet/agentic/eval-result/2' $runSchema = 'codebeltnet/agentic/eval-run/1' +$executionProfileSchema = 'codebeltnet/agentic/eval-execution-profile/1' +$executionResultSchema = 'codebeltnet/agentic/eval-execution-result/1' +$runnerProtocolSchema = 'codebeltnet/agentic/eval-runner-protocol/1' $maxFixtureInlineBytes = 32768 -# A materialized run is hermetic: the harness treats the run directory as the worker's sandbox root, mounts repo/ as +# A materialized run is self-contained: the runner treats the run directory as the worker's staged root, uses repo/ as # the working directory and home/ as the isolated user profile, and exposes skill/ only for a with_skill run. Nothing -# else in the package - the grading key, the paired run, other evals, or results - lives inside a run directory, so a -# worker confined to its run directory cannot reach any of it. +# else in the package - the grading key, the paired run, other evals, or results - is staged inside a run directory, so a +# worker that stays within its run directory is never handed any of it. Hard filesystem confinement is an optional +# confidence signal a runner may add on top; it is not required for this staging boundary. $runDirectoryNames = [ordered]@{ Working = 'repo' Home = 'home' @@ -127,6 +199,7 @@ $runDirectoryNames = [ordered]@{ } $reportToolRelativePath = 'tools/generate-eval-report.ps1' $skillCreatorToolRelativePath = 'tools/skill-creator' +$evalRunnerToolRelativePath = 'tools/eval-runners' $skillCreatorEvalFiles = @( 'LICENSE.txt', 'agents/grader.md', @@ -315,6 +388,91 @@ function Assert-WorkspaceLocation { } } +function Get-HarnessName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + switch ($RunnerName) { + 'github-copilot' { return 'GitHub Copilot CLI' } + 'codex' { return 'Codex CLI' } + 'opencode' { return 'OpenCode' } + 'cline' { return 'Cline' } + 'fake' { return 'Deterministic fake runner' } + default { return $RunnerName } + } +} + +function Get-SupportedRunnerIds { + param([Parameter(Mandatory = $true)][string]$RepoRoot) + + $runnerRoot = Join-Path $RepoRoot 'scripts/eval-runners' + if (-not (Test-Path -LiteralPath $runnerRoot -PathType Container)) { + return @() + } + + return @(Get-ChildItem -LiteralPath $runnerRoot -Directory -Force | + Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName 'runner.ps1') -PathType Leaf } | + Sort-Object Name | + ForEach-Object { $_.Name }) +} + +function Resolve-ExecutionSelection { + param([Parameter(Mandatory = $true)][string]$RepoRoot) + + $referenceRunner = 'github-copilot' + $referenceModel = 'claude-haiku-4.5' + $supportedRunners = @(Get-SupportedRunnerIds -RepoRoot $RepoRoot) + $supportedText = if ($supportedRunners.Count -gt 0) { $supportedRunners -join ', ' } else { '(none found)' } + + if ($CodebeltReference -and (-not [string]::IsNullOrWhiteSpace($Runner) -or -not [string]::IsNullOrWhiteSpace($Model))) { + throw 'Choose either -CodebeltReference or an explicit -Runner/-Model pair, not both.' + } + + if ($CodebeltReference) { + if ($supportedRunners -notcontains $referenceRunner) { + throw "Codebelt Reference requires runner '$referenceRunner', but it is unavailable. Supported runner IDs: $supportedText." + } + $discoveryScript = Join-Path $RepoRoot 'scripts/Get-HarnessModels.ps1' + if (-not (Test-Path -LiteralPath $discoveryScript -PathType Leaf)) { + throw "Cannot resolve Codebelt Reference because '$discoveryScript' is missing." + } + + $arguments = @('-Runner', $referenceRunner, '-RequireModel', $referenceModel) + if (-not [string]::IsNullOrWhiteSpace($ModelCatalogPath)) { + $arguments += @('-CatalogPath', $ModelCatalogPath) + } + $discoveryOutput = & pwsh -NoProfile -File $discoveryScript @arguments 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Codebelt Reference requires $referenceRunner + $referenceModel, but current model discovery could not verify it. $($discoveryOutput -join [Environment]::NewLine)" + } + + return [pscustomobject]@{ + Runner = $referenceRunner + Model = $referenceModel + Harness = Get-HarnessName -RunnerName $referenceRunner + Preset = 'Codebelt Reference' + } + } + + $hasRunner = -not [string]::IsNullOrWhiteSpace($Runner) + $hasModel = -not [string]::IsNullOrWhiteSpace($Model) + if (-not $hasRunner -and -not $hasModel) { + throw "Evaluation preparation requires a resolved Harness + Model before RUN-THIS.prompt.md can be generated. Pass -Runner and -Model, or use -CodebeltReference after verifying the current catalog. Supported runner IDs: $supportedText." + } + if ($hasRunner -ne $hasModel) { + throw 'Runner/model selection is atomic: pass both -Runner and -Model, or neither when no package will be generated.' + } + if ($supportedRunners -notcontains $Runner) { + throw "Unsupported runner '$Runner'. Supported runner IDs: $supportedText." + } + + return [pscustomobject]@{ + Runner = $Runner + Model = $Model + Harness = Get-HarnessName -RunnerName $Runner + Preset = 'Custom' + } +} + function Get-EvalName { param([object]$EvalEntry) @@ -506,7 +664,6 @@ function New-ResultStub { eval_name = $EvalName configuration = $Configuration model = '' - provider = '' harness = '' executed_utc = '' output = '' @@ -518,6 +675,9 @@ function New-ResultStub { stdout = '' stderr = '' exit_status = $null + execution_status = 'unrun' + execution_run_id = '' + execution_result_file = '' duration_seconds = $null total_tokens = $null tool_calls = $null @@ -542,6 +702,21 @@ function New-ResultStub { } } +function New-ExecutionProfile { + param([Parameter(Mandatory = $true)][object]$ExecutionSelection) + + return [ordered]@{ + schema = $executionProfileSchema + runner = $ExecutionSelection.Runner + model = $ExecutionSelection.Model + reasoning_effort = if ([string]::IsNullOrWhiteSpace($ReasoningEffort)) { $null } else { $ReasoningEffort } + configuration_profile = $ConfigurationProfile + tool_profile = $ToolProfile + timeout_seconds = $TimeoutSeconds + concurrency = $Concurrency + } +} + function Get-JsonProperty { param( [object]$Object, @@ -879,6 +1054,32 @@ function Copy-SkillCreatorEvalTools { return $destinationRoot } +function Copy-EvalRunnerTools { + param( + [string]$RepoRoot, + [string]$IterationDirectory + ) + + $sourceRoot = Join-Path (Join-Path $RepoRoot 'scripts') 'eval-runners' + if (-not (Test-Path -LiteralPath $sourceRoot -PathType Container)) { + throw "Missing Eval Runner protocol source '$sourceRoot'." + } + + $destinationRoot = Join-Path $IterationDirectory $evalRunnerToolRelativePath + $files = Get-ChildItem -LiteralPath $sourceRoot -Recurse -File -Force | + Where-Object { $_.FullName -notmatch '[\\/]tests[\\/]' } | + ForEach-Object { Get-RelativePath -BasePath $sourceRoot -FullPath $_.FullName } | + Sort-Object + foreach ($relative in $files) { + $source = Join-Path $sourceRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destination = Join-Path $destinationRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $destination) -Force | Out-Null + Copy-Item -LiteralPath $source -Destination $destination -Force + } + + return $destinationRoot +} + # The candidate skill's fingerprint, computed from the source over exactly the files Copy-SkillTree stages. The staged # copy in each with_skill run must reproduce this value, which is how preparation proves the worker received the # revision under development rather than a globally installed one. @@ -946,8 +1147,9 @@ function New-RunManifest { } } -# Fail package generation the moment a run violates an isolation invariant, so a contaminated package never reaches a -# harness. These checks operate on the materialized run directories, not on prose. +# Fail package generation the moment a run violates an experimental isolation invariant, so a contaminated package never +# reaches a harness. The filesystemIsolationRequired field declares the staged workspace boundary; hard OS confinement is +# evaluated separately by the selected runner and reported as strict or pragmatic confidence. function Assert-RunIsolation { param( [string]$EvalName, @@ -1000,7 +1202,7 @@ function Assert-RunIsolation { throw "$EvalName/$configuration run.json must require fresh context." } if (-not [bool]$runManifest.filesystemIsolationRequired -or -not [bool]$runManifest.isolatedHomeRequired) { - throw "$EvalName/$configuration run.json must require filesystem and home isolation." + throw "$EvalName/$configuration run.json must require the staged workspace boundary and isolated home." } # 6. No run manifest references the source repository, and 7. none references a global skill install. @@ -1075,6 +1277,8 @@ function Invoke-PrepareMode { throw "No evals selected for '$Skill'." } + $executionSelection = Resolve-ExecutionSelection -RepoRoot $repoRoot + $workspaceRoot = if ([string]::IsNullOrWhiteSpace($OutputRoot)) { Join-Path (Join-Path $repoRoot '.bot') "$Skill-workspace" } else { @@ -1108,6 +1312,8 @@ function Invoke-PrepareMode { [void](Copy-ReportTool -RepoRoot $repoRoot -IterationDirectory $iterationDirectory) $skillCreatorSourcePath = Resolve-SkillCreatorSourcePath -RequestedPath $null [void](Copy-SkillCreatorEvalTools -IterationDirectory $iterationDirectory -SourcePath $skillCreatorSourcePath) + [void](Copy-EvalRunnerTools -RepoRoot $repoRoot -IterationDirectory $iterationDirectory) + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value (New-ExecutionProfile -ExecutionSelection $executionSelection) $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } @@ -1157,9 +1363,9 @@ function Invoke-PrepareMode { $repoFiles = @($fixtures | ForEach-Object { $_.RepoRelative } | Sort-Object) - # Materialize both runs. Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an + # Materialize both runs. Each run directory is the worker's staged root: repo/ is the working tree, home/ is an # isolated profile, and skill/ (with_skill only) holds the candidate. The grading key and results live one level - # up, outside every run directory, so a worker confined to its run directory can never reach them. + # up, outside every run directory, so a worker that stays within its run directory is never handed them. foreach ($configuration in @('with_skill', 'without_skill')) { $runDir = Join-Path $evalDirectory $configuration New-Item -ItemType Directory -Path $runDir -Force | Out-Null @@ -1202,7 +1408,7 @@ function Invoke-PrepareMode { $assumptions = [System.Collections.Generic.List[string]]::new() $assumptions.Add('Run with_skill and without_skill on the same model, same version, and same configuration. Different models measure the model, not the skill.') - $assumptions.Add('Each run is hermetic: launch a fresh worker with its run directory as the sandbox root, its repo/ as the working directory, and its home/ as the isolated profile.') + $assumptions.Add('Each run is isolated: launch a fresh worker with its run directory as the staged root, its repo/ as the working directory, and its home/ as the isolated profile. The runner reports strict confidence when it also proves hard filesystem confinement and pragmatic confidence when it does not.') $assumptions.Add("Both runs share an identical materialized repository. Only the with_skill run exposes the candidate skill under skill/$Skill/.") $notInlinedFixtures = @($fixtures | Where-Object { -not $_.Inlined }) if ($notInlinedFixtures.Count -gt 0) { @@ -1239,12 +1445,14 @@ function Invoke-PrepareMode { prompt_file = "with_skill/$($runDirectoryNames.Prompt)" run_manifest = "with_skill/$($runDirectoryNames.Run)" result_file = 'results/with-skill.result.json' + execution_result_file = 'results/with-skill.execution-result.json' } without_skill = [ordered]@{ run_directory = 'without_skill' prompt_file = "without_skill/$($runDirectoryNames.Prompt)" run_manifest = "without_skill/$($runDirectoryNames.Run)" result_file = 'results/without-skill.result.json' + execution_result_file = 'results/without-skill.execution-result.json' } } assumptions = @($assumptions) @@ -1275,6 +1483,7 @@ function Invoke-PrepareMode { home_directory = "$evalName/with_skill/$($runDirectoryNames.Home)" skill_directory = "$evalName/with_skill/$($runDirectoryNames.Skill)/$Skill" result = "$evalName/results/with-skill.result.json" + execution_result = "$evalName/results/with-skill.execution-result.json" } without_skill = [ordered]@{ mode = 'without_skill' @@ -1285,6 +1494,7 @@ function Invoke-PrepareMode { home_directory = "$evalName/without_skill/$($runDirectoryNames.Home)" skill_directory = $null result = "$evalName/results/without-skill.result.json" + execution_result = "$evalName/results/without-skill.execution-result.json" } } }) @@ -1297,8 +1507,18 @@ function Invoke-PrepareMode { iteration = $iterationNumber generated_utc = $generatedUtc configurations = @('with_skill', 'without_skill') - execution = 'external_handoff' + execution = 'runner_handoff' + execution_selection = [ordered]@{ + harness = $executionSelection.Harness + runner = $executionSelection.Runner + model = $executionSelection.Model + preset = $executionSelection.Preset + } runner_prompt = 'RUN-THIS.prompt.md' + execution_profile = 'execution-profile.json' + runner_protocol = $runnerProtocolSchema + runner_tools = $evalRunnerToolRelativePath + execution_result_schema = $executionResultSchema report = [ordered]@{ tool = $reportToolRelativePath template = 'tools/eval-report-template.html' @@ -1328,7 +1548,7 @@ function Invoke-PrepareMode { 'fresh context', 'isolated HOME/config', 'isolated CWD', - 'filesystem sandbox', + 'staged filesystem/workspace boundary', 'candidate skill exposure', 'transcript capture' ) @@ -1345,7 +1565,20 @@ function Invoke-PrepareMode { $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals)) - Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) hermetic run package(s)." + Write-Host 'Evaluation package prepared.' + Write-Host '' + Write-Host 'Execution:' + Write-Host " Harness: $($executionSelection.Harness)" + Write-Host " Runner: $($executionSelection.Runner)" + Write-Host " Model: $($executionSelection.Model)" + if (-not [string]::IsNullOrWhiteSpace([string]$executionSelection.Preset)) { + Write-Host " Preset: $($executionSelection.Preset)" + } + Write-Host '' + Write-Host "Cases: $($manifestEvals.Count)" + Write-Host "Arms: $($manifestEvals.Count * 2)" + Write-Host '' + Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) isolated run package(s)." Write-Host "Package: $iterationDirectory" Write-Host '' Write-Host 'Every run is a self-contained directory: repo/ is the working tree, home/ is an isolated' @@ -1358,8 +1591,9 @@ function Invoke-PrepareMode { Write-Host 'Point the harness at that path. Do not reproduce its contents in chat: a pasted copy' Write-Host 'loses the absolute paths it depends on, and the harness then cannot find the package.' Write-Host '' - Write-Host 'The runner makes the selected agent the evaluator, grader, and report producer. It must create' - Write-Host 'one isolated fresh worker per run, then grade the collected results and generate both report artifacts.' + Write-Host 'The runner-aware handoff uses execution-profile.json and the package-local Eval Runner protocol.' + Write-Host 'The selected external orchestrator must preflight and execute one runner process per blind arm,' + Write-Host 'then bridge, grade, and report the collected results.' Write-Host '' Write-Host 'This script prepared prompts only. It did not run them, and nothing here will.' Write-Host 'The selected evaluator should finish the package in one run. If it cannot write back to this package,' @@ -1375,122 +1609,53 @@ function New-RunnerPrompt { ) $builder = [System.Text.StringBuilder]::new() - [void]$builder.AppendLine('# Run, grade, and report this evaluation package') - [void]$builder.AppendLine() - [void]$builder.AppendLine('START NOW. You are the evaluator, grader, and report producer for this package. Do not ask me which role to perform, whether to run the workers, or whether to continue to grading. Execute the complete workflow in this one handoff and return the finished report.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Do not execute evaluation prompts in the current agent context. Create an isolated fresh-context worker for every run, then continue in this coordinator context with grading and report generation after all workers finish.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Everything you need is in this directory:') - [void]$builder.AppendLine() - [void]$builder.AppendLine(" $IterationDirectory") + $profilePath = Join-Path $IterationDirectory 'execution-profile.json' + $resolverPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/resolve-runner.ps1" + $bridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-execution-result.ps1" + $reportPath = Join-Path $IterationDirectory $reportToolRelativePath + [void]$builder.AppendLine('# Execute, grade, and report this evaluation package') [void]$builder.AppendLine() - [void]$builder.AppendLine("This file is `"$(Join-Path $IterationDirectory 'RUN-THIS.prompt.md')`". Use that package path as authoritative. If the package is inaccessible, report the concrete access failure; do not ask the user to choose another workflow.") + [void]$builder.AppendLine('START NOW. You are the external Eval Orchestrator for this user-directed handoff. Complete execution, deterministic grading, optional judgement, aggregation, and reporting in this run. Do not execute evaluation prompts in the current agent context.') [void]$builder.AppendLine() - [void]$builder.AppendLine("It holds $(@($ManifestEvals).Count) eval case(s), each with a `with_skill` and `without_skill` run. Every run is a self-contained directory that holds a `prompt.md`, a `run.json` contract, a `repo/` working tree, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate. The runs differ only in whether that candidate skill is present. You run the experiment, grade the completed results, and generate the report.") - [void]$builder.AppendLine() - [void]$builder.AppendLine('A person selected this harness and handed you this package. That authorizes these runs, grading, aggregation, and report generation. The repository prohibition on automated model evaluation binds repository scripts and the agent that prepared the package; it does not bind the user-directed external executor completing this specific handoff.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('## Execute now') - [void]$builder.AppendLine() - [void]$builder.AppendLine('This package assumes the harness can give each run a fresh context, an isolated HOME/config root, an isolated working directory, and a filesystem sandbox scoped to the run directory. A harness that cannot provide fresh, independent sessions is incompatible with these evals. Continue without asking for role clarification, and report per run which guarantees you satisfied - fresh context, isolated HOME/config, isolated CWD, filesystem sandbox, candidate skill exposure, and transcript capture.') + [void]$builder.AppendLine("Package: $IterationDirectory") + [void]$builder.AppendLine("Profile: $profilePath") + [void]$builder.AppendLine("Runner resolver: $resolverPath") [void]$builder.AppendLine() - [void]$builder.AppendLine('## Orchestration contract') + [void]$builder.AppendLine('This is a runner-aware package. `run.json` is the existing portable one-arm contract: it defines the prompt, staged files, working directory, isolated home, candidate-skill exposure, and required isolation. `execution-profile.json` selects the runner/model/configuration and carries the execution limits. The selected runner defines how its harness satisfies the contract.') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. For every eval case, create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`. Never reuse a worker or session between runs, between cases, or between iterations.') - [void]$builder.AppendLine('2. Launch each worker from its own run directory, which is the worker''s sandbox root. Set the working directory to that run''s `repo/`, set HOME and the platform-equivalent profile and config roots to its `home/`, and confine filesystem access to the run directory. Read the run''s `run.json` for the exact contract: `workingDirectory`, `homeDirectory`, `skillDirectory`, and the fresh-context, filesystem, and home isolation flags.') - [void]$builder.AppendLine('3. Give each worker only its `prompt.md` and the files already staged in its run directory. Do not expose this runner, `manifest.json`, any `eval-metadata.json`, `comparison.md`, result files, grading criteria, expectations, the paired run, another case''s output, or any note that an experiment is underway. All of those live outside the run directory, so keeping the worker inside it keeps them hidden.') - [void]$builder.AppendLine('4. The candidate skill is already inlined in the with_skill run''s `prompt.md` and staged under its `skill/` directory. Do not load, summarize, or add it yourself. The without_skill run carries no skill instructions and no `skill/` directory; do not expose the candidate skill to that worker by any route, including a globally installed copy.') - [void]$builder.AppendLine('5. Send each `prompt.md` unchanged as the worker''s first message. The input files are already real files in the worker''s `repo/`; the worker reads and edits them there rather than from attachments.') - [void]$builder.AppendLine('6. Use the same model, version, configuration, tools, and limits for every worker. Disable persistent memory or cross-session recall. Independent runs may execute concurrently when the selected harness and token budget allow it.') - [void]$builder.AppendLine('7. Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count. When the harness exposes them, also record the shell commands, files read and written, stdout and stderr, and exit status, and which isolation guarantees you satisfied. Record refusals, questions, and failures as results. Do not retry to improve an answer.') - [void]$builder.AppendLine('8. Work only inside this package. Do not read or modify the source repository around it. Do not begin grading until every available worker has completed or failed and its result is recorded.') + [void]$builder.AppendLine('A human selected the external orchestrator and authorized this handoff. Repository preparation, validation, CI, hooks, and automatic completion gates remain model-free. Do not substitute a generic worker, another runner, or an improvised isolation scheme if the selected runner is unavailable or incompatible.') [void]$builder.AppendLine() - [void]$builder.AppendLine('For each case in `manifest.json`, the `runs.with_skill` and `runs.without_skill` entries give each run''s directory, its `prompt`, its `run_manifest` (`run.json`), and the `result` file to write. Run the two prompts in separate workers, then overwrite the matching result file without reading its existing contents. A partial package is valid: record every completed run, continue to grading/reporting, and mark missing arms honestly instead of asking what to do next.') + [void]$builder.AppendLine('## Phase 1: execute blind arms') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Result shape') + [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` or `model` is null, unavailable, or unsupported, fail clearly and list the supported package-local runner IDs; do not guess a default. The profile contains no credentials.') + [void]$builder.AppendLine('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and validate its protocol, descriptor, and capability declarations before running an arm. Do not invent harness-specific CLI commands.') + [void]$builder.AppendLine('3. For every eval case, use the exact `run_manifest` path from `manifest.json` and the same profile path. Preflight each arm, then invoke the runner exactly once with `execute`. The runner receives only `run.json` and `execution-profile.json`; it must never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback.') + [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same runner-native model selector, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and the staged filesystem/workspace boundary.') + [void]$builder.AppendLine('5. Save the runner''s single normalized JSON response unchanged as the matching `execution_result` path. Preserve the complete final response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') + [void]$builder.AppendLine('6. If the runner cannot satisfy a required guarantee, keep the normalized status `incompatible` and stop that arm. Never fall back to the old generic isolated-worker behavior and never substitute a different runner.') [void]$builder.AppendLine() - [void]$builder.AppendLine('```json') - [void]$builder.AppendLine('{') - [void]$builder.AppendLine(' "schema": "codebeltnet/agentic/eval-result/2",') - [void]$builder.AppendLine(" `"iteration`": $IterationNumber,") - [void]$builder.AppendLine(' "eval_id": 1,') - [void]$builder.AppendLine(' "eval_name": "the directory name",') - [void]$builder.AppendLine(' "configuration": "with_skill",') - [void]$builder.AppendLine(' "model": "the exact model id you used",') - [void]$builder.AppendLine(' "provider": "who served it",') - [void]$builder.AppendLine(' "harness": "what you are",') - [void]$builder.AppendLine(' "executed_utc": "2026-01-01T00:00:00Z",') - [void]$builder.AppendLine(' "output": "the complete response the run produced",') - [void]$builder.AppendLine(' "output_files": ["paths of any files the run wrote"],') - [void]$builder.AppendLine(' "transcript": "the complete worker transcript when the harness exposes it",') - [void]$builder.AppendLine(' "shell_commands": ["commands the run executed, when exposed"],') - [void]$builder.AppendLine(' "files_read": ["paths the run read, when exposed"],') - [void]$builder.AppendLine(' "files_written": ["paths the run wrote, when exposed"],') - [void]$builder.AppendLine(' "exit_status": 0,') - [void]$builder.AppendLine(' "duration_seconds": 12.5,') - [void]$builder.AppendLine(' "total_tokens": 1234,') - [void]$builder.AppendLine(' "tool_calls": 6,') - [void]$builder.AppendLine(' "turns": 12,') - [void]$builder.AppendLine(' "base_input_tokens": 27,') - [void]$builder.AppendLine(' "output_tokens": 3800,') - [void]$builder.AppendLine(' "cache_read_tokens": 515605,') - [void]$builder.AppendLine(' "cache_write_1h_tokens": 129582,') - [void]$builder.AppendLine(' "estimated_cost_usd": 2.27,') - [void]$builder.AppendLine(' "model_effort": "high",') - [void]$builder.AppendLine(' "isolation": {') - [void]$builder.AppendLine(' "fresh_context": true,') - [void]$builder.AppendLine(' "isolated_home": true,') - [void]$builder.AppendLine(' "isolated_cwd": true,') - [void]$builder.AppendLine(' "filesystem_sandbox": true,') - [void]$builder.AppendLine(' "candidate_skill_exposed": true,') - [void]$builder.AppendLine(' "transcript_captured": true') - [void]$builder.AppendLine(' },') - [void]$builder.AppendLine(' "grading": [],') - [void]$builder.AppendLine(' "notes": "anything that would change how this result reads"') - [void]$builder.AppendLine('}') + [void]$builder.AppendLine('The package-local process surface is:') + [void]$builder.AppendLine('```text') + [void]$builder.AppendLine("pwsh -NoProfile -File `"$resolverPath`" ") + [void]$builder.AppendLine('runner.ps1 describe') + [void]$builder.AppendLine("runner.ps1 preflight -Run `"`" -Profile `"$profilePath`"") + [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`"") [void]$builder.AppendLine('```') + [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. The runner command must be invoked once per arm and its stdout must remain one JSON execution result.') [void]$builder.AppendLine() - [void]$builder.AppendLine('`transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls`, the optional efficiency telemetry fields, and every `isolation` flag are optional. Include each when the harness exposes it and omit it otherwise. Never estimate a missing value. For `with_skill`, set `isolation.candidate_skill_exposed` to how the skill actually reached the worker.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('`configuration` is `with_skill` or `without_skill` and must match the prompt you ran. Read `eval_id` and `eval_name` from `manifest.json`; do not send them to the worker. Put the full model response in `output`. If it is very long, write it beside the result file and list that path in `output_files` with a summary in `output`.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('`output` is the model''s message in full, including questions, caveats, explanations, or a refusal. Tool output is evidence from the run, not a replacement for the model response. Put the full worker event history in `transcript` when the harness exposes it.') + [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, or paired output during Phase 1. Those files remain outside every run directory and are the grading key.') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Grade and report immediately') + [void]$builder.AppendLine('## Phase 2: bridge, grade, and report') [void]$builder.AppendLine() - [void]$builder.AppendLine('After all available workers finish, read each eval''s `eval-metadata.json`. Only now may you read `expected_output` and `assertions`; they are the grading key and were intentionally hidden from the workers.') - [void]$builder.AppendLine('1. Grade every completed result against every assertion. Use deterministic checks for mechanical assertions and concrete output, transcript, and file evidence for process assertions. Use judgement only where the assertion is genuinely qualitative, and say so in the evidence. Never infer a tool or file action from the model''s self-report when process evidence is absent.') - [void]$builder.AppendLine('2. Write grading back into the matching result file using exactly `grading[].text`, `grading[].passed`, and `grading[].evidence`. Use `passed: null` when an assertion cannot be judged from captured evidence. Do not grade a missing run as passed.') - [void]$builder.AppendLine('The package carries Anthropic skill-creator under `tools/skill-creator`. Use `tools/skill-creator/agents/grader.md` for grading guidance, and use `tools/skill-creator/scripts/aggregate_benchmark.py` plus `tools/skill-creator/eval-viewer/generate_review.py` as the source-of-truth aggregation and review tools.') - $reportCommand = 'pwsh -NoProfile -File "' + (Join-Path $IterationDirectory $reportToolRelativePath) + '" -IterationDirectory "' + $IterationDirectory + '"' - [void]$builder.AppendLine(('3. Run the package report adapter now; do not ask the user to run a second command: ' + $reportCommand + '. It stages the recorded results into the upstream skill-creator workspace contract, invokes the exact upstream aggregator and static viewer, then writes the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root.')) - [void]$builder.AppendLine('4. If the harness can open local files, open `report.html` after it is written. Otherwise return its absolute path as the primary artifact. Do not wait for browser feedback before finishing the handoff.') + [void]$builder.AppendLine('1. After every available arm has completed or failed, validate each `execution-result.json` and run the package bridge. The bridge checks prompt/run/profile hashes and artifact confinement, then writes the existing `eval-result/2` file while preserving unavailable telemetry as null/unavailable.') + [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $bridgePath + '" -Run "" -ExecutionResult "" -Result ""`')) + [void]$builder.AppendLine('2. Only now read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') + [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') + [void]$builder.AppendLine(('4. Run the existing package report adapter now: `pwsh -NoProfile -File "' + $reportPath + '" -IterationDirectory "' + $IterationDirectory + '"`. It remains the bridge to Anthropic skill-creator''s grader-compatible aggregator and viewer; do not replace it with harness-specific reporting. The packaged compatibility tools remain `scripts/aggregate_benchmark.py` and `eval-viewer/generate_review.py`.')) [void]$builder.AppendLine() - [void]$builder.AppendLine('The report is the completion artifact. Do not stop after worker execution, do not return a prose-only recap, and do not ask whether grading or HTML generation is wanted.') + [void]$builder.AppendLine('The completion artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root. Return their absolute paths, completed and missing arm counts, runner/model identity, and a concise evidence-backed summary. If the package cannot be written from the external environment, return one paste-ready block containing the completed result objects and report artifacts.') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Final handoff') - [void]$builder.AppendLine() - [void]$builder.AppendLine('The finished artifacts are the first-party paired review and the exact upstream skill-creator viewer report, not a request for another command. If you can write to the package machine, leave every result, grading field, `benchmark.json`, `benchmark.md`, `report.html`, and `skill-creator-report.html` in place. Return the absolute first-party report path, the completed/expected run count, any missing arms, the model/provider, and a concise quality summary.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('If you cannot write to the package machine, return one fenced JSON block containing every completed result object, including its `grading` array, plus the generated report as an artifact when the harness supports file handoff. Do not return separate blocks or a human summary in place of the result objects. State any missing arms and the concrete artifact-transfer limitation.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('If you cannot write to that machine - a different product, a browser, a sandbox that shares no disk with it - the results have to travel as text. End with one fenced block, and say plainly that it is meant to be pasted into the repository session as-is:') - [void]$builder.AppendLine() - [void]$builder.AppendLine('```') - [void]$builder.AppendLine('Eval results, grading, and report artifact.') - [void]$builder.AppendLine("Package: $IterationDirectory") - [void]$builder.AppendLine('Model: via , harness ') - [void]$builder.AppendLine() - [void]$builder.AppendLine('') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Still unfilled: ') - [void]$builder.AppendLine('```') - [void]$builder.AppendLine() - [void]$builder.AppendLine('One block covering everything you ran, not one per case, and the outputs go in it verbatim - a summary written for a human to skim cannot be graded against assertions.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('The repository collector is only a fallback when result files were transferred without the report artifacts: `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `. It validates the returned files and invokes the same packaged skill-creator aggregator and viewer; it is not the normal next step after this prompt.') - + [void]$builder.AppendLine("This package contains $(@($ManifestEvals).Count) eval case(s), each with paired `with_skill` and `without_skill` runs. The human reviewer remains the final evaluator.") return $builder.ToString() } @@ -1505,7 +1670,7 @@ function New-PackageReadme { $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. You choose the harness, provider, and model; the selected external evaluator runs both configurations, grades them, and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator runs both configurations, grades them, and generates the report.') [void]$builder.AppendLine() [void]$builder.AppendLine('## What is here') [void]$builder.AppendLine() @@ -1513,23 +1678,23 @@ function New-PackageReadme { [void]$builder.AppendLine("- ``$($entry.eval_name)/`` - eval $($entry.eval_id)") } [void]$builder.AppendLine() - [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two hermetic run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them.') + [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two isolated run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker that stays within its run directory is never handed them.') + [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how.') [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation model') [void]$builder.AppendLine() [void]$builder.AppendLine('The package guarantees what a generator can: identical materialized repositories for both runs, the candidate skill staged only under `with_skill/skill/`, an empty isolated `home/` per run, and a `run.json` that names only paths inside the run directory. Fixture and skill hashes are recorded so you can prove what each worker received.') [void]$builder.AppendLine() - [void]$builder.AppendLine('The harness must supply the rest at runtime: a fresh context per run, the run directory as the working and config root (working directory `repo/`, HOME `home/`), and a filesystem sandbox that keeps the worker inside its run directory so global skills, global config, the source repository, the paired run, and the grading key stay out of reach. Prompt wording alone does not enforce this; the sandbox does.') + [void]$builder.AppendLine('The harness must supply the rest at runtime: a fresh context per run, the run directory as the working and config root (working directory `repo/`, HOME `home/`), controlled candidate-skill exposure, prompt fidelity, and sufficient response capture. Because global skills, global config, the source repository, the paired run, and the grading key are never staged inside a run directory, a worker that stays within its run directory is not handed them. The selected runner enforces these controls - prompt wording alone does not - and reports strict confidence when it also proves hard filesystem confinement or pragmatic confidence when the mandatory controls hold without it. Hard filesystem confinement is an added confidence signal, not a prerequisite, so Windows and other hosts without a hard sandbox run in pragmatic mode.') [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Pick one model and configuration. Use the same one for every run in this iteration.') - [void]$builder.AppendLine('2. For each eval, launch a fresh worker for `with_skill/` with its run directory as the sandbox root, `repo/` as the working directory, and `home/` as HOME. Send `prompt.md` as the first message. Read `run.json` for the contract.') - [void]$builder.AppendLine('3. Launch a second fresh worker for `without_skill/` the same way. Never reuse a worker between runs.') - [void]$builder.AppendLine('4. Save each response into the matching file under the eval''s `results/` directory, grade every completed result using the packaged `agents/grader.md` guidance, and run `tools/generate-eval-report.ps1`. The resulting `report.html` is the first-party side-by-side review; `skill-creator-report.html` is the exact upstream viewer.') + [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, for every arm.') + [void]$builder.AppendLine('2. Invoke `execute` exactly once for each `run.json`, preserving its one JSON `execution-result.json` unchanged. The runner must provide a fresh process/session, isolated home/config, isolated CWD, baseline skill exclusion, prompt fidelity, model/configuration lock, and complete response capture, or return `incompatible`. Hard filesystem confinement is not one of these mandatory controls: when the runner proves it the run reports strict isolation, and when it does not the run reports pragmatic isolation.') + [void]$builder.AppendLine('3. After all arms complete or fail, run `tools/eval-runners/bridge-execution-result.ps1` for each raw result. It writes the existing `eval-result/2` file and preserves explicit unavailable telemetry. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1`.') [void]$builder.AppendLine() - [void]$builder.AppendLine('`RUN-THIS.prompt.md` turns a harness that can create isolated workers or sessions into the evaluator, grader, and report producer. It reads the package, creates one new worker per run from its run directory, keeps runner instructions and grading data out of every worker, records results, grades after collection, and invokes Anthropic skill-creator''s aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') + [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, invokes the common protocol once per blind arm, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. `-CollectResults` accepts a partial iteration and reports unfilled runs as missing.') [void]$builder.AppendLine() @@ -1539,7 +1704,7 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('Fill in each `results/*.result.json`:') [void]$builder.AppendLine() - [void]$builder.AppendLine('- `model`, `provider`, `harness` - what actually ran it, as specifically as you know') + [void]$builder.AppendLine('- `model`, `harness` - what actually ran it, as specifically as you know') [void]$builder.AppendLine('- `executed_utc` - when') [void]$builder.AppendLine('- `output` - the produced output, or a summary plus paths in `output_files`') [void]$builder.AppendLine('- `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls` - include the values the harness exposes; omit unavailable values rather than estimating them') @@ -1686,10 +1851,23 @@ function Invoke-CollectMode { $evalDirectory = Join-Path $iterationDirectory $entry.directory $metadata = [System.IO.File]::ReadAllText((Join-Path $evalDirectory 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json $observed = @{} + $runnerAware = $manifest.PSObject.Properties.Name -contains 'execution_profile' + $bridgePath = if ($runnerAware) { Join-Path $iterationDirectory ($manifest.runner_tools + '/bridge-execution-result.ps1') } else { $null } foreach ($configuration in @('with_skill', 'without_skill')) { $fileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } $resultPath = Join-Path (Join-Path $evalDirectory 'results') $fileName + $runEntry = Get-JsonProperty -Object $entry.runs -Name $configuration -Default $null + $rawRelative = Get-JsonProperty -Object $runEntry -Name 'execution_result' -Default $null + $rawPath = if ([string]::IsNullOrWhiteSpace([string]$rawRelative)) { $null } else { Join-Path $iterationDirectory $rawRelative } + $runPath = Join-Path $iterationDirectory (Get-JsonProperty -Object $runEntry -Name 'run_manifest' -Default '') + if ($runnerAware -and $null -ne $rawPath -and (Test-Path -LiteralPath $rawPath -PathType Leaf)) { + $bridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $runPath -ExecutionResult $rawPath -Result $resultPath 2>&1 + if ($LASTEXITCODE -ne 0) { + $errors.Add("$($entry.eval_name)/$configuration - execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))") + continue + } + } if (-not (Test-Path -LiteralPath $resultPath)) { $warnings.Add("$($entry.eval_name)/$configuration - no result file at results/$fileName.") continue @@ -1713,7 +1891,9 @@ function Invoke-CollectMode { $outputText = [string](Get-JsonProperty -Object $result -Name 'output' -Default '') $outputFiles = @(Get-JsonProperty -Object $result -Name 'output_files' -Default @()) - $hasOutput = -not [string]::IsNullOrWhiteSpace($outputText) -or $outputFiles.Count -gt 0 + $executionStatus = [string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '') + $hasExecution = -not [string]::IsNullOrWhiteSpace($executionStatus) -and $executionStatus -ne 'unrun' + $hasOutput = -not [string]::IsNullOrWhiteSpace($outputText) -or $outputFiles.Count -gt 0 -or $hasExecution if (-not $hasOutput) { $warnings.Add("$($entry.eval_name)/$configuration - not run yet (empty output and no output_files).") continue @@ -1748,13 +1928,13 @@ function Invoke-CollectMode { $observed[$configuration] = [pscustomobject]@{ Model = $model - Provider = [string](Get-JsonProperty -Object $result -Name 'provider' -Default '') Graded = $graded.Count Passed = $passed Total = $total TranscriptRecorded = -not [string]::IsNullOrWhiteSpace($transcriptText) ProcessEvidence = $hasProcessEvidence IsolationReport = $isolationReport + ExecutionStatus = $executionStatus DurationSeconds = Get-JsonProperty -Object $result -Name 'duration_seconds' TotalTokens = Get-JsonProperty -Object $result -Name 'total_tokens' ToolCalls = Get-JsonProperty -Object $result -Name 'tool_calls' @@ -1815,7 +1995,7 @@ function Invoke-CollectMode { [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation reported') [void]$builder.AppendLine() - [void]$builder.AppendLine('Flags each run''s harness confirmed: fresh context, isolated home, isolated cwd, filesystem sandbox, candidate skill exposure, transcript capture (Y/N, ? unknown). Process-dependent assertions are only gradeable from a run with process evidence.') + [void]$builder.AppendLine('Flags each run''s harness confirmed: fresh context, isolated home, isolated cwd, filesystem confinement (strict when a hard sandbox is proven, otherwise pragmatic), candidate skill exposure, transcript capture (Y/N, ? unknown). Process-dependent assertions are only gradeable from a run with process evidence.') [void]$builder.AppendLine() [void]$builder.AppendLine('| Eval | Configuration | Isolation | Process evidence |') [void]$builder.AppendLine('| --- | --- | --- | --- |') diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index f66755c..163ca4a 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1191,6 +1191,7 @@ Add-ValidationResult -Results $results -Name 'Repository automation cannot launc $automationPaths = @($automationPaths | Where-Object { $normalized = $_ -replace '\\', '/' ($normalized.StartsWith('scripts/') -or $normalized.StartsWith('.github/')) -and + -not $normalized.StartsWith('scripts/eval-runners/') -and $normalized -ne 'scripts/validate-skill-templates.ps1' -and $automationExtensions -contains [System.IO.Path]::GetExtension($normalized).ToLowerInvariant() } | Sort-Object -Unique) @@ -1231,10 +1232,29 @@ Add-ValidationResult -Results $results -Name 'Repository automation cannot launc Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'this repository does not provide an opt-in path around that rule.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Model-backed comparisons are not a repository completion gate.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'This rule is Priority 1.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner' Assert-Contains -Name 'README.md' -Content $readme -Needle 'There is no repository opt-in switch.' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Eval Runner' Assert-Contains -Name 'README.md' -Content $readme -Needle 'validate-skill-templates.ps1 -MetadataOnly' } +Add-ValidationResult -Results $results -Name 'Eval Runner protocol conformance remains deterministic' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $conformancePath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-runner-conformance.ps1' + if (-not (Test-Path -LiteralPath $conformancePath -PathType Leaf)) { + throw 'The Eval Runner conformance suite is missing.' + } + $conformanceOutput = & pwsh -NoProfile -File $conformancePath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Eval Runner conformance failed: $($conformanceOutput -join [Environment]::NewLine)" + } + if (@($conformanceOutput -join [Environment]::NewLine) -notmatch 'Eval Runner conformance:\s+PASS') { + throw 'Eval Runner conformance did not report PASS.' + } +} + Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Action { $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref @@ -1249,7 +1269,10 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'the same model, the same version, and the same configuration' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'a baseline handed the answer key is not a baseline' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' - Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Cline and OpenCode discovery is free-only' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Handing the package over' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Executing a package you were handed' @@ -1264,7 +1287,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'The user asked for eval results, not a second workflow decision.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Asking for an eval' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`eval `, `evaluate `' - Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Run the script immediately when asked. Do not reply with a plan, a menu of options' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Eval preparation is a completion gate' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Adding or modifying any repo-managed skill triggers this workflow.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' @@ -1273,7 +1296,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' - Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'Before running the script, choose a Harness + Model.' Assert-NotContains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'run-skill-benchmark.ps1' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'Eval packages inside this repository must live under .bot/.' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'git does not ignore it' @@ -1286,10 +1310,98 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $scriptPath = Join-Path $repoRoot 'scripts/prepare-skill-evals.ps1' + $modelDiscoveryPath = Join-Path $repoRoot 'scripts/Get-HarnessModels.ps1' $packageRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-package-' + [Guid]::NewGuid().ToString('N')) $taskMarker = "`n# Task`n" try { - $prepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot 2>&1 + $catalogPath = Join-Path $packageRoot 'fake-model-catalog.json' + New-Item -ItemType Directory -Path $packageRoot -Force | Out-Null + [System.IO.File]::WriteAllText($catalogPath, (@' +{ + "models": [ + { "id": "claude-haiku-4.5", "display_name": "Claude Haiku 4.5", "availability": "paid", "operation": "language" }, + { "id": "gpt-5.6-luna", "display_name": "GPT-5.6 Luna", "availability": "paid", "operation": "language" }, + { "id": "deepseek/deepseek-v4-flash", "display_name": "DeepSeek V4 Flash", "availability": "free", "operation": "language" }, + { "id": "paid-model", "display_name": "Paid Model", "availability": "paid", "operation": "language" }, + { "id": "unknown-model", "display_name": "Unknown Model", "availability": "unknown", "operation": "language" }, + { "id": "opencode/muse-spark-1.2-contributor-free", "display_name": "Muse Spark 1.2", "cost": { "input": 0, "output": 0, "cache": { "read": 0, "write": 0 } }, "operation": "language" } + ] +} +'@), $utf8NoBom) + + $copilotDiscovery = (& pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $catalogPath 2>&1) + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for Copilot fixture: $($copilotDiscovery -join [Environment]::NewLine)" } + $copilotModels = ($copilotDiscovery -join [Environment]::NewLine) | ConvertFrom-Json + if (@($copilotModels.models).Count -ne 6) { throw 'Copilot discovery must return all available fixture models.' } + + $codexDiscovery = (& pwsh -NoProfile -File $modelDiscoveryPath -Runner 'codex' -CatalogPath $catalogPath 2>&1) + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for Codex fixture: $($codexDiscovery -join [Environment]::NewLine)" } + $codexModels = ($codexDiscovery -join [Environment]::NewLine) | ConvertFrom-Json + if (@($codexModels.models).Count -ne 6) { throw 'Codex discovery must return all available fixture models.' } + + foreach ($runnerName in @('cline', 'opencode')) { + $discoveryOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner $runnerName -CatalogPath $catalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for ${runnerName}: $($discoveryOutput -join [Environment]::NewLine)" } + $discovery = ($discoveryOutput -join [Environment]::NewLine) | ConvertFrom-Json + $ids = @($discovery.models | ForEach-Object { [string]$_.id }) + if ($ids -notcontains 'deepseek/deepseek-v4-flash' -or $ids -notcontains 'opencode/muse-spark-1.2-contributor-free') { + throw "$runnerName discovery must retain free fixture model selectors." + } + if ($ids -contains 'paid-model' -or $ids -contains 'unknown-model') { + throw "$runnerName discovery must not include paid or unknown-availability models." + } + } + + $paidCatalogPath = Join-Path $packageRoot 'paid-model-catalog.json' + [System.IO.File]::WriteAllText($paidCatalogPath, (@' +{ + "models": [ + { "id": "paid-model", "display_name": "Paid Model", "availability": "paid", "operation": "language" }, + { "id": "unknown-model", "display_name": "Unknown Model", "availability": "unknown", "operation": "language" } + ] +} +'@), $utf8NoBom) + $noFreeOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'opencode' -CatalogPath $paidCatalogPath 2>&1 + if ($LASTEXITCODE -eq 0 -or ($noFreeOutput -join ' ') -notmatch 'No free OpenCode models') { + throw 'OpenCode discovery must fail clearly when free discovery returns zero models.' + } + + $missingCatalogOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'codex' -CatalogPath (Join-Path $packageRoot 'missing-catalog.json') 2>&1 + if ($LASTEXITCODE -eq 0 -or ($missingCatalogOutput -join ' ') -notmatch 'does not exist') { + throw 'Discovery failures must remain local and must not invent fallback models.' + } + + $referenceOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $catalogPath -RequireModel 'claude-haiku-4.5' 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Codebelt Reference fixture should resolve: $($referenceOutput -join [Environment]::NewLine)" } + $missingReferenceOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $paidCatalogPath -RequireModel 'claude-haiku-4.5' 2>&1 + if ($LASTEXITCODE -eq 0 -or ($missingReferenceOutput -join ' ') -notmatch 'Required model') { + throw 'Codebelt Reference discovery must fail instead of silently substituting a model.' + } + + $referencePackageRoot = Join-Path $packageRoot 'reference-package' + $referencePrepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $referencePackageRoot -CodebeltReference -ModelCatalogPath $catalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -CodebeltReference failed against the fake current catalog: $($referencePrepareOutput -join [Environment]::NewLine)" + } + $referenceProfile = [System.IO.File]::ReadAllText((Join-Path $referencePackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$referenceProfile.runner -ne 'github-copilot' -or [string]$referenceProfile.model -ne 'claude-haiku-4.5') { + throw 'Codebelt Reference preparation must write github-copilot + claude-haiku-4.5 atomically.' + } + + $missingSelectionRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-missing-selection-' + [Guid]::NewGuid().ToString('N')) + $missingSelectionOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $missingSelectionRoot 2>&1 + if ($LASTEXITCODE -eq 0) { + throw 'prepare-skill-evals.ps1 must refuse to generate RUN-THIS.prompt.md without a resolved runner/model selection.' + } + if (($missingSelectionOutput -join ' ') -notmatch 'requires a resolved Harness \+ Model') { + throw 'prepare-skill-evals.ps1 must explain that Harness + Model selection is required before handoff generation.' + } + if (Test-Path -LiteralPath $missingSelectionRoot) { + Remove-Item -LiteralPath $missingSelectionRoot -Recurse -Force + throw 'prepare-skill-evals.ps1 must not create an unresolved eval package.' + } + + $prepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed: $($prepareOutput -join [Environment]::NewLine)" } @@ -1323,7 +1435,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } foreach ($isolationField in @('fresh_context_required', 'isolated_home_required', 'isolated_cwd_required')) { if (-not [bool]$manifest.isolation.$isolationField) { - throw "manifest.isolation.$isolationField must be true so a harness knows the run is hermetic." + throw "manifest.isolation.$isolationField must be true so a harness knows the run requires an isolated context." } } @@ -1333,17 +1445,15 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $runner = [System.IO.File]::ReadAllText($runnerPath, $utf8NoBom) foreach ($needle in @( - 'START NOW. You are the evaluator, grader, and report producer', + 'START NOW. You are the external Eval Orchestrator', 'Do not execute evaluation prompts in the current agent context.', - 'create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`', - 'Never reuse a worker or session between runs', - 'Do not expose this runner', - 'The candidate skill is already inlined in the with_skill run', - 'Launch each worker from its own run directory', - 'Use the same model, version, configuration, tools, and limits for every worker.', - 'Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count.', - 'Do not begin grading until every available worker has completed or failed', - '## Grade and report immediately', + 'execution-profile.json` selects the runner/model/configuration', + 'invoke the runner exactly once with `execute`', + 'never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback', + 'Never fall back to the old generic isolated-worker behavior', + 'execution-result.json', + 'The bridge checks prompt/run/profile hashes and artifact confinement', + 'Only now read each eval''s `eval-metadata.json`', 'grading[].text', 'tools/skill-creator/agents/grader.md', 'scripts/aggregate_benchmark.py', @@ -1357,10 +1467,13 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable foreach ($forbidden in @( 'this context has read the runner instructions and can no longer produce a clean run', '## If you can only hold one context', - 'If you truly cannot, this package is not for you' + 'If you truly cannot, this package is not for you', + 'Choose evaluation configuration', + 'Codebelt Reference', + 'discover current models' )) { if ($runner.Contains($forbidden)) { - throw "RUN-THIS.prompt.md must not contain the refusal path '$forbidden'." + throw "RUN-THIS.prompt.md must not contain forbidden handoff text '$forbidden'." } } foreach ($entry in @($manifest.evals)) { @@ -1381,6 +1494,9 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable throw "$($entry.eval_name)/$configuration manifest entry must declare '$pathProperty'." } } + if ($run.PSObject.Properties.Name -notcontains 'execution_result') { + throw "$($entry.eval_name)/$configuration manifest entry must declare 'execution_result'." + } foreach ($mustExist in @($run.prompt, $run.run_manifest, $run.working_directory, $run.home_directory)) { if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory $mustExist))) { throw "$($entry.eval_name)/$configuration manifest path '$mustExist' does not exist." @@ -1411,14 +1527,14 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } } - # 10. run.json requires fresh context and isolation; 6/7. it references nothing outside the run package. + # 10. run.json requires fresh context, the staged workspace boundary, and isolated home; 6/7. it references nothing outside the run package. $withRunJson = [System.IO.File]::ReadAllText((Join-Path $withRunDir 'run.json'), $utf8NoBom) $withoutRunJson = [System.IO.File]::ReadAllText((Join-Path $withoutRunDir 'run.json'), $utf8NoBom) $withRun = $withRunJson | ConvertFrom-Json $withoutRun = $withoutRunJson | ConvertFrom-Json foreach ($run in @($withRun, $withoutRun)) { if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { - throw "$($entry.eval_name) run.json must require fresh context, filesystem, and home isolation." + throw "$($entry.eval_name) run.json must require fresh context, the staged workspace boundary, and isolated home." } if ([string]$run.workingDirectory -ne 'repo' -or [string]$run.homeDirectory -ne 'home') { throw "$($entry.eval_name) run.json must set workingDirectory=repo and homeDirectory=home." @@ -1509,6 +1625,43 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } } + if ([string]$manifest.execution -ne 'runner_handoff' -or + [string]$manifest.execution_profile -ne 'execution-profile.json' -or + [string]$manifest.runner_protocol -ne 'codebeltnet/agentic/eval-runner-protocol/1' -or + [string]$manifest.runner_tools -ne 'tools/eval-runners' -or + [string]$manifest.execution_result_schema -ne 'codebeltnet/agentic/eval-execution-result/1') { + throw 'Runner-aware packages must declare the execution profile, runner protocol, runner tools, and execution-result schema.' + } + $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + $profile = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json + foreach ($profileField in @('schema', 'runner', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency')) { + if ($profile.PSObject.Properties.Name -notcontains $profileField) { + throw "execution-profile.json must declare '$profileField'." + } + } + if ($profile.PSObject.Properties.Name -contains 'provider') { + throw 'execution-profile.json must not declare provider.' + } + if ([string]$profile.schema -ne 'codebeltnet/agentic/eval-execution-profile/1' -or + [string]::IsNullOrWhiteSpace([string]$profile.runner) -or + [string]::IsNullOrWhiteSpace([string]$profile.model) -or + [int]$profile.timeout_seconds -lt 1 -or [int]$profile.concurrency -lt 1) { + throw 'execution-profile.json has an invalid schema or execution limit.' + } + $runnerTools = [System.Collections.Generic.List[string]]::new() + foreach ($runnerTool in @('runner-common.ps1', 'resolve-runner.ps1', 'bridge-execution-result.ps1', 'contracts/execution-profile.schema.json', 'contracts/execution-result.schema.json')) { $runnerTools.Add($runnerTool) } + $runnerSourceRoot = Join-Path $repoRoot 'scripts/eval-runners' + foreach ($runnerDirectory in Get-ChildItem -LiteralPath $runnerSourceRoot -Directory -Force | Sort-Object Name) { + if (Test-Path -LiteralPath (Join-Path $runnerDirectory.FullName 'runner.ps1') -PathType Leaf) { + $runnerTools.Add("$($runnerDirectory.Name)/runner.ps1") + } + } + foreach ($runnerTool in $runnerTools) { + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory "tools/eval-runners/$runnerTool") -PathType Leaf)) { + throw "Prepared package is missing runner tool '$runnerTool'." + } + } + $collectOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 -CollectResults failed on an unrun package: $($collectOutput -join [Environment]::NewLine)" @@ -1532,7 +1685,6 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable $resultPath = Join-Path (Join-Path $firstEvalDirectory 'results') $resultFile $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json $result.model = 'validator-model' - $result.provider = 'validator-provider' $result.harness = 'validator-harness' $result.executed_utc = '2026-01-01T00:00:00Z' $result.output = 'validator output' @@ -1595,7 +1747,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $insideRepo = Join-Path $repoRoot 'agentic-eval-isolation-check' - $isolationOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo 2>&1 + $isolationOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -eq 0) { throw 'prepare-skill-evals.ps1 must refuse an output root inside this repository but outside .bot/.' } @@ -1610,7 +1762,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable # .bot/ is the sanctioned in-repository home, and it only works while git ignores it. $botRoot = Join-Path (Join-Path $repoRoot '.bot') 'agentic-eval-bot-check' try { - $botOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot 2>&1 + $botOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 must accept an output root under .bot/: $($botOutput -join [Environment]::NewLine)" } @@ -2111,6 +2263,11 @@ Add-ValidationResult -Results $results -Name 'Git keep a changelog skill updates Assert-Contains -Name 'git-keep-a-changelog/scripts/resolve-release-entity.ps1' -Content $entityResolver -Needle "'Added'" Assert-Contains -Name 'git-keep-a-changelog/scripts/resolve-release-entity.ps1' -Content $entityResolver -Needle "'Unchanged'" Assert-Contains -Name 'git-keep-a-changelog/scripts/test-resolve-release-entity.ps1' -Content $entityResolverTests -Needle "Assert-Classification -EntityPath 'skills/dotnet-test' -Expected 'Added'" + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle '### Layered Capability Classification' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Do not use a top-level directory or the first framework commit as the only release entity.' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'A child adapter absent at the resolved base and present at `HEAD` is `Added`' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Do not repeat an adapter in a parent `Added` bullet and again in `Changed`' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'A planned, blocked, or unsupported CLI/TUI/harness is not support' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Updates CHANGELOG.md directly instead of only drafting notes in chat' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Reads full commit subjects and bodies before writing the release entry' @@ -2155,6 +2312,8 @@ Add-ValidationResult -Results $results -Name 'Git summary skills reduce ranges t Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not add a Security or other section entry when the final diff contradicts the commit message claim' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not create a Changed section or Changed bullet for dotnet-test refinements made before its first release' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not preserve the earlier draft bullet as a frozen baseline that forces later refinements into `Changed`' + Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not treat the top-level scripts/eval-runners directory as the only release entity when independently selectable adapters have distinct final states' + Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not claim the planned or blocked Freebuff TUI is a supported runner or a completed Added capability' Assert-Contains -Name 'git-nuget-release-notes/SKILL.md' -Content $nugetSkill -Needle 'History is evidence; the resulting state is truth.' Assert-Contains -Name 'git-nuget-release-notes/SKILL.md' -Content $nugetSkill -Needle 'Classify each user-facing package capability from whether it existed at the resolved base' diff --git a/skills/agent-smith/SKILL.md b/skills/agent-smith/SKILL.md index 8a22b7a..ed86fb2 100644 --- a/skills/agent-smith/SKILL.md +++ b/skills/agent-smith/SKILL.md @@ -1,7 +1,7 @@ --- name: agent-smith -description: > - Use this skill to apply a rigorous, evidence-driven software-craftsmanship standard across an engineering task. Invoke explicitly as `/agent-smith task`, or automatically for architecture, implementation, refactoring, review, API compatibility and Semantic Versioning, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, governance, and engineering assessment. For .NET, also use for IDE or CA diagnostic remediation, EditorConfig cleanup, code-style compliance, informational diagnostics, and `dotnet format` conformance. Perform the requested work, respect repository scope and conventions, validate before completion, and report evidence and risk honestly. Technology-neutral core; specialist guidance loads on demand. Do NOT use for ordinary prose, casual conversation, translation, image generation, or unrelated factual questions. +description: > + Use when the user wants evidence-driven architecture, implementation, refactoring, review, API compatibility/SemVer, testing, performance, skill authoring, documentation, security/DevSecOps, CI/CD, delivery, governance, or .NET IDE/CA and EditorConfig remediation. --- # Agent Smith @@ -14,21 +14,21 @@ The name is a deliberate, understated nod to a relentless *agent* combined with Apply **one coherent engineering standard** across design, implementation, validation, documentation, delivery, and governance. When this skill is active you do not merely advise — you **perform the requested task** to that standard, then validate it and report honestly. -The standard is technology-neutral. Specialist guidance (including .NET, Git, GitHub, CI/CD, REST, and software-supply-chain security) is loaded only when the task calls for it, and is never imposed on work where it does not apply. - -## Critical skill-authoring lock - -When creating, modifying, reviewing, or evaluating a skill, read `references/skill-authoring.md` before editing. Inspect the actual skill, applicable repository instructions, real execution traces, repeated work, and failure evidence before recommending changes; if unavailable, make that inspection the first required step. Always analyze the task graph for safe parallelism and concurrency. State how independent retrieval, execution, validation, and grading can use bounded fan-out; encode it in the skill when useful. Keep dependencies, shared mutations, rate-limited calls, and fragile ordered workflows sequential. - -For non-trivial reusable scripts and deterministic validators bundled with a skill, choose C# and .NET by default in this .NET-first skill collection. Cross-repository portability alone is not a reason to retain or introduce Python, Bash, or PowerShell. Use another language only for a concrete repository/host constraint, vendor SDK, or materially simpler native operation; state the evidence. Resolve the latest supported .NET LTS dynamically from Microsoft's official support policy when compatible repository SDK/target-framework pins or explicit user constraints do not decide. Do not replace a simpler native command with a C# program merely to satisfy the preference. - -## Critical .NET conformance lock - -When the task selects .NET EditorConfig conformance mode, read both `references/dotnet.md` and `references/dotnet-editorconfig-conformance.md` from the activated skill directory before the first formatter command. Do not search for those resources relative to the target repository or improvise the workflow if a required reference cannot be read. - -For informational diagnostics, informational-or-higher conformance, and any conformance task that does not explicitly set a different minimum, every discovery, investigation, retry, and final `dotnet format` command must include both `--severity info` and `--verify-no-changes`. The formatter defaults to `warn` when `--severity` is omitted, which can hide the findings that define success. Keep the resolved severity explicit and identical throughout the remediation loop. `--no-restore` changes restore behaviour only; it never replaces either required flag or proves conformance. - -If a prior mutating formatter pass has produced `Unmerged change from project` annotations, use the bundled `scripts/repair-roslyn-multiproject-artifacts.ps1` from this skill directory. The tool detects the neutral Roslyn multi-project artifact signature and reports the structural pattern independently of diagnostic ID. It currently repairs only the proven `whole-document-namespace-conversion` pattern. Run its default check mode first and use `-Apply` only when every artifact is reported as `recoverable`; an unrecognized or differing candidate prevents all writes. Pattern-specific recovery is not permission to use the formatter in mutating mode. +The standard is technology-neutral. Specialist guidance (including .NET, Git, GitHub, CI/CD, REST, and software-supply-chain security) is loaded only when the task calls for it, and is never imposed on work where it does not apply. + +## Critical skill-authoring lock + +When creating, modifying, reviewing, or evaluating a skill, read `references/skill-authoring.md` before editing. Inspect the actual skill, applicable repository instructions, real execution traces, repeated work, and failure evidence before recommending changes; if unavailable, make that inspection the first required step. Always analyze the task graph for safe parallelism and concurrency. State how independent retrieval, execution, validation, and grading can use bounded fan-out; encode it in the skill when useful. Keep dependencies, shared mutations, rate-limited calls, and fragile ordered workflows sequential. + +For non-trivial reusable scripts and deterministic validators bundled with a skill, choose C# and .NET by default in this .NET-first skill collection. Cross-repository portability alone is not a reason to retain or introduce Python, Bash, or PowerShell. Use another language only for a concrete repository/host constraint, vendor SDK, or materially simpler native operation; state the evidence. Resolve the latest supported .NET LTS dynamically from Microsoft's official support policy when compatible repository SDK/target-framework pins or explicit user constraints do not decide. Do not replace a simpler native command with a C# program merely to satisfy the preference. + +## Critical .NET conformance lock + +When the task selects .NET EditorConfig conformance mode, read both `references/dotnet.md` and `references/dotnet-editorconfig-conformance.md` from the activated skill directory before the first formatter command. Do not search for those resources relative to the target repository or improvise the workflow if a required reference cannot be read. + +For informational diagnostics, informational-or-higher conformance, and any conformance task that does not explicitly set a different minimum, every discovery, investigation, retry, and final `dotnet format` command must include both `--severity info` and `--verify-no-changes`. The formatter defaults to `warn` when `--severity` is omitted, which can hide the findings that define success. Keep the resolved severity explicit and identical throughout the remediation loop. `--no-restore` changes restore behaviour only; it never replaces either required flag or proves conformance. + +If a prior mutating formatter pass has produced `Unmerged change from project` annotations, use the bundled `scripts/repair-roslyn-multiproject-artifacts.ps1` from this skill directory. The tool detects the neutral Roslyn multi-project artifact signature and reports the structural pattern independently of diagnostic ID. It currently repairs only the proven `whole-document-namespace-conversion` pattern. Run its default check mode first and use `-Apply` only when every artifact is reported as `recoverable`; an unrecognized or differing candidate prevents all writes. Pattern-specific recovery is not permission to use the formatter in mutating mode. ## Activation and invocation @@ -74,16 +74,16 @@ A task may select **multiple** modes. Load core principles for every invocation, | Mode | Use when the task involves | Load | |------|----------------------------|------| | Architecture | system design, boundaries, distributed systems, integration, DDD, CQRS, event-driven design, deployment topology, migration | `references/architecture.md` | -| API design & compatibility | public/HTTP APIs, libraries, contracts, serialization, versioning, Semantic Versioning | `references/api-design-and-compatibility.md` | -| Implementation | coding and refactoring | `references/implementation.md` | -| .NET | .NET or C# is relevant | `references/dotnet.md` | -| .NET EditorConfig conformance | the user explicitly requests EditorConfig, code-style, formatter-supported analyzer, informational IDE, or named diagnostic remediation or verification | `references/dotnet.md` + `references/dotnet-editorconfig-conformance.md` | -| Testing | test design/review, regression, functional/integration/contract testing | `references/testing.md` | +| API design & compatibility | public/HTTP APIs, libraries, contracts, serialization, versioning, Semantic Versioning | `references/api-design-and-compatibility.md` | +| Implementation | coding and refactoring | `references/implementation.md` | +| .NET | .NET or C# is relevant | `references/dotnet.md` | +| .NET EditorConfig conformance | the user explicitly requests EditorConfig, code-style, formatter-supported analyzer, informational IDE, or named diagnostic remediation or verification | `references/dotnet.md` + `references/dotnet-editorconfig-conformance.md` | +| Testing | test design/review, regression, functional/integration/contract testing | `references/testing.md` | | Performance | benchmarking, profiling, optimization, latency, throughput, allocation, scalability | `references/performance.md` | | Security & DevSecOps | identity, authorization, secrets, dependencies, pipelines, supply chain, permissions, deployment security | `references/security-and-devsecops.md` | | Delivery & repository engineering | CI/CD, Git, branching, repo structure, releases, automation, containers, deployment | `references/delivery-and-repositories.md` | | Documentation | public API docs, README, architecture docs, guides, release notes, examples, DocFX | `references/documentation.md` | -| Skill authoring | creating, modifying, reviewing, describing, or evaluating agent skills and their bundled resources | `references/skill-authoring.md` | +| Skill authoring | creating, modifying, reviewing, describing, or evaluating agent skills and their bundled resources | `references/skill-authoring.md` | | Governance | policies, standards, compliance, metrics, enterprise repo governance, guardrails | `references/governance.md` | **Load .NET guidance only when .NET or C# is actually relevant.** For non-.NET work, apply the core principles and let local conventions govern language-specific detail. @@ -92,10 +92,10 @@ A task may select **multiple** modes. Load core principles for every invocation, - **Small implementation** (`/agent-smith add validation for an optional config property`): core principles + implementation (+ platform reference if relevant) + testing. Proportional process, no architecture document. - **Benchmark assessment**: core principles + decision framework + performance + implementation + platform reference (e.g. `dotnet.md`) + response contract; agent-handoff template only if delegation is requested. -- **Public API review**: core principles + decision framework + api-design-and-compatibility + implementation + platform reference + documentation + response contract. -- **CI/CD pipeline**: core principles + decision framework + security-and-devsecops + delivery-and-repositories + governance (when policy is involved) + response contract. -- **Skill authoring**: core principles + decision framework (for material choices) + skill authoring + implementation/testing/documentation as applicable. Make concurrency and script-runtime choices explicit. -- **Scoped .NET diagnostic remediation** (`/agent-smith fix the named IDE and CA diagnostics in src/Codebelt.Core`): core principles + implementation + .NET + .NET EditorConfig conformance + testing. Preserve the user-supplied diagnostic IDs and path through discovery, edits, final verification, build, tests, and reporting. +- **Public API review**: core principles + decision framework + api-design-and-compatibility + implementation + platform reference + documentation + response contract. +- **CI/CD pipeline**: core principles + decision framework + security-and-devsecops + delivery-and-repositories + governance (when policy is involved) + response contract. +- **Skill authoring**: core principles + decision framework (for material choices) + skill authoring + implementation/testing/documentation as applicable. Make concurrency and script-runtime choices explicit. +- **Scoped .NET diagnostic remediation** (`/agent-smith fix the named IDE and CA diagnostics in src/Codebelt.Core`): core principles + implementation + .NET + .NET EditorConfig conformance + testing. Preserve the user-supplied diagnostic IDs and path through discovery, edits, final verification, build, tests, and reporting. ## Repository precedence @@ -130,7 +130,7 @@ For each material finding: Issue → Why it matters → Evidence or reasoning ## Response behaviour -**Be concise. Sacrifice grammar for the sake of concision.** Prefer clear fragments when they shorten feedback. Remain respectful and technically defensible. Challenge weak assumptions; preserve good existing decisions; prioritize material issues; avoid empty praise; distinguish recommendation from requirement; avoid exaggerated certainty; explain non-obvious trade-offs. Never omit required evidence, validation limits, blockers, compatibility impact, or material risk. +**Be concise. Sacrifice grammar for the sake of concision.** Prefer clear fragments when they shorten feedback. Remain respectful and technically defensible. Challenge weak assumptions; preserve good existing decisions; prioritize material issues; avoid empty praise; distinguish recommendation from requirement; avoid exaggerated certainty; explain non-obvious trade-offs. Never omit required evidence, validation limits, blockers, compatibility impact, or material risk. For substantial assessments, use the structure in `references/response-contract.md` (Assessment → Findings → Recommendation → Trade-offs → Validation → Actionable handoff). Do not force that structure onto every response. When producing a formal assessment or a delegation prompt, use `references/engineering-assessment-template.md` or `references/agent-handoff-template.md`. @@ -162,7 +162,7 @@ This skill must not: - turn every small task into an architecture exercise; - produce advice without completing the requested work when implementation is possible; - fabricate evidence or claim unperformed validation; -- sacrifice correctness, required evidence, or material context for terseness; +- sacrifice correctness, required evidence, or material context for terseness; - broaden the task without justification; - introduce dependencies or abstractions without demonstrating value. @@ -174,15 +174,15 @@ Load on demand, per the routing table: - `references/decision-framework.md` — structured reasoning for material decisions. - `references/architecture.md` — system design and boundaries. - `references/api-design-and-compatibility.md` — public and HTTP API contracts and versioning. -- `references/implementation.md` — coding and refactoring discipline. -- `references/dotnet.md` — .NET/C#-specific guidance (load only when relevant). -- `references/dotnet-editorconfig-conformance.md` — scope-aware, read-only `dotnet format` discovery and verification for explicit .NET EditorConfig, code-style, and supported analyzer conformance work. -- `references/testing.md` — test design and review. +- `references/implementation.md` — coding and refactoring discipline. +- `references/dotnet.md` — .NET/C#-specific guidance (load only when relevant). +- `references/dotnet-editorconfig-conformance.md` — scope-aware, read-only `dotnet format` discovery and verification for explicit .NET EditorConfig, code-style, and supported analyzer conformance work. +- `references/testing.md` — test design and review. - `references/performance.md` — benchmarking, profiling, optimization. - `references/security-and-devsecops.md` — identity, secrets, dependencies, pipelines, supply chain. - `references/delivery-and-repositories.md` — CI/CD, Git, releases, repository engineering. - `references/documentation.md` — documentation as part of the product. -- `references/skill-authoring.md` — skill design, parallelism, .NET-first scripts, descriptions, and eval loops. +- `references/skill-authoring.md` — skill design, parallelism, .NET-first scripts, descriptions, and eval loops. - `references/governance.md` — policies, standards, and metrics (Intent → Drivers → Metrics → Actions). - `references/response-contract.md` — review severity, finding shape, and assessment structure. - `references/engineering-assessment-template.md` — fill-in template for a formal assessment. diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index 8d35f41..be8bc2c 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-benchmark description: > - Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects source and usage evidence, ranks high-value operations instead of every public member, selects representative workloads, rejects misleading microbenchmarks, creates or reuses the tuning/ and tooling/ harness, preflights existing-report skips, semantic-preflights workload correctness, validates discovery, and keeps full runs human-initiated. When the user says yolo, it auto-accepts routine defaults and proceeds through safe validation without confirmation churn. + Use when the user wants to design, author, review, or diagnose BenchmarkDotNet experiments for .NET code, compare implementations, measure allocations or contention, or benchmark a specific type or method. Also use to judge whether profiling or load testing is the better instrument. --- # Evidence-Driven .NET Benchmarking diff --git a/skills/dotnet-change-impact/SKILL.md b/skills/dotnet-change-impact/SKILL.md index 47e5b64..be53457 100644 --- a/skills/dotnet-change-impact/SKILL.md +++ b/skills/dotnet-change-impact/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-change-impact -description: > - Classifies .NET library or NuGet package changes and recommends the correct release bump: Major, Minor, or Patch. Applies both Semantic Versioning (MAJOR.MINOR.PATCH) and .NET assembly/file versioning (Major.Minor.Build.Revision), grounded in Microsoft’s official .NET library compatibility rules. Use when evaluating the current branch, breaking changes, API diffs, public API changes, dependency updates, TFM/platform support, interface or enum changes, overloads, analyzers, source generators, or binary/source/behavioral/design-time/backwards compatibility. When no explicit change details or compare range are provided, inspects the current Git branch and compares it against the upstream default branch automatically. Always returns structured compatibility reasoning with the recommendation. +description: > + Use when the user wants a Major, Minor, or Patch recommendation for .NET library or NuGet package changes, including API diffs, behavior changes, dependencies, target frameworks, analyzers, or source generators, based on compatibility impact. --- # .NET Change Impact diff --git a/skills/dotnet-docfx-digest/SKILL.md b/skills/dotnet-docfx-digest/SKILL.md index 9fa6aad..3ec78d7 100644 --- a/skills/dotnet-docfx-digest/SKILL.md +++ b/skills/dotnet-docfx-digest/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-docfx-digest description: > - Create and maintain developer-friendly DocFX documentation digests for .NET public APIs: repo-wide no-input audits, namespace pages, purpose-first API summaries, extension-member documentation, overwrite files, examples, availability notes, AGENTS.md maintenance, and verification. Use when the user asks to document a .NET API, update DocFX docs, create namespace pages, improve API summaries, add extension-member tables, update XML comments, add examples, maintain DocFX overwrite files, or verify documentation builds. Treat "use dotnet-docfx-digest", "complete missing documentation", and public .NET API changes as automatic triggers. + Use when the user wants to create, repair, audit, or complete DocFX docs for .NET public APIs, or has changed public API that needs namespace pages, XML comments, overwrite files, extension-member tables, examples, or build verification. Exclude private/internal APIs. --- # .NET DocFX Digest Steward diff --git a/skills/dotnet-new-app-slnx/SKILL.md b/skills/dotnet-new-app-slnx/SKILL.md index 3bffe36..625e48f 100644 --- a/skills/dotnet-new-app-slnx/SKILL.md +++ b/skills/dotnet-new-app-slnx/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-new-app-slnx -description: > - Scaffold a new .NET standalone application solution following codebelt engineering conventions. Use this skill when the user wants to create a new .NET application — Console, Web, or Worker service. Also use when the user mentions "new app", "new console app", "new web api", "new mvc app", "new razor app", "new web app", "new worker service", "scaffold app", "dotnet new web", "dotnet new webapi", "dotnet new mvc", "dotnet new webapp", "dotnet new worker", "dotnet new console", or wants a .NET application project with CI/CD pipeline, functional tests, and code quality tooling. ALWAYS use this skill when asked to scaffold or create a new .NET application solution. +description: > + Use when the user wants to scaffold a complete codebelt-style .NET application solution (`.slnx`) for Console, Web API, MVC, Razor, empty Web, or Worker hosts, with repository tooling and functional tests. Do not use for a quick throwaway project or application logic. --- # .NET Application Solution Setup (Codebelt Conventions) diff --git a/skills/dotnet-new-lib-slnx/SKILL.md b/skills/dotnet-new-lib-slnx/SKILL.md index 0f464d1..5a54b6e 100644 --- a/skills/dotnet-new-lib-slnx/SKILL.md +++ b/skills/dotnet-new-lib-slnx/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-new-lib-slnx description: > - Scaffold a new .NET NuGet library solution following codebelt engineering conventions. Use this skill when the user wants to create a new NuGet library, class library, or reusable .NET package. Also use when the user mentions "new library", "new NuGet package", "scaffold library", "class library solution", "dotnet new classlib", or wants a .NET library project with multi-target frameworks, strong-name signing, NuGet packaging, DocFX documentation, CI/CD pipeline, and code quality tooling. ALWAYS use this skill when asked to scaffold or create a new .NET library solution. + Use when the user wants to scaffold a complete codebelt-style .NET library or NuGet package solution (`.slnx`), including packaging, tests, DocFX, benchmarks, and strong-name signing. Do not use merely to add a library project to an established solution. --- # .NET Library Solution Setup (Codebelt Conventions) diff --git a/skills/dotnet-remote-testing/SKILL.md b/skills/dotnet-remote-testing/SKILL.md index 1d9a698..c9cd53b 100644 --- a/skills/dotnet-remote-testing/SKILL.md +++ b/skills/dotnet-remote-testing/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-remote-testing -description: > - Run .NET tests inside a resolved remote Docker environment — Visual Studio's Remote Testing without hand-writing container plumbing. Invoking this skill IS the request: run the tests immediately. Never reply with a menu of options or a questionnaire. Use when asked to remote test, run tests in Docker or a container, target a specific .NET SDK, list or select test environments, or honor an existing testenvironments.json. Honors configured Docker environments, or derives them from Microsoft's live .NET release index using mcr.microsoft.com/dotnet/sdk images, plus codebeltnet/ubuntu-testrunner for multi-targeted repos, via the runner scripts/remote-test.cs. Docker only; WSL and SSH are unsupported. Do NOT use to author or refactor test code, choose a testing framework, generate Dockerfiles, or run tests on the host. +description: > + Use when the user wants to run, list, or plan .NET tests in Docker remote-test environments, including `testenvironments.json` or a requested SDK/container. Do not use to write or refactor tests, create Dockerfiles, use WSL/SSH, or run tests on the host. compatibility: > Requires the .NET 10 SDK or later (`dotnet run --file`), a running Docker daemon, and PowerShell 7+. Zero-config discovery needs network access; a cache enables offline reuse. --- diff --git a/skills/dotnet-segregated-assets/SKILL.md b/skills/dotnet-segregated-assets/SKILL.md index cb2513f..feef224 100644 --- a/skills/dotnet-segregated-assets/SKILL.md +++ b/skills/dotnet-segregated-assets/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-segregated-assets description: > - Migrate or configure an ASP.NET Core web application so developers keep authoring static files in the conventional wwwroot while deployed static content is served by Codebelt Static Content Provider (codebeltnet/web-cdn-origin:2.0.0), a separate asset host rather than the web app. Use when asked to segregate static assets, move wwwroot off the web app, stop shipping wwwroot with the app, or reconcile Cuemon App/CDN TagHelpers with a segregated topology. Reuse existing Cuemon or project abstractions, distinguish App assets from shared CDN assets, preserve Static Web Assets, and verify publish/local invariants deterministically. Do NOT use to build a general-purpose CDN or migrate non-ASP.NET static sites. + Use when the user wants an ASP.NET Core app to keep authoring static files in `wwwroot` while serving deployed app assets from `codebeltnet/web-cdn-origin`, including Cuemon `app-*`/`cdn-*` migration. Do not use for general CDN design or non-ASP.NET sites. compatibility: > Requires the .NET SDK 10+ and PowerShell 7+. NuGet.org access is required when plan resolves an existing Cuemon package reference. Docker is optional (only for the local origin). CI guidance targets GitHub Actions, which is the assumed delivery surface. --- diff --git a/skills/dotnet-strong-name-signing/SKILL.md b/skills/dotnet-strong-name-signing/SKILL.md index 634d786..543d600 100644 --- a/skills/dotnet-strong-name-signing/SKILL.md +++ b/skills/dotnet-strong-name-signing/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-strong-name-signing -description: > - Generate a strong name key (.snk) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or sn.exe required. Works in any terminal. Use this skill when the user wants to create a strong name key, generate an .snk file, sign .NET assemblies, or mentions "strong-name", "snk", "AssemblyOriginatorKeyFile", "SignAssembly", or asks how to sign a .NET library. Also use when scaffolding .NET libraries or NuGet packages that need assembly signing. ALWAYS use this skill when asked to generate or create a strong name key file. +description: > + Use when the user wants to generate a `.snk` strong-name key, configure `SignAssembly` or `AssemblyOriginatorKeyFile`, or sign a .NET assembly, library, or NuGet package. Do not use for Authenticode, code-signing certificates, or secret-management tasks. --- # Strong Name Signing for .NET diff --git a/skills/dotnet-test/SKILL.md b/skills/dotnet-test/SKILL.md index c095854..869d159 100644 --- a/skills/dotnet-test/SKILL.md +++ b/skills/dotnet-test/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-test description: > - Move .NET xUnit test projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's WebApplicationFactory and hand-rolled host plumbing with WebApplicationTestFactory, WebApplicationTest, ApplicationTestFactory, and ApplicationTest — for ASP.NET Core, console, and worker applications alike. Invoking this skill IS the request: inspect the repository and refactor immediately, never opening with a menu, a capability list, or a questionnaire. Use for WebApplicationFactory migration, xUnit v2-to-v3 modernization, Microsoft Testing Platform adoption, managed fixtures, reusable functional-test harnesses, in-process console or worker tests, and unit-test bootstrap. Preserve behavior, test names, and package ownership, then validate restore/build/test. Do NOT use for NUnit/MSTest-only work, production refactoring without a test-project goal, or process-launching end-to-end harnesses. + Use when the user wants to bootstrap or refactor .NET xUnit tests onto Codebelt entrypoint-owned hosts, migrate `WebApplicationFactory`, modernize xUnit v2 to v3 or Microsoft Testing Platform, or add web, console, or worker fixtures. Exclude NUnit/MSTest and process-based end-to-end harnesses. compatibility: > Requires .NET SDK, PowerShell 7+, and network access to NuGet for dynamic package resolution. --- diff --git a/skills/git-keep-a-changelog/SKILL.md b/skills/git-keep-a-changelog/SKILL.md index c151fd0..2ec032b 100644 --- a/skills/git-keep-a-changelog/SKILL.md +++ b/skills/git-keep-a-changelog/SKILL.md @@ -1,7 +1,7 @@ --- name: git-keep-a-changelog description: > - Create or update CHANGELOG.md from git history using Keep a Changelog 1.1.0 style. Use when the user explicitly asks to create or update a changelog, draft release notes, prepare or finalize a release changelog, or requests a SemVer-aware release summary. Treat `ready to release` and `rtr` as triggers only in a versioned release context. Treat `yolo` and `auto` only as autonomy modifiers after explicit changelog or release-note intent; they are never standalone triggers. Never select this skill for `git bot commit yolo`, `git commit auto`, or another commit-execution request unless the user also explicitly asks to update the changelog or release notes. Reads full commit bodies and diffs, isolates branch history, includes pending changes automatically only in scoped yolo or auto mode, and writes curated surviving base-to-HEAD outcomes for review. + Use when the user wants to create or update `CHANGELOG.md`, follow Keep a Changelog, or finalize a versioned changelog. Treat `ready to release` or `rtr` as triggers only with version context. Do not trigger for GitHub releases, NuGet package notes, commit execution, or bare `yolo`/`auto`. compatibility: > Requires Git and PowerShell 7+ for deterministic branch-scope resolution. --- @@ -80,6 +80,17 @@ Reduce first. Interpret second. Summarize last. Establish the classification baseline at the user-facing release-entity boundary, not independently for every changed file. For a repo-managed skill, the entity is the skill capability together with its dedicated files and inseparable registration, catalog, documentation, validation, and eval wiring. If that entity is absent at the base and present at `HEAD`, its introduction is `Added`; intermediate commits that refine, fix, document, or validate it cannot create `Changed` or `Fixed` outcomes for that same new entity. A change to a separately pre-existing shared capability remains its own outcome and is classified from its own base state. +### Layered Capability Classification + +Do not use a top-level directory or the first framework commit as the only release entity. Classify at the smallest independently selectable user-facing boundary. For layered eval tooling, the protocol/framework and each selectable runner, CLI, TUI, or harness adapter can have different release states. + +- A child adapter absent at the resolved base and present at `HEAD` is `Added` even when its parent directory already existed at the base or was introduced earlier in the branch. Run the entity resolver for the parent and each independent child path when the diff supports that decomposition. +- Keep implementation, fixtures, conformance tests, and wiring with the capability they introduce. Do not repeat an adapter in a parent `Added` bullet and again in `Changed` as “added to the lineup.” +- Refinements to a pre-existing adapter or framework are `Changed` or `Fixed` from their final delta. A defect repaired before first release of a base-absent capability remains part of that capability's `Added` outcome. +- A planned, blocked, or unsupported CLI/TUI/harness is not support and must not be listed as an `Added` adapter. + +Use the final state, not commit verbs: newly usable execution support belongs under `Added`; changes to existing runner, orchestration, report, telemetry, or package behavior belong under `Changed`; and distinct supported repairs belong under `Fixed`. For example, a new framework followed by Cline or GitHub Copilot adapters gets `Added` outcomes for the framework and adapters, while changes to an existing Codex adapter are classified separately. + 1. Inspect cumulative manifest and version deltas across `diff_range`. 2. Inspect the cumulative base-to-`HEAD` diff. 3. Inspect any approved pending worktree changes that are part of the draft. diff --git a/skills/git-keep-a-changelog/evals/evals.json b/skills/git-keep-a-changelog/evals/evals.json index 4d080ab..c9504aa 100644 --- a/skills/git-keep-a-changelog/evals/evals.json +++ b/skills/git-keep-a-changelog/evals/evals.json @@ -251,6 +251,20 @@ "Does not reinterpret yolo as a release or changelog trigger", "Defers the request to the git-visual-commits workflow" ] + }, + { + "id": 22, + "prompt": "Create a deterministic temp git repo outside the current repository under `$env:TEMP`, then use git-keep-a-changelog there. Start from a tagged base release containing `CHANGELOG.md`, an existing `scripts/eval-runners/README.md`, an existing `scripts/eval-runners/codex/runner.ps1`, an existing report generator, and an existing conformance test. On branch `v1.1.0/eval-runner-support`, add the shared execution contracts and package bridge, add `scripts/eval-runners/cline/runner.ps1` and `scripts/eval-runners/github-copilot/runner.ps1` with their fixtures and documentation, refine the existing report telemetry and Codex command behavior, add a `freebuff-readiness.md` file that says its TUI transport is planned and blocked, and repair a distinct existing Codex evidence-directory defect. Include commits from multiple contributors. Update `CHANGELOG.md` and stop after the edit.", + "expected_output": "The changelog separates the new execution framework and independently selectable Cline and GitHub Copilot adapters as Added outcomes, classifies changes to the pre-existing report/Codex behavior separately, records the supported Codex repair when justified, does not repeat the new adapters under Changed, and does not claim the planned Freebuff TUI is supported.", + "expectations": [ + "Does not treat the top-level scripts/eval-runners directory as the only release entity when independently selectable adapters have distinct final states", + "Runs or applies path-backed base-versus-HEAD classification to the shared framework and the Cline and GitHub Copilot adapter boundaries", + "Places first-time Cline and GitHub Copilot support under Added even when the parent eval-runner framework already existed at the base or earlier in the branch", + "Classifies refinements to the pre-existing report generator or Codex adapter from their surviving base-to-HEAD behavior rather than calling them new support", + "Does not repeat a newly supported adapter in both Added and Changed or describe it merely as added to an existing lineup under Changed", + "Does not claim the planned or blocked Freebuff TUI is a supported runner or a completed Added capability", + "Keeps the changelog grounded in the final state and includes contributions from every selected author" + ] } ] } diff --git a/skills/git-nuget-readme/SKILL.md b/skills/git-nuget-readme/SKILL.md index 79c187c..94a6b4f 100644 --- a/skills/git-nuget-readme/SKILL.md +++ b/skills/git-nuget-readme/SKILL.md @@ -1,7 +1,7 @@ --- name: git-nuget-readme description: > - Create or update a NuGet package README.md from git history and real .NET project metadata for repositories that ship a package from `src/`. Use this skill whenever the user asks to write a package README, refresh NuGet-facing docs, improve the repo README for a library, summarize the current branch into README copy, or make a package more compelling to adopt on NuGet. Treat requests like "update the README for this package", "write a NuGet README from git", "refresh the library README", "make this NuGet package easier to pick", or "generate a devex-friendly README for this assembly" as automatic triggers. The skill discovers the advertised packable project, grounds the README in real package and source metadata, preserves honest claims, and writes forthcoming, adoption-friendly copy instead of generic marketing fluff. + Use when the user wants to create or refresh a NuGet-facing `README.md` for a .NET package, grounded in the packable `src/` project and current repository changes. Do not use for general repository docs, DocFX API pages, changelogs, or release notes. --- # Git NuGet README diff --git a/skills/git-nuget-release-notes/SKILL.md b/skills/git-nuget-release-notes/SKILL.md index b91002e..2b49771 100644 --- a/skills/git-nuget-release-notes/SKILL.md +++ b/skills/git-nuget-release-notes/SKILL.md @@ -1,7 +1,7 @@ --- name: git-nuget-release-notes description: > - Create or update per-package NuGet release notes from git history for .NET repositories that store cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Use this skill whenever the user asks for NuGet release notes, `PackageReleaseNotes.txt`, per-assembly or per-package release notes, or wants git commits turned into package release notes instead of a repo-wide changelog. Treat requests like "update PackageReleaseNotes.txt", "write NuGet release notes from git", "summarize this release per assembly", or "create missing package release notes under .nuget" as automatic triggers. The skill discovers packable `src/` projects, resolves concrete release version and availability per package, creates missing files when needed, preserves cumulative newest-first history, reduces each package to its surviving base-to-HEAD delta before using history as context, and avoids raw commit-log dumps or unsupported claims. + Use when the user wants per-package NuGet release notes in cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files, including creating missing files for packable `src/` projects. Do not use for repository `CHANGELOG.md`, GitHub releases, or package README work. --- # Git NuGet Release Notes diff --git a/skills/git-remote-release/SKILL.md b/skills/git-remote-release/SKILL.md index 1610c65..7409e03 100644 --- a/skills/git-remote-release/SKILL.md +++ b/skills/git-remote-release/SKILL.md @@ -1,7 +1,7 @@ --- name: git-remote-release description: > - Generate GitHub release notes by summarizing all commits and pull requests between two Git tags, branches, or the current branch and the upstream default branch. Use when the user asks to write release notes, generate release notes, draft a GitHub release, create release notes from tags, summarize changes between versions, summarize the current branch, or provides a GitHub compare URL. Trigger phrases: "release notes", "generate release notes", "what changed between", "summarize changes from v1 to v2", "GitHub release", "summarize this branch", compare URLs like "github.com/owner/repo/compare/v1...v2". When no explicit input is given, detects the current branch and compares against the upstream default branch automatically. + Use when the user wants GitHub release notes or a human-readable change summary from commits and pull requests between tags, branches, or a GitHub compare URL. Do not use for `CHANGELOG.md`, NuGet `PackageReleaseNotes.txt`, commit messages, or squash summaries. --- # Git Remote Release diff --git a/skills/git-repo-digest/SKILL.md b/skills/git-repo-digest/SKILL.md index f1e0dc0..0d41847 100644 --- a/skills/git-repo-digest/SKILL.md +++ b/skills/git-repo-digest/SKILL.md @@ -1,7 +1,7 @@ --- name: git-repo-digest description: > - Generate source-grounded repository digest markdown from deterministic local evidence bundles. Use when the user asks to create, refresh, or complete repo/package digests, family or project overview pages, .bot/digests output, digest workspace workflows, or result/Index.md plus result/{PackageName}.md files for any repository URL. The skill runs its bundled .NET file-based evidence generator over a git clone, separates authoritative XML evidence from Markdown prompts and reading aids, writes package digests first, then writes the overview from completed package digests, and enforces complete-read grounding and no-invention rules even when file output is capped. + Use when the user wants source-grounded repository or package digest Markdown from a repository URL or existing `.bot/digests` workspace, especially `result/Index.md` and per-package pages. Do not use for README, DocFX, changelog, release-note, or one-off prose summaries. --- # Git Repo Digest diff --git a/skills/git-visual-commits/SKILL.md b/skills/git-visual-commits/SKILL.md index d5664f1..d8f4eb5 100644 --- a/skills/git-visual-commits/SKILL.md +++ b/skills/git-visual-commits/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-commits description: > - Execute the structured git commit workflow whenever the user says `git bot commit`, `git commit`, or `git our commit`; asks the agent to commit or stage changes; or asks to write or review a commit message. Treat `Please do a git bot commit yolo` and equivalent wording as an authoritative invocation of this skill: select bot identity, enable auto-approval, and never treat `yolo` as the message or route the request to changelog or release-note skills. Treat commit wording as an automatic trigger for this skill, not as a casual hint. `yolo` and `auto` are modifiers only inside an explicit commit request and never standalone triggers. Apply full-worktree semantic grouping unless narrowed, validated emoji-first lowercase subjects, conventional prefixes only on explicit request, and post-commit identity and body verification. + Use when the user asks to stage or commit changes, write or review a commit message, or says `git bot commit`, `git commit`, or `git our commit`. Treat commit wording as an automatic trigger for this skill, not as a casual hint. `yolo` and `auto` only modify an explicit commit request. --- # Git Visual Commits diff --git a/skills/git-visual-squash-summary/SKILL.md b/skills/git-visual-squash-summary/SKILL.md index 71c5ddf..3e6fe6d 100644 --- a/skills/git-visual-squash-summary/SKILL.md +++ b/skills/git-visual-squash-summary/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-squash-summary description: > - Turn many commits into a curated grouped squash summary for squash-and-merge contexts. Use when the user asks to squash a branch, summarize PR commits, or clean up history. Defaults to full feature branch against base (not tracking remote), includes all authors unless narrowed, and acts immediately—the skill is read-only with no permission-seeking. Returns grouped lines only, resolving the cumulative diff to drop reverted churn, preserving identifiers and overlap, and avoiding changelog wording. A bare invocation is a complete request: run git commands immediately and return summary lines, never an instruction recap or permission question. + Use when the user wants a read-only, ready-to-paste grouped squash-and-merge or PR branch summary from a commit range or the current feature branch. Do not use to mutate history, write changelog or release notes, execute commits, or summarize a single commit message. --- # Git Visual Squash Summary diff --git a/skills/markdown-illustrator/SKILL.md b/skills/markdown-illustrator/SKILL.md index 22329df..f0af52c 100644 --- a/skills/markdown-illustrator/SKILL.md +++ b/skills/markdown-illustrator/SKILL.md @@ -1,7 +1,7 @@ --- name: markdown-illustrator description: > - Turn a markdown document into a visualization-first chat response consisting of one Visual Brief and one high-quality diffuser prompt generated with best-effort reasoning. Use when the user references a .md file and wants a hero image, cover image, visual digest, keynote opener, illustration, or diffuser prompt, especially for requests like "turn roadmap.md into a keynote opener image" or "create a visual digest for onboarding-notes.md". Default to zero follow-up questions, no file creation, and no style/theme/model menus; infer a compact visual strategy from the request and document, and only honor extra specificity when the user explicitly asks for a named model, aesthetic, or visual treatment such as whiteboard or blackboard. + Use when the user references a Markdown file and wants one hero, cover, keynote-opener, visual-digest, or illustration prompt derived from it, including whiteboard, blackboard, or named image-model styling. Do not trigger for image generation itself or general Markdown summarization. --- # Markdown Illustrator diff --git a/skills/skill-creator-agnostic/SKILL.md b/skills/skill-creator-agnostic/SKILL.md index e8ac43a..4f68913 100644 --- a/skills/skill-creator-agnostic/SKILL.md +++ b/skills/skill-creator-agnostic/SKILL.md @@ -1,7 +1,8 @@ --- name: skill-creator-agnostic description: > - DEPRECATED — no longer maintained and scheduled for removal in 1.0.0. Retained only for backward compatibility. Do not use for new skill creation, modification, or benchmarking. Use Anthropic's `skill-creator` directly and apply this repository's skill-authoring rules from `AGENTS.md` instead. + DEPRECATED — no longer maintained and scheduled for removal in 1.0.0. Use only when the user explicitly invokes `skill-creator-agnostic` or asks about its status. Redirect skill creation, modification, evaluation, and benchmarking to Anthropic's `skill-creator` plus repository `AGENTS.md`. +disable-model-invocation: true --- # Skill Creator Agnostic diff --git a/skills/trunk-first-repo/SKILL.md b/skills/trunk-first-repo/SKILL.md index a1168d6..d1aabb0 100644 --- a/skills/trunk-first-repo/SKILL.md +++ b/skills/trunk-first-repo/SKILL.md @@ -1,7 +1,7 @@ --- name: trunk-first-repo -description: > - Initialize a folder as a git repository following scaled trunk-based development. Sets up an empty main branch (seed commit only), creates a versioned feature branch, pushes main before feature branches, and enforces a PR-first workflow where content only reaches main through pull requests. Use this skill when the user wants to initialize a git repo, set up a new repository, start a project with proper git workflow, safely push the first trunk-first branches later with "push remote", or mentions "trunk-based", "PR workflow", "branch protection", "git init", or wants to follow GitHub PR best practices. ALWAYS use this skill when asked to initialize or set up a git repository. +description: > + Use when the user wants to initialize a new Git repository with a scaled trunk-based, PR-first workflow, including an empty seeded `main`, a versioned feature branch, branch protection, or a guarded first remote push. Do not use for commits or history cleanup in an existing repository. --- # Trunk-First Repo