feat(plugin): ship coder_eval as a Claude Code plugin + marketplace

.claude-plugin/marketplace.json

@@ -0,0 +1,17 @@
+{
+  "name": "coder-eval",
+  "owner": {
+    "name": "UiPath",
+    "url": "https://github.com/UiPath/coder_eval"
+  },
+  "description": "Evaluate and benchmark AI coding agents and Claude Code skills.",
+  "plugins": [
+    {
+      "name": "coder-eval",
+      "source": "./plugins/coder-eval",
+      "description": "Author, run, and analyze coder-eval suites — including whether your Claude Code skills actually trigger.",
+      "category": "testing",
+      "keywords": ["evaluation", "testing", "skills", "benchmark", "ci"]
+    }
+  ]
+}

.claude/commands/coder-eval-run-analysis.md

@@ -23,18 +23,33 @@ Inspect the target path:
 
 **Task scope (single)**: read `task.json`.
 
-**Task scope (aggregate replicates)**: read every `??/task.json` and merge — per-replicate arrays for `final_status`, `weighted_score`, `iteration_count`, `duration_seconds`, `total_cost_usd`; union of `success_criteria_results` keyed by criterion `description`. Drive recommendations from the aggregate ("3/5 replicates failed criterion X"), not cherry-picked replicates.
+**Task scope (aggregate replicates)**: read every `??/task.json` and merge — per-replicate arrays for `final_status`, `weighted_score`, `iteration_count`, `duration_seconds`, `total_token_usage.total_cost_usd`; union of `success_criteria_results` keyed by criterion `description`. Drive recommendations from the aggregate ("3/5 replicates failed criterion X"), not cherry-picked replicates.
 
-**Variant / run scope, > 20 tasks**: do NOT read the full `task.json` files — the `turns` arrays are large and only useful per-task. Use `jq` (or `python3` if missing) to extract a compact summary per task:
+**Variant / run scope, > 20 tasks**: do NOT read the full `task.json` files — the `iterations` arrays are large and only useful per-task. Use `jq` (or `python3` if missing) to extract a compact summary per task:
 
 ```
-{task_id, final_status, weighted_score, duration_seconds, total_cost_usd,
- total_tokens, assistant_turn_count, max_turns, max_turns_exhausted,
- iteration_count, model_used, criteria_count, all_criteria_perfect,
- failed_criteria: [{type, description, score, error_excerpt}]}
+{
+  task_id, final_status, weighted_score, duration_seconds,
+  iteration_count, model_used, max_turns_exhausted,
+  total_cost_usd:  .total_token_usage.total_cost_usd,
+  total_tokens:    (.total_token_usage.input_tokens + .total_token_usage.output_tokens),
+  assistant_turns: .total_assistant_turns,
+  max_turns:       .task_config.resolved.run_limits.max_turns,
+  criteria_count:  (.success_criteria_results | length),
+  all_criteria_perfect:
+    (.success_criteria_results | length > 0 and all(.[]; .score == 1.0)),
+  failed_criteria: [
+    .success_criteria_results[]
+    | select(.score < .pass_threshold)
+    | {criterion_type, description, score,
+       error_excerpt: ((.error // .details // "")[0:200])}
+  ]
+}
 ```
 
-`error_excerpt` = first ~200 chars of each failing criterion's `error` / `output` / `Instructions` field. This is what enables clustering in Step 3.
+Those paths are the real `task.json` shape. `jq` returns `null` for a key that does not exist rather than failing, so a mistyped path yields a table of nulls that reads like a run with no data. There is no top-level `turns`, `total_tokens`, `total_cost_usd`, `max_turns` or `criteria_count`: turn records live under `iterations`, token and cost figures under `total_token_usage`, the turn cap under `task_config.resolved.run_limits`, and a criterion's type is `criterion_type`. A criterion passes when `score >= pass_threshold` — there is no `passed` boolean.
+
+`error_excerpt` = first ~200 chars of each failing criterion's `error`, falling back to `details`. Those are the only two free-text fields a criterion result carries: `error` holds an exception, `details` the checker's own diagnostic, so a criterion that simply did not match has `error: null` and all its signal in `details`. This is what enables clustering in Step 3.
 
 **Variant / run scope, ≤ 20 tasks**: read all `??/task.json` files directly.
 

.claude/commands/coder-eval-task-create.md

@@ -63,10 +63,18 @@ Choose criteria types based on what needs to be verified:
 | Observed vs expected label | `classification_match` | File-based label match for classification suites (emits P/R/F1) |
 | UiPath agent eval | `uipath_eval` | UiPath agent evaluation results |
 
-**Criteria design rules:**
-- Every task needs at least one criterion that validates the **output content**, not just existence
+Before choosing criteria, read `plugins/coder-eval/reference/task-rubric.md` — the shared
+adversarial checklist ("could this pass for the wrong reason?", fixture lifecycle, scope
+match). It is the single declaration for all three consumers: this command and the
+plugin's `task` and `lint-tasks` skills.
+
+The rubric owns the *correctness* checks — that something validates output content, that
+`require_success: true` is set whenever a command's success is what you are grading, that a
+criterion cannot pass for the wrong reason. Do not restate them here; apply them from there.
+
+**Criteria design rules** (conventions the rubric does not cover):
 - Use `run_command` with `expected_stdout` + `stdout_match: regex` to validate script output
-- Use `command_executed` sparingly — only when verifying the agent used a specific tool matters. Set `require_success: false` unless the command must succeed.
+- Use `command_executed` sparingly — only when verifying the agent used a specific tool matters
 - Use `file_check` instead of separate `file_exists` + `file_contains` when checking the same file
 - Set `weight` to reflect importance: 0.5 for nice-to-have, 1.0 for standard, 1.5-2.0 for critical
 - Default `pass_threshold: 0.9` is fine for most criteria. Use `1.0` only for binary checks.

.claude/harness-candidates.md

@@ -206,3 +206,63 @@ harness once for all of them.
   preceding `&&`/`||`, NOT on the string's shape — a "contains 'ubuntu'" heuristic
   silently fails on `uipath-ubunut-latest`, the exact transposition typo the rule is
   for. Caught in the multi-model review of PR #86.
+
+## From the 2026-08-04 Claude Code plugin marketplace run
+
+- [ ] **Plugin skills must not name a file that exists only in THIS repo** — the
+  `test_bundled_files_reference_no_repo_paths` denylist (`docs/`, `src/`,
+  `.claude/shared/`, `.claude/commands/`, `uv run`, `../`) deliberately allows
+  `tasks/` and `.claude/skills/`, because those are user-workspace paths the
+  skills legitimately scan and scaffold. So a skill body naming a specific repo
+  file (e.g. `tasks/hello_date.yaml`) would slip past the guard even though an
+  installed plugin is copied to `~/.claude/plugins/cache/` without it. The
+  obvious rule — "extract path-shaped tokens, fail if the path exists at the repo
+  root" — is NOT cheap: `init` legitimately tells users to scan `pyproject.toml`
+  and `package.json`, and `pyproject.toml` exists here, so the heuristic
+  false-positives on correct prose. Needs a token classifier that distinguishes
+  "a file to look for in the user's repo" from "a file in ours", which is a
+  design problem, not a 30-minute one. No skill violates it today (grepped) —
+  caught in the 2026-08-04 claude-code-plugin-marketplace implementation run.
+  *Update (2026-08-04, plugin-audit-p0-p1 run): the guard was renamed and widened
+  from `skills/*/SKILL.md` to every shipped text file under `plugins/coder-eval/`
+  (`PLUGIN_TEXT_FILES`), which closed the coverage half of this gap — a bundled
+  reference now cannot name a repo path either. The token-classifier problem
+  described above is unchanged and still deferred.*
+
+## From 2026-08-04 plugin-audit-p0-p1 run
+
+- [ ] **A skill's advertised `description` must not promise a check that no bundled
+  reference declares.** `lint-tasks` ships a user-facing description claiming it
+  finds "prompts that give away the answer", but that check was declared only in
+  `skills/task/SKILL.md` prose — a file `lint-tasks` never reads — so the two
+  rubric readers had already forked on it before the skill shipped. Caught by a
+  reviewer, not by a test; fixed by promoting it to rubric check 7. A guard would
+  have to map claim-phrases in a description onto declarations in
+  `reference/task-rubric.md`, which is natural-language matching, not a token
+  grep — the phrasings are deliberately different (a description sells, a rubric
+  check instructs), so any cheap version either misses the real case or fails on
+  correct prose. Needs a fixed vocabulary of claim tags shared between the two
+  files to become mechanical, which is a design change rather than a 30-minute
+  rule — caught in the 2026-08-04 plugin-audit-p0-p1 implementation run.
+
+## From the PR #82 review follow-up (2026-08-10)
+
+- [ ] **CE035 — documented `coder-eval` invocations must be executable as written.**
+  `init/SKILL.md` told the agent to run `coder-eval plan <task-directory>` and
+  "iterate until it exits 0", which the CLI rejects outright (`plan` takes files;
+  a directory argument exits 1 with a hint) — an unreachable loop condition
+  shipped in a skill. A rule would scan inline-code spans and fenced `bash` blocks
+  across `README.md`, `docs/**/*.md` and `plugins/**/*.md`, assert the subcommand
+  exists in the Typer app, and — the harder half — that the *argument shape* is
+  one the command accepts. The subcommand check is cheap and would not have caught
+  this; the argument-shape check is what matters and needs either a real
+  invocation (see the live-smoke candidate below) or a per-command arity model
+  that duplicates the CLI signature. Deferred on that split — caught in the PR #82
+  review, fixed by hand in `init/SKILL.md`.
+
+- [ ] **Documented-CLI live smoke.** The behavioural counterpart to CE035: in a
+  `-m live`/`-m slow` test, materialize a fixture repo with one task YAML and
+  execute every fenced `coder-eval …` command extracted from the shipped skills
+  and docs, asserting exit 0 (or an explicitly-expected non-zero). This is the
+  only form that proves argument shape rather than command existence. Not
+  statically reachable, hence separate from CE035 — proposed in the PR #82 review.

.claude/shared/run-layout.md

@@ -1,8 +1,9 @@
-# Run layout (shared)
+<!-- Mirrored verbatim at plugins/coder-eval/reference/run-layout.md — update both together. -->
+# Run layout
 
-The on-disk structure of a coder_eval evaluation run — the factual contract that
-`coder-eval-run-analysis` and `coder-eval-review` both read. If the run directory
-structure changes, update it here and every consumer follows.
+The on-disk structure of a coder_eval evaluation run — the factual contract every
+run-reading command and skill follows. If the run directory structure changes, update it
+here and every consumer follows.
 
 ```
 runs/<run_id>/<variant_id>/<task_id>/<NN>/{task.json, task.log, artifacts/}

.github/workflows/pr-checks.yml

@@ -207,6 +207,67 @@ jobs:
         working-directory: evalboard
         run: pnpm verify
 
+  plugin-validate:
+    # Proves the Claude Code plugin marketplace is installable and that the suite
+    # `skill-check` scaffolds is real: the manifests pass strict validation, and the
+    # bundled activation template both schema-validates and expands to one task per
+    # dataset row. Needs no credentials — nothing here invokes a model.
+    name: Claude Code Plugin (manifests + offline scaffold)
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      # `claude plugin validate` ships in the Claude Code npm package.
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version: "20"
+
+      - name: Install Claude CLI
+        run: npm install -g @anthropic-ai/claude-code
+
+      - name: Validate plugin manifest (strict)
+        run: claude plugin validate ./plugins/coder-eval --strict
+
+      - name: Validate marketplace manifest (strict)
+        run: claude plugin validate . --strict
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0
+
+      # Runs OUTSIDE the source tree, the way a user's copy of the template does:
+      # no experiments/, no tasks/, no coder_eval checkout on the path. `plan` is a
+      # schema check only (it exits 0 even when dataset.paths names a nonexistent
+      # file), so the row-count assert goes through expand_dataset — otherwise this
+      # step would pass even if activation-rows.jsonl were never copied.
+      # Reproduce locally with: SCRATCH=$(mktemp -d) VENV=$(mktemp -d)/venv
+      - name: Scaffold assert (no source tree)
+        run: |
+          set -euo pipefail
+          SCRATCH="$RUNNER_TEMP/scratch"
+          VENV="$RUNNER_TEMP/venv"
+          mkdir -p "$SCRATCH"
+          cp plugins/coder-eval/reference/templates/activation.yaml "$SCRATCH/"
+          cp plugins/coder-eval/reference/templates/activation-rows.jsonl "$SCRATCH/"
+          # A venv (not `uv tool install`) because the expansion assert needs
+          # `coder_eval` importable, not just the `coder-eval` CLI on PATH.
+          uv venv "$VENV"
+          VIRTUAL_ENV="$VENV" uv pip install .
+          cd "$SCRATCH"
+          "$VENV/bin/coder-eval" plan activation.yaml
+          "$VENV/bin/python" - <<'PY'
+          from pathlib import Path
+          from coder_eval.orchestration.task_loader import expand_dataset, load_task
+          task, _ = load_task(Path("activation.yaml"))
+          rows = expand_dataset(task, Path("."))
+          assert len(rows) == 6, f"expected 6 row-tasks, got {len(rows)}"
+          labels = {c.expected_skill for t in rows for c in t.success_criteria}
+          assert labels == {"my-skill", ""}, labels
+          print(f"ok: {len(rows)} row-tasks")
+          PY
+
   no-uipath-extra:
     # Proves that `pip install coder-eval` (without the optional `[uipath]`
     # extra) yields a working framework: imports succeed, the criterion

.github/workflows/release.yml

@@ -186,7 +186,7 @@ jobs:
           echo "version=$V" >> "$GITHUB_OUTPUT"
           echo "Publishing version: $V"
 
-      - name: Regenerate uv.lock, bump action.yml pin, and amend release commit
+      - name: Regenerate uv.lock, bump action.yml + plugin.json pins, and amend release commit
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
         env:
           # Passed via env (not interpolated into the script) per GitHub's
@@ -204,10 +204,18 @@ jobs:
           sed -i -E 's/^([[:space:]]*default: ")[0-9]+\.[0-9]+\.[0-9]+("   # <-- kept in sync)/\1'"${VERSION}"'\2/' action.yml
           grep -q "default: \"${VERSION}\"" action.yml || { echo "action.yml version bump failed"; exit 1; }
           git add action.yml
+          # Keep the Claude Code plugin manifest's version in lockstep. `claude
+          # plugin validate --strict` (run in pr-checks) rejects a manifest with
+          # no version, and a stale one strands users on a cached copy.
+          sed -i -E 's/^([[:space:]]*"version": ")[0-9]+\.[0-9]+\.[0-9]+(",)/\1'"${VERSION}"'\2/' \
+            plugins/coder-eval/.claude-plugin/plugin.json
+          grep -q "\"version\": \"${VERSION}\"" plugins/coder-eval/.claude-plugin/plugin.json \
+            || { echo "plugin.json version bump failed"; exit 1; }
+          git add plugins/coder-eval/.claude-plugin/plugin.json
           # Regenerate the lock too; stage it (a no-op if unchanged).
           uv lock
           git add uv.lock
-          # Amend only if action.yml/uv.lock actually changed the tree.
+          # Amend only if action.yml/plugin.json/uv.lock actually changed the tree.
           if ! git diff --cached --quiet; then
             git commit --amend --no-edit
             # Amend replaced the commit the tag points at; re-point it before pushing.
@@ -216,7 +224,11 @@ jobs:
 
       - name: Push release commit and tags
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
-        run: git push origin main "v${{ steps.release.outputs.version }}"
+        env:
+          # Passed via env (not interpolated into the script) per GitHub's
+          # injection guidance — matching the step above.
+          VERSION: ${{ steps.release.outputs.version }}
+        run: git push origin main "v${VERSION}"
 
       - name: Move major action tag (vN -> this release)
         if: steps.release.outputs.version != ''