.yaml
my-skill-workspace/
iteration-1/
result.json
```
-When `evals/eval.yaml` lives under a directory containing `SKILL.md`,
-`skill-up` automatically installs that local Skill for the run, so you usually
-do not need to list the Skill path manually in `eval.yaml`.
-
-## Installation
-
-Install with the script:
-
-```bash
-curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash
-```
-
-The installer downloads the matching binary from [GitHub Releases](https://github.com/alibaba/skill-up/releases).
-
-To build locally from a checkout, install [Go](https://go.dev/dl/) 1.25 or later:
-
-```bash
-make build
-# or
-go build -o bin/skill-up ./cmd/skill-up
-```
-
-**Windows users**: skill-up runs natively on Windows. See
-[Windows Support](docs/guide/windows.md) for the recommended workflow,
-known limitations (notably: native agent CLI execution requires Git
-Bash), and the PowerShell tooling under `scripts/windows/`.
-
-## Quick Start
-
-### 1. Create Eval Config
-
-In your Skill directory, create `evals/eval.yaml`:
+### 3. Fix, regress, and iterate
-```yaml
-schema_version: v1alpha1
-
-environment:
- type: none
+Continue in the same conversation:
-engine:
- name: claude_code
-
-cases:
- files:
- - evals/cases/hello-world.yaml
+```markdown
+Review the latest skill-up results. For each failure, determine whether the
+Skill or the eval is wrong. Fix SKILL.md and supporting files, or repair the
+eval case and judge as appropriate. Add regression cases for the bugs you
+found, rerun skill-up, and continue until the important behaviors pass.
```
-When `evals/eval.yaml` lives under a directory that contains `SKILL.md`, skill-up installs the current Skill automatically. The omitted fields use defaults: JSON report output, `timeout_seconds: 300`, `max_turns: 10`, and `parallelism: 1`.
-
-For the full `eval.yaml` schema, see [Writing Evals](docs/guide/writing-evals.md).
+This is the evolution loop: reports become fixes, fixes become regression
+cases, and every iteration makes the Skill and its eval suite stronger.
-### 2. Write an Eval Case
+### Prefer manual setup?
-Create `evals/cases/hello-world.yaml`:
-
-```yaml
-input:
- prompt: |
- Please generate a Hello World program
-
-expect:
- must_contain:
- - "Hello"
- - "World"
-```
-
-The case `id` defaults to the filename (`hello-world`). Add a `judge` block only when you need script-based or agent-based grading.
-
-### 3. Validate Config
+You can still install the CLI directly and hand-write `eval.yaml` and case
+files:
```bash
-skill-up validate
-```
-
-This step is optional, but useful before the first run: it checks `eval.yaml` and all referenced case files without starting an Agent Engine.
-
-### 4. Run Evaluation
-
-```bash
-skill-up run
+curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash
```
-Results are written to `-workspace/iteration-1/`.
-
-For engineering conventions (Conventional Commits, Git hooks, golangci-lint), see [CONTRIBUTING.md](CONTRIBUTING.md).
+See the official documentation for
+[Getting Started](https://alibaba.github.io/skill-up/guide/getting-started),
+[Writing Evals](https://alibaba.github.io/skill-up/guide/writing-evals),
+[CLI Reference](https://alibaba.github.io/skill-up/guide/cli-reference), and
+[User Configuration](https://alibaba.github.io/skill-up/guide/user-config).
+Windows-specific setup and limitations are covered in the
+[Windows guide](https://alibaba.github.io/skill-up/guide/windows).
## User config
diff --git a/README.zh.md b/README.zh.md
index 4bfd4148..b6af190a 100644
--- a/README.zh.md
+++ b/README.zh.md
@@ -5,6 +5,10 @@
skill-up
+
+ Agent Skill 的评测与演进工具。
+
+
@@ -42,10 +46,16 @@
## 简介
-**skill-up** 是面向 Agent Skill 开发者的 CLI 评测框架。在 Skill 包内通过 `evals/eval.yaml` 与 `evals/cases/*.yaml` 声明评测环境、依赖、用例与评估方式，在本地或 CI 中运行评测并生成结构化报告。
+**skill-up** 是 Agent Skill 的评测与演进工具。
+
+- **评测（Evaluation）**让 Skill 质量可度量、可复现：声明式 YAML 用例可在多个 Agent Engine 中运行，通过规则、脚本或 Agent Judge 评分，并在本地或 CI 中生成结构化报告。
+- **演进（Evolution）**把评测结果变成下一轮改进：通过对话，**skill-upper** 读取失败、自动修复或补充 eval 用例、重新运行 skill-up，并与你持续迭代。
+
+
## 特性
+- **skill-upper 从评测到演进的闭环**：通过自然对话创建评测、诊断失败、自动修复或补充用例并重新运行 skill-up，让 eval 评测集持续演进。
- **声明式评测配置**：通过 YAML（`eval.yaml` + `cases/*.yaml`）定义评测环境、引擎、模型和用例。
- **多引擎支持**：内置支持 Qoder CLI、Claude Code、Codex；亦可通过 `engine.custom` 接入用户自定义 Agent（本地传输，详见 [docs/design/custom-engine.md](docs/design/custom-engine.md)）。
- **灵活评分**：支持 `rule_based`（规则匹配）、`script`（脚本评分）、`agent_judge`（Agent 评分）三种评估策略。
@@ -58,19 +68,18 @@
官方的 [Agent Skills 评测指南](https://agentskills.io/skill-creation/evaluating-skills) 说明了正确的评测循环：编写真实用例，分别运行 with/without Skill，评分输出，汇总结果，然后持续迭代。`skill-up` 的价值是把这套流程产品化成一个可复用的 CLI：
- 用声明式的 `eval.yaml` + `cases/*.yaml` 取代临时拼出来的运行目录。
+- 补齐持续改进闭环：skill-upper 可以解读失败报告、修复或新增 eval 用例，并通过对话驱动下一轮 skill-up 运行。
- 自动完成 workspace 准备、Skill 安装、Agent Engine 调用、评分和报告生成。
- 支持多个引擎（`claude_code`、`codex`、`qodercli`、`qwen_code`），不绑定单一客户端。
- 兼容 Anthropic 风格的 `evals.json`，同时提供更丰富的 judge、适合 CI 的命令和结构化报告。
-## 推荐使用方式：AI 辅助配合 skill-upper
-
-推荐使用仓库内置的 **skill-upper** Agent Skill。它会引导 AI Agent
-为目标 Skill 生成评测配置、校验、运行并解释结果，避免一开始就手写所有
-YAML。
+## 快速上手：使用 skill-upper 演进 Skill
-### 1. 安装 `skill-upper` Agent Skill
+推荐通过仓库内置的 **skill-upper** Agent Skill 使用 skill-up。它可以让
+AI Agent 通过对话创建评测、运行 skill-up、理解失败原因、修复 Skill 或
+eval、补充回归用例，并持续完成下一轮迭代。
-推荐使用 `skills` CLI 安装：
+### 第一步：安装 skill-upper
```bash
# Codex，全局安装
@@ -80,30 +89,21 @@ npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a claude-code -y
```
-安装这个 Skill 前不需要先安装 `skill-up`。`skill-upper` 在运行时会检查
-`skill-up` 命令是否可用；如果缺失，它会引导 Agent 完成安装。
-
-### 2. 添加与运行评测
-
-在 AI Agent 中打开目标 Skill 项目。目标项目至少应包含：
-
-```text
-my-skill/
- SKILL.md
-```
+通常不需要提前安装 skill-up。skill-upper 运行时会检查 CLI；如果缺失，
+它会引导 Agent 完成安装。
-然后直接给 Agent 一个明确任务：
+### 第二步：创建并运行第一组评测
-```text
-使用 skill-upper 给这个 Skill 添加评测。
-添加这个评测用例：
-- 输入：写一个 hello world 的程序。
-- 评测：是否包含 hello 和 world 打印。
+在 Codex、Claude Code 或其他兼容 Agent 中打开包含 `SKILL.md` 的 Skill
+项目，然后直接对话：
-然后运行 skill-up 完成校验和评测。
+```markdown
+使用 skill-upper 评测这个 Skill。
+阅读 SKILL.md，识别最重要的能力，创建真实的 eval 用例并选择合适的
+Judge，校验配置后运行 skill-up。最后总结结果和影响最大的失败项。
```
-Agent 应该会生成类似结构：
+skill-upper 会生成声明式评测集并替你驱动 CLI：
```text
my-skill/
@@ -111,100 +111,40 @@ my-skill/
evals/
eval.yaml
cases/
- basic.yaml
+ .yaml
my-skill-workspace/
iteration-1/
result.json
```
-当 `evals/eval.yaml` 位于包含 `SKILL.md` 的目录下时，`skill-up` 会在运行时
-自动安装这个本地 Skill，通常不需要在 `eval.yaml` 里手动写 Skill 路径。
-
-## 安装
-
-使用安装脚本：
-
-```bash
-curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash
-```
-
-安装脚本会从 [GitHub Releases](https://github.com/alibaba/skill-up/releases) 下载当前平台对应的二进制文件。
-
-如需从仓库 checkout 后本地构建，需要安装 [Go](https://go.dev/dl/) 1.25 或更高版本：
-
-```bash
-make build
-# 或
-go build -o bin/skill-up ./cmd/skill-up
-```
-
-**Windows 用户**：skill-up 原生支持 Windows。请参阅
-[Windows 支持指南](docs/zh/guide/windows.md) 了解推荐工作流、已知限制
-（特别是：原生运行 agent CLI 需要 Git Bash）以及 `scripts/windows/`
-下的 PowerShell 工具脚本。
-
-## 快速上手
-
-### 第一步：创建评测配置
-
-在 Skill 目录下创建 `evals/eval.yaml`：
-
-```yaml
-schema_version: v1alpha1
-
-environment:
- type: none
-
-engine:
- name: claude_code
-
-cases:
- files:
- - evals/cases/hello-world.yaml
-```
-
-当 `evals/eval.yaml` 位于包含 `SKILL.md` 的目录下时，skill-up 会自动安装当前 Skill。未写出的字段会使用默认值：JSON 报告、`timeout_seconds: 300`、`max_turns: 10`、`parallelism: 1`。
+### 第三步：修复、回归并持续迭代
-完整的 `eval.yaml` 配置说明见 [编写评测配置与用例](docs/zh/guide/writing-evals.md)。
+在同一段对话中继续：
-### 第二步：编写 Eval Case
-
-创建 `evals/cases/hello-world.yaml`：
-
-```yaml
-input:
- prompt: |
- 请帮我生成一个 Hello World 程序
-
-expect:
- must_contain:
- - "Hello"
- - "World"
+```markdown
+检查最新的 skill-up 评测结果。逐个判断失败来自 Skill 还是 eval：
+按需修复 SKILL.md 和相关文件，或修复 eval 用例与 Judge；为发现的问题
+补充回归用例，然后重新运行 skill-up，直到关键能力通过评测。
```
-用例 `id` 默认取文件名（这里是 `hello-world`）。只有在需要脚本评测或 Agent 评测时，才需要额外添加 `judge` 配置。
-
-### 第三步：校验配置
-
-```bash
-skill-up validate
-```
+这就是演进闭环：报告转化为修复，修复沉淀为回归用例，每轮迭代都会让
+Skill 和它的评测集更可靠。
-这一步是可选的，但建议首次运行前执行：它只检查 `eval.yaml` 和引用的用例文件，不会启动 Agent Engine。
+### 更喜欢手工配置？
-### 第四步：运行评测
+你仍然可以直接安装 CLI，并手写 `eval.yaml` 与用例文件：
```bash
-skill-up run
+curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash
```
-评测结果将写入 `-workspace/iteration-1/` 目录。
-
-### 从 Anthropic 格式导入
-
-```bash
-skill-up import ./evals/evals.json --output ./evals
-```
+详细步骤请查看官网的
+[快速开始](https://alibaba.github.io/skill-up/zh/guide/getting-started)、
+[编写评测](https://alibaba.github.io/skill-up/zh/guide/writing-evals)、
+[CLI 命令参考](https://alibaba.github.io/skill-up/zh/guide/cli-reference)和
+[用户配置](https://alibaba.github.io/skill-up/zh/guide/user-config)。
+Windows 的安装方式与已知限制请参阅
+[Windows 指南](https://alibaba.github.io/skill-up/zh/guide/windows)。
## CLI 命令概览
diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md
index fc48ef4a..59ff4036 100644
--- a/docs/guide/getting-started.md
+++ b/docs/guide/getting-started.md
@@ -1,14 +1,20 @@
# Getting Started
-skill-up is an evaluation tool for Agent Skill developers. Use it to verify that your Skill behaves correctly inside real Agent Engines (Claude Code, Codex, Qoder CLI) and to run continuous regression locally or in CI.
+skill-up is an evaluation and evolution tool for Agent Skill developers. Use it
+to verify that your Skill behaves correctly inside real Agent Engines (Claude
+Code, Codex, Qoder CLI), turn failures into targeted fixes, and run continuous
+regression locally or in CI.
---
-## Recommended Usage: AI-Assisted with skill-upper
+## Eval-to-Evolution with skill-upper
-For the best experience, use **skill-upper** — the Agent Skill shipped in this
-repository. It lets you ask an AI agent to scaffold, validate, run, and explain
-evals instead of hand-writing every YAML file first.
+For the best experience, use **skill-upper** — the conversational evolution
+driver shipped with this repository. It guides an AI agent to create evals, run
+skill-up, diagnose failures, repair the target Skill or its supporting files,
+strengthen the eval suite when it finds coverage gaps, and rerun the evaluation.
+This turns a single test run into a repeatable evaluate → diagnose → fix →
+rerun loop.
### 1. Install the `skill-upper` Agent Skill
@@ -26,7 +32,7 @@ You do not need to install `skill-up` before installing this Skill.
`skill-upper` checks whether the `skill-up` command is available when it runs
and guides the agent through installation if it is missing.
-### 2. Add and run evals
+### 2. Create and run the first evals
Open the target Skill project in your AI agent. The target project should have
this shape:
@@ -65,6 +71,22 @@ When `evals/eval.yaml` lives under a directory containing `SKILL.md`,
`skill-up` automatically installs that local Skill for the run, so you usually
do not need to list the Skill path manually in `eval.yaml`.
+### 3. Diagnose, fix, and iterate
+
+After the first run, continue the conversation instead of manually interpreting
+every report:
+
+```text
+Use skill-upper to inspect the latest skill-up results.
+Diagnose each failure, fix SKILL.md or its supporting files, and add or refine
+eval cases when the suite is missing coverage. Then rerun skill-up.
+Continue until the evals pass, or explain what remains blocked.
+```
+
+skill-upper uses the structured skill-up results to drive the next change. Each
+iteration keeps the Skill implementation and its eval suite aligned, so fixes
+become regression coverage rather than one-off patches.
+
---
## Manual Installation
diff --git a/docs/index.md b/docs/index.md
index 15a588a5..3bad5d4d 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -3,8 +3,8 @@ layout: home
hero:
name: skill-up
- text: Evaluation framework for Agent Skill developers
- tagline: Declare your eval environment, dependencies, test cases, and grading strategy in YAML — run it locally or in CI.
+ text: Evaluate and evolve Agent Skills
+ tagline: Measure Skill quality, then turn failures into automatic eval fixes and the next iteration.
image:
src: /logo.png
alt: skill-up
@@ -20,9 +20,9 @@ hero:
link: https://github.com/alibaba/skill-up
features:
- - title: AI-Assisted with skill-upper
- details: Use the skill-upper Agent Skill to create and run evals through natural conversation with AI agents like Cursor, Claude Code, or Qoder. No need to memorize CLI syntax.
- link: /guide/getting-started#recommended-usage-ai-assisted-with-skill-upper
+ - title: Eval-to-Evolution with skill-upper
+ details: Create evals through natural conversation, then let skill-upper diagnose failures, repair or expand cases, and rerun skill-up until the suite evolves.
+ link: /guide/getting-started#eval-to-evolution-with-skill-upper
linkText: Learn more
- title: Declarative Eval Config
details: Define evaluation environment, engine, model, and cases through YAML (eval.yaml + cases/*.yaml).
@@ -47,3 +47,11 @@ features:
link: /guide/cli-reference#exit-codes
linkText: Exit codes
---
+
+## Overview
+
+skill-up combines **evaluation** and **evolution** for Agent Skills. It makes quality measurable through declarative YAML evals, isolated multi-engine runs, flexible judges, and structured reports. Then skill-upper turns failures into progress: it can automatically repair or expand the eval suite and rerun the loop with you.
+
+The same workflow runs locally or in CI, supports Anthropic `evals.json` imports, and emits JSON, JUnit, and HTML reports.
+
+
diff --git a/docs/public/skill-up-overview.png b/docs/public/skill-up-overview.png
new file mode 100644
index 00000000..3e6c15b8
Binary files /dev/null and b/docs/public/skill-up-overview.png differ
diff --git a/docs/zh/guide/getting-started.md b/docs/zh/guide/getting-started.md
index 6ca673af..47c59528 100644
--- a/docs/zh/guide/getting-started.md
+++ b/docs/zh/guide/getting-started.md
@@ -1,14 +1,17 @@
# 快速上手
-skill-up 是一个面向 Agent Skill 开发者的评测工具。你可以用它来验证 Skill 在真实 Agent Engine（如 Claude Code、Codex、 Qodercli）中的功能正确性，并在本地或 CI 中持续回归。
+skill-up 是一个面向 Agent Skill 开发者的评测与演进工具。你可以用它验证
+Skill 在真实 Agent Engine（如 Claude Code、Codex、Qoder CLI）中的功能
+正确性，把失败转化为有针对性的修复，并在本地或 CI 中持续回归。
---
-## 推荐使用方式：AI 辅助配合 skill-upper
+## 使用 skill-upper 从评测到演进
-推荐使用仓库内置的 **skill-upper** Agent Skill。它会引导 AI Agent
-为目标 Skill 生成评测配置、校验、运行并解释结果，避免一开始就手写所有
-YAML。
+推荐使用仓库内置的 **skill-upper** Agent Skill，作为对话式的演进驱动器。
+它会引导 AI Agent 创建评测、运行 skill-up、诊断失败、修复目标 Skill
+或配套文件，在发现覆盖缺口时补充或改进 eval 用例，然后重新运行评测，
+把单次测试变成可重复的“评测 → 诊断 → 修复 → 重跑”闭环。
### 1. 安装 `skill-upper` Agent Skill
@@ -25,7 +28,7 @@ npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper
安装这个 Skill 前不需要先安装 `skill-up`。`skill-upper` 在运行时会检查
`skill-up` 命令是否可用；如果缺失，它会引导 Agent 完成安装。
-### 2. 添加与运行评测
+### 2. 创建并运行第一组评测
在 AI Agent 中打开目标 Skill 项目。目标项目至少应包含：
@@ -62,6 +65,20 @@ my-skill-workspace/
当 `evals/eval.yaml` 位于包含 `SKILL.md` 的目录下时，`skill-up` 会在运行时
自动安装这个本地 Skill，通常不需要在 `eval.yaml` 里手动写 Skill 路径。
+### 3. 诊断、修复并持续迭代
+
+首次运行后，不必手工逐条解读报告，可以继续与 Agent 对话：
+
+```text
+使用 skill-upper 检查最近一次 skill-up 的评测结果。
+逐项诊断失败，修复 SKILL.md 或配套文件；如果评测覆盖不足，
+补充或改进 eval 用例，然后重新运行 skill-up。
+持续迭代直到评测通过，或说明仍然受阻的原因。
+```
+
+skill-upper 会根据 skill-up 的结构化结果驱动下一次修改。每轮迭代都会让
+Skill 实现和 eval 评测集保持同步，使修复沉淀为回归保障，而不是一次性补丁。
+
---
## 手动安装
diff --git a/docs/zh/index.md b/docs/zh/index.md
index 2b3a58cf..920e2c6d 100644
--- a/docs/zh/index.md
+++ b/docs/zh/index.md
@@ -3,8 +3,8 @@ layout: home
hero:
name: skill-up
- text: 面向 Agent Skill 开发者的评测框架
- tagline: 用 YAML 声明评测环境、依赖、用例与评分策略，本地或 CI 中一键运行。
+ text: 评测并持续演进 Agent Skill
+ tagline: 度量 Skill 质量，再把失败转化为自动修复和下一轮迭代。
image:
src: /logo.png
alt: skill-up
@@ -20,9 +20,9 @@ hero:
link: https://github.com/alibaba/skill-up
features:
- - title: AI 辅助配合 skill-upper
- details: 使用 skill-upper Agent Skill，通过自然对话与 AI Agent（如 Cursor、Claude Code、Qoder 等）创建和运行评测，无需记忆 CLI 语法。
- link: /zh/guide/getting-started#推荐使用方式-ai-辅助配合-skill-upper
+ - title: skill-upper 从评测到演进
+ details: 通过自然对话创建评测，让 skill-upper 诊断失败、修复或补充用例，并持续重新运行 skill-up，推动评测集不断演进。
+ link: /zh/guide/getting-started#使用-skill-upper-从评测到演进
linkText: 了解更多
- title: 声明式评测配置
details: 通过 YAML（eval.yaml + cases/*.yaml）定义评测环境、引擎、模型与用例。
@@ -47,3 +47,11 @@ features:
link: /zh/guide/cli-reference#退出码
linkText: 退出码说明
---
+
+## 工作原理
+
+skill-up 将 Agent Skill 的**评测**与**演进**合为一个闭环：通过声明式 YAML 评测、隔离的多引擎运行、灵活的 Judge 和结构化报告，让质量可度量；再由 skill-upper 把失败转化为改进，自动修复或补充 eval 用例，并与你持续重跑和迭代。
+
+同一套流程既可在本地运行，也可接入 CI；同时兼容 Anthropic `evals.json` 导入，并输出 JSON、JUnit 和 HTML 报告。
+
+
diff --git a/skills/skill-upper/README.md b/skills/skill-upper/README.md
index 9442de55..039bd730 100644
--- a/skills/skill-upper/README.md
+++ b/skills/skill-upper/README.md
@@ -1,21 +1,23 @@
# skill-upper
-An Agent Skill that helps you set up, run, and interpret evaluations (evals) for other Agent Skills using the `skill-up` CLI.
+An Agent Skill that helps you evaluate and evolve other Agent Skills using the
+`skill-up` CLI.
## What it does
-`skill-upper` guides you through the full evaluation lifecycle:
+`skill-upper` guides you through an evaluation-to-evolution loop:
- **Locate** the target Skill and understand its capabilities
- **Scaffold** `evals/eval.yaml` and `evals/cases/*.yaml` with proper judge types
- **Validate** configuration before running
- **Run** evaluations against real Agent Engines (Claude Code, Codex, qodercli, etc.)
-- **Interpret** results — pass rates, failing assertions, benchmark deltas, and HTML reports
+- **Diagnose** failures from structured reports and output evidence
+- **Evolve** the Skill or strengthen eval coverage, then rerun the suite
## When to use
- You want to evaluate, test, or regress a Skill
+- You want to fix or iterate a Skill from eval failures
- You need to write `eval.yaml` / `case.yaml` or choose a judge type
- You're running `skill-up run/validate/list-cases/report/import/init`
- You're migrating from Anthropic `evals.json`
-
diff --git a/skills/skill-upper/README.zh.md b/skills/skill-upper/README.zh.md
index 8da72e91..a29afd39 100644
--- a/skills/skill-upper/README.zh.md
+++ b/skills/skill-upper/README.zh.md
@@ -1,21 +1,22 @@
# skill-upper
-一个帮助你使用 `skill-up` CLI 为 Agent Skill 搭建、运行和解读评测（evals）的 Agent Skill。
+一个帮助你使用 `skill-up` CLI 评测并持续演进其他 Agent Skill 的 Agent Skill。
## 功能概述
-`skill-upper` 引导你完成评测的完整生命周期：
+`skill-upper` 引导你完成从评测到演进的闭环：
- **定位** 目标 Skill，理解其能力边界
- **搭建** `evals/eval.yaml` 和 `evals/cases/*.yaml` 脚手架，选择合适的 judge 类型
- **校验** 配置，在运行前发现 schema 错误
- **运行** 评测，调用真实 Agent Engine（Claude Code、Codex、qodercli 等）
-- **解读** 结果——通过率、失败断言、基线对比和 HTML 报告
+- **诊断** 结构化报告和输出证据中的失败原因
+- **演进** 修复目标 Skill 或增强 eval 覆盖，然后重新运行评测
## 使用场景
- 需要对某个 Skill 进行评测、测试或回归验证
+- 需要根据评测失败修复并持续迭代某个 Skill
- 需要编写 `eval.yaml` / `case.yaml` 或选择 judge 类型
- 运行 `skill-up run/validate/list-cases/report/import/init`
- 从 Anthropic `evals.json` 迁移到 skill-up 格式
-
diff --git a/skills/skill-upper/SKILL.md b/skills/skill-upper/SKILL.md
index 3db27d0c..fda6833c 100644
--- a/skills/skill-upper/SKILL.md
+++ b/skills/skill-upper/SKILL.md
@@ -1,11 +1,11 @@
---
name: skill-upper
-description: "Set up, run, and interpret Agent Skill evaluations (evals) with the skill-up CLI / 使用 skill-up CLI 给 Agent Skill 搭建和运行评测. Use when the user asks to evaluate, test, regress, or verify a Skill; add evals or cases; write eval.yaml/case.yaml; run skill-up run/validate/list-cases/report/import/init; or migrate from Anthropic evals.json. Handles Skill discovery, evals scaffolding, judge authoring, credentials, user config, validation, runs, and reports."
+description: "Create, run, diagnose, and iteratively improve Agent Skill evaluations (evals) with the skill-up CLI / 使用 skill-up CLI 创建、运行、诊断并持续改进 Agent Skill 评测. Use when the user asks to evaluate, test, regress, verify, fix, improve, iterate, or evolve a Skill; add or strengthen eval cases; write eval.yaml/case.yaml; run skill-up run/validate/list-cases/report/import/init; or migrate from Anthropic evals.json. Handles Skill discovery, eval scaffolding, judge authoring, validation, runs, reports, and evidence-based repair loops."
---
# use-skill-up-cli
-Help the user set up, run, and interpret evaluations for Agent Skills via the `skill-up` CLI.
+Help the user evaluate and evolve Agent Skills through the `skill-up` CLI.
Manual:
@@ -54,6 +54,7 @@ my-skill/
Use this skill in any of the following situations:
- The user asks to "run / evaluate / verify / test this skill".
+- The user asks to "fix / improve / iterate / evolve this skill" from eval failures.
- The user wants to "add evals, test cases, or regression cases to a skill".
- The user wants to edit `eval.yaml` / `case.yaml`, or asks you to choose an appropriate `judge` type.
- The user mentions `skill-up run/validate/list-cases/report/import/init`.
@@ -181,6 +182,19 @@ Artifacts under `/-workspace/iteration-N/`:
Summarize: pass rate and timing; for failures, case id, assertion `text`, and `evidence`; benchmark deltas if enabled; offer HTML path or `skill-up report result.json --format html`.
+### Step 8: Evolve the Skill when requested
+
+Only enter this loop when the user asks to fix, improve, iterate, or evolve the
+target Skill. If the user only asks to evaluate or report results, stop after
+Step 7 without modifying it.
+
+1. Diagnose failures from `result.json`, `grading.json`, and output evidence.
+2. Fix `SKILL.md` or supporting files when the Skill behavior is incorrect.
+3. Add or refine eval cases when coverage is missing.
+4. Do not weaken valid assertions merely to make a failure pass.
+5. Rerun failed cases first, then run the full eval suite.
+6. Continue until the evals pass or clearly report what remains blocked.
+
## Command quick reference
| Command | Purpose |
diff --git a/skills/skill-upper/evals/cases/evolve-existing-skill.yaml b/skills/skill-upper/evals/cases/evolve-existing-skill.yaml
new file mode 100644
index 00000000..b66186ab
--- /dev/null
+++ b/skills/skill-upper/evals/cases/evolve-existing-skill.yaml
@@ -0,0 +1,36 @@
+id: evolve-existing-skill
+title: 根据评测失败修复 Skill 并补充回归用例
+description: |
+ 验证 skill-upper 在用户明确要求演进时，能够根据失败现象修复目标 Skill，
+ 同时补充回归用例，而不是删除或弱化已有评测。
+
+context:
+ repo_fixture: evals/fixtures/skills/sample-with-evals
+ git:
+ init: true
+
+input:
+ prompt: |
+ todo-manager 最近的评测发现：当没有待办事项时，Agent 没有明确告诉用户列表为空。
+ 请使用 skill-upper 演进这个 Skill：
+ 1. 修复 SKILL.md，明确空列表时的预期行为；
+ 2. 添加一个 ID 为 list-empty-todos 的 rule_based 回归用例；
+ 3. 保留且不要弱化已有的 add-todo 用例。
+
+expect:
+ files_exist:
+ - "SKILL.md"
+ - "evals/eval.yaml"
+ - "evals/cases/add-todo.yaml"
+ - "evals/cases/list-empty-todos.yaml"
+
+judge:
+ type: agent_judge
+ model: anthropic/claude-sonnet-4-6
+ criteria:
+ - "SKILL.md 明确说明没有待办事项时应告知用户列表为空"
+ - "新增 evals/cases/list-empty-todos.yaml，且 id 为 list-empty-todos、judge.type 为 rule_based"
+ - "新用例覆盖空待办列表场景，并检查输出能明确表达列表为空"
+ - "原有 evals/cases/add-todo.yaml 被保留，原有断言没有被删除或弱化"
+ - "eval.yaml 的 cases.files 同时引用 add-todo.yaml 和 list-empty-todos.yaml"
+ pass_threshold: 0.8
diff --git a/skills/skill-upper/evals/eval.yaml b/skills/skill-upper/evals/eval.yaml
index 437dc918..86b81b8a 100644
--- a/skills/skill-upper/evals/eval.yaml
+++ b/skills/skill-upper/evals/eval.yaml
@@ -17,6 +17,7 @@ cases:
- evals/cases/scaffold-with-qodercli-engine.yaml
- evals/cases/scaffold-with-script-judge.yaml
- evals/cases/english-context-generates-english-only-cases.yaml
+ - evals/cases/evolve-existing-skill.yaml
defaults:
# Budget sized to be engine-agnostic: claude_code finishes well under this,
# but slower coding agents (codex on a coder model) need more wall-clock and