Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions docs/examples/skill-benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,12 @@ clean**, the rest splitting into vendored-duplicate `SKILL.md` (e.g. [nextjs](ra

## From the web — real published skills

Beyond installed plugins, **[from-the-web.md](from-the-web.md)** scores **69 community
skills** pulled from three public GitHub collections (every one safety-scanned as inert
text first — 0 tripped the gate, none executed). It surfaced a real bug class —
**invalid YAML frontmatter** (an unquoted colon in `description`) — that used to *crash*
skval; skval now scores it instead, and the
[50 → 100 case study](from-the-web.md#case-study--invalid-frontmatter-50--100) shows the
one-line fix.
Beyond installed plugins, **[from-the-web.md](from-the-web.md)** scores **123 community
skills** pulled from four public GitHub collections (every one safety-scanned as inert
text first — 0 tripped the gate, none executed). It surfaced real, fixable classes —
missing/invalid frontmatter and `<>` in the description — and several
[score → fix → re-score case studies](from-the-web.md#case-studies--score--fix--re-score),
the biggest being **`daydream` 38 → 100** (no frontmatter → Ship).

## Case study 1 — `docx`: 92 → 100 (real skill)

Expand Down
42 changes: 25 additions & 17 deletions docs/examples/skill-benchmark/from-the-web.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# From the web — skval over real published skills

A real `skval` **structural** run over **69 community skills** pulled from three public
A real `skval` **structural** run over **123 community skills** pulled from four public
GitHub collections — [alirezarezvani/claude-skills](https://github.com/alirezarezvani/claude-skills),
[glebis/claude-skills](https://github.com/glebis/claude-skills), and
[smartnews/claude-skills](https://github.com/smartnews/claude-skills) — deterministic, no
model calls. Most are well-built; this page is about the ones skval flags, and what fixing
them looks like.
[glebis/claude-skills](https://github.com/glebis/claude-skills),
[smartnews/claude-skills](https://github.com/smartnews/claude-skills), and
[anthropics/skills](https://github.com/anthropics/skills) — deterministic, no model calls.
Most are well-built; this page is about the ones skval flags, and what fixing them looks like.

> **Safety first.** Every fetched skill was treated as **inert text** and run through skval's
> D6 safety gate *before* anything else. **0 / 69** tripped it, and none were ever executed —
> D6 safety gate *before* anything else. **0 / 123** tripped it, and none were ever executed —
> only the `SKILL.md` text was read and statically analysed.
>
> *(Symlink/pointer entries — where the raw file is just a path to another location — were
Expand All @@ -33,20 +33,28 @@ finding and scores the skill. Pinned in
[`tests/test_precision.py`](../../../tests/test_precision.py)
(`test_invalid_yaml_frontmatter_scores_not_crashes`).

## Case study`annotate`: 50 → 100
## Case studiesscore → fix → re-score

[`annotate`](https://github.com/glebis/claude-skills) (from glebis/claude-skills) — a real
published skill whose `description` has an unquoted colon, scored verbatim:
Real published skills, scored verbatim, then fixed against skval's ranked findings and
re-scored. Every "after" is the actual re-score of the fixed skill (no remaining findings).

| | Score | Grade | Verdict | Finding |
|---|------:|-------|---------|---------|
| **Before** ([scorecard](improved/web-frontmatter.before.md)) | 50 | D | Revise | invalid YAML frontmatter |
| **After** ([scorecard](improved/web-frontmatter.after.md)) | 100 | A | Ship | — |
| Skill | Source | Finding class | Before | After | Δ |
|-------|--------|---------------|:------:|:-----:|:--:|
| [`daydream`](https://github.com/glebis/claude-skills) | glebis | no frontmatter at all | [38 / F / Reject](improved/web-daydream.before.md) | [100 / A / Ship](improved/web-daydream.after.md) | **+62** |
| [`disk-cleanup`](https://github.com/glebis/claude-skills) | glebis | invalid YAML frontmatter | [50 / D / Revise](improved/web-disk-cleanup.before.md) | [100 / A / Ship](improved/web-disk-cleanup.after.md) | **+50** |
| [`annotate`](https://github.com/glebis/claude-skills) | glebis | invalid YAML frontmatter | [50 / D / Revise](improved/web-frontmatter.before.md) | [100 / A / Ship](improved/web-frontmatter.after.md) | **+50** |
| [`rehydrate`](https://github.com/glebis/claude-skills) | glebis | `<` / `>` in description | [92 / A / Ship](improved/web-rehydrate.before.md) | [100 / A / Ship](improved/web-rehydrate.after.md) | **+8** |

The fix skval points to — **quote the description** — is one line. skval's own diff
([`improved/web-frontmatter.compare.json`](improved/web-frontmatter.compare.json)):
`overall_delta: +50`, verdict `Revise → Ship`. What changed:
[`improved/web-frontmatter.change.md`](improved/web-frontmatter.change.md).
**The fixes are tiny, one each:**

- **`daydream`** had *no YAML frontmatter* — no `name`, no `description`, so it can't trigger.
Adding a frontmatter block takes it from **38 / F / Reject → 100 / A / Ship** (skval's biggest
turnaround in the set, `overall_delta +62`). What changed:
[`web-frontmatter.change.md`](improved/web-frontmatter.change.md) shows the same class on `annotate`.
- **`disk-cleanup` / `annotate`** ship an unquoted colon in `description` (`… Local-only: synthetic …`)
that breaks the YAML — quote the description and both go **50 → 100**.
- **`rehydrate`** has `<` / `>` in `description` (breaks parsers / triggering) — drop the angle
brackets, **92 → 100**.

---

Expand Down
112 changes: 112 additions & 0 deletions docs/examples/skill-benchmark/improved/web-daydream.after.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
{
"score": 100,
"grade": "A",
"verdict": "Ship",
"safety_pass": true,
"dimensions": {
"D1": {
"score": 1.0,
"weight": 0.15,
"checks": [
{
"id": "frontmatter_present",
"passed": true,
"severity": "critical",
"detail": ""
},
{
"id": "frontmatter_valid_yaml",
"passed": true,
"severity": "critical",
"detail": ""
},
{
"id": "name_present",
"passed": true,
"severity": "critical",
"detail": ""
},
{
"id": "name_kebab_case",
"passed": true,
"severity": "major",
"detail": ""
},
{
"id": "description_present",
"passed": true,
"severity": "critical",
"detail": ""
},
{
"id": "description_len",
"passed": true,
"severity": "major",
"detail": ""
},
{
"id": "description_no_angle_brackets",
"passed": true,
"severity": "major",
"detail": ""
},
{
"id": "allowed_frontmatter_keys",
"passed": true,
"severity": "minor",
"detail": ""
},
{
"id": "single_skill_md",
"passed": true,
"severity": "critical",
"detail": ""
},
{
"id": "skill_md_line_budget",
"passed": true,
"severity": "minor",
"detail": ""
},
{
"id": "token_budget",
"passed": true,
"severity": "minor",
"detail": ""
},
{
"id": "no_broken_local_refs",
"passed": true,
"severity": "major",
"detail": ""
}
]
}
},
"safety": {
"safety_pass": true,
"findings": []
},
"findings": [],
"provenance": {
"source": "github.com/glebis/claude-skills \u00b7 daydream",
"kind": "dir",
"resolved_path": "github.com/glebis/claude-skills \u00b7 daydream"
},
"metadata": {
"mode": "structural-only",
"skill_name": "daydream",
"classification": {
"type": "task",
"confidence": "medium",
"also": []
},
"timestamp": "2026-06-26T15:59:31Z",
"skipped_dimensions": [
"D2",
"D3",
"D4",
"D5"
]
}
}
14 changes: 14 additions & 0 deletions docs/examples/skill-benchmark/improved/web-daydream.after.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Skill Scorecard — `daydream`
Source: github.com/glebis/claude-skills · daydream · mode: structural-only
Type: task (confidence: medium)

## ██████████████ 100 / 100 Grade: A Verdict: Ship

| Dimension | Score | Weight | Detail |
|-----------|-------|--------|--------|
| D1 Structural | 1.00 | 0.15 | all checks pass |

Safety gate: PASS
Skipped dimensions (structural-only): D2, D3, D4, D5

No findings — clean.
143 changes: 143 additions & 0 deletions docs/examples/skill-benchmark/improved/web-daydream.before.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
{
"score": 38,
"grade": "F",
"verdict": "Reject",
"safety_pass": true,
"dimensions": {
"D1": {
"score": 0.3846,
"weight": 0.15,
"checks": [
{
"id": "frontmatter_present",
"passed": false,
"severity": "critical",
"detail": "missing leading '---'"
},
{
"id": "frontmatter_valid_yaml",
"passed": false,
"severity": "critical",
"detail": "no YAML frontmatter found"
},
{
"id": "name_present",
"passed": false,
"severity": "critical",
"detail": "missing 'name'"
},
{
"id": "name_kebab_case",
"passed": false,
"severity": "major",
"detail": "name '' must be kebab-case, <=64 chars"
},
{
"id": "description_present",
"passed": false,
"severity": "critical",
"detail": "missing 'description'"
},
{
"id": "description_len",
"passed": false,
"severity": "major",
"detail": "description is 0 chars (max 1024)"
},
{
"id": "description_no_angle_brackets",
"passed": true,
"severity": "major",
"detail": ""
},
{
"id": "allowed_frontmatter_keys",
"passed": true,
"severity": "minor",
"detail": ""
},
{
"id": "single_skill_md",
"passed": true,
"severity": "critical",
"detail": ""
},
{
"id": "skill_md_line_budget",
"passed": true,
"severity": "minor",
"detail": ""
},
{
"id": "token_budget",
"passed": true,
"severity": "minor",
"detail": ""
},
{
"id": "no_broken_local_refs",
"passed": true,
"severity": "major",
"detail": ""
}
]
}
},
"safety": {
"safety_pass": true,
"findings": []
},
"findings": [
{
"dimension": "D1",
"impact_estimate": 3,
"message": "missing leading '---'"
},
{
"dimension": "D1",
"impact_estimate": 3,
"message": "no YAML frontmatter found"
},
{
"dimension": "D1",
"impact_estimate": 3,
"message": "missing 'name'"
},
{
"dimension": "D1",
"impact_estimate": 3,
"message": "missing 'description'"
},
{
"dimension": "D1",
"impact_estimate": 2,
"message": "name '' must be kebab-case, <=64 chars"
},
{
"dimension": "D1",
"impact_estimate": 2,
"message": "description is 0 chars (max 1024)"
}
],
"provenance": {
"source": "github.com/glebis/claude-skills \u00b7 daydream",
"kind": "dir",
"resolved_path": "github.com/glebis/claude-skills \u00b7 daydream"
},
"metadata": {
"mode": "structural-only",
"skill_name": null,
"classification": {
"type": "task",
"confidence": "medium",
"also": []
},
"timestamp": "2026-06-26T15:59:31Z",
"skipped_dimensions": [
"D2",
"D3",
"D4",
"D5"
]
}
}
20 changes: 20 additions & 0 deletions docs/examples/skill-benchmark/improved/web-daydream.before.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Skill Scorecard — `claude-skills · daydream`
Source: github.com/glebis/claude-skills · daydream · mode: structural-only
Type: task (confidence: medium)

## █████░░░░░░░░░ 38 / 100 Grade: F Verdict: Reject

| Dimension | Score | Weight | Detail |
|-----------|-------|--------|--------|
| D1 Structural | 0.38 | 0.15 | 6/12 checks failed |

Safety gate: PASS
Skipped dimensions (structural-only): D2, D3, D4, D5

## Top findings (ranked by score impact)
1. [D1] missing leading '---'
2. [D1] no YAML frontmatter found
3. [D1] missing 'name'
4. [D1] missing 'description'
5. [D1] name '' must be kebab-case, <=64 chars
6. [D1] description is 0 chars (max 1024)
11 changes: 11 additions & 0 deletions docs/examples/skill-benchmark/improved/web-daydream.compare.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"overall_delta": 62,
"dimensions": {
"D1": {
"a": 0.3846,
"b": 1.0,
"delta": 0.6154
}
},
"verdict_change": "Reject -> Ship"
}
Loading
Loading