Skip to content

docs: name the real skill (annotate) in the 50->100 case study #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DCCA merged 1 commit into from
Jun 23, 2026
Merged

docs: name the real skill (annotate) in the 50->100 case study #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/examples/skill-benchmark/from-the-web.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,10 @@ finding and scores the skill. Pinned in
[`tests/test_precision.py`](../../../tests/test_precision.py)
(`test_invalid_yaml_frontmatter_scores_not_crashes`).

## Case study — invalid frontmatter: 50 → 100
## Case study — `annotate`: 50 → 100

A real-world bug class: a `description` with an unquoted colon.
[`annotate`](https://github.com/glebis/claude-skills) (from glebis/claude-skills) — a real
published skill whose `description` has an unquoted colon, scored verbatim:

| | Score | Grade | Verdict | Finding |
|---|------:|-------|---------|---------|
Expand Down
6 changes: 3 additions & 3 deletions docs/examples/skill-benchmark/improved/web-frontmatter.after.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,9 +89,9 @@
},
"findings": [],
"provenance": {
"source": "community/claude-skills/annotate",
"source": "github.com/glebis/claude-skills \u00b7 annotate",
"kind": "dir",
"resolved_path": "community/claude-skills/annotate"
"resolved_path": "github.com/glebis/claude-skills \u00b7 annotate"
},
"metadata": {
"mode": "structural-only",
Expand All @@ -101,7 +101,7 @@
"confidence": "medium",
"also": []
},
"timestamp": "2026-06-23T01:49:56Z",
"timestamp": "2026-06-23T10:59:29Z",
"skipped_dimensions": [
"D2",
"D3",
Expand Down
2 changes: 1 addition & 1 deletion docs/examples/skill-benchmark/improved/web-frontmatter.after.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Skill Scorecard — `annotate`
Source: community/claude-skills/annotate · mode: structural-only
Source: github.com/glebis/claude-skills · annotate · mode: structural-only
Type: task (confidence: medium)

## ██████████████ 100 / 100 Grade: A Verdict: Ship
Expand Down
6 changes: 3 additions & 3 deletions docs/examples/skill-benchmark/improved/web-frontmatter.before.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,9 @@
}
],
"provenance": {
"source": "community/claude-skills/annotate",
"source": "github.com/glebis/claude-skills \u00b7 annotate",
"kind": "dir",
"resolved_path": "community/claude-skills/annotate"
"resolved_path": "github.com/glebis/claude-skills \u00b7 annotate"
},
"metadata": {
"mode": "structural-only",
Expand All @@ -127,7 +127,7 @@
"confidence": "medium",
"also": []
},
"timestamp": "2026-06-23T01:49:56Z",
"timestamp": "2026-06-23T10:59:29Z",
"skipped_dimensions": [
"D2",
"D3",
Expand Down
4 changes: 2 additions & 2 deletions docs/examples/skill-benchmark/improved/web-frontmatter.before.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Skill Scorecard — `annotate`
Source: community/claude-skills/annotate · mode: structural-only
# Skill Scorecard — `claude-skills · annotate`
Source: github.com/glebis/claude-skills · annotate · mode: structural-only
Type: task (confidence: medium)

## ███████░░░░░░░ 50 / 100 Grade: D Verdict: Revise
Expand Down
14 changes: 8 additions & 6 deletions docs/examples/skill-benchmark/improved/web-frontmatter.change.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,26 @@
# web-frontmatter — the fix

A bug class found in **real published skills** on GitHub: the `description` contains an
unquoted colon, so the YAML frontmatter is invalid and the skill has no usable
name/description (it won't trigger or load correctly).
A real published skill: **`annotate`** from
[glebis/claude-skills](https://github.com/glebis/claude-skills). Its `description` contains an
unquoted colon (`… Local-only: synthetic or consented data only …`), so the YAML frontmatter
is invalid and the skill has no usable name/description — it won't trigger or load correctly.
Scored verbatim, before and after the one-line fix.

**Before** (invalid YAML — the colon after `Local-only` starts a YAML mapping):

```yaml
---
name: annotate
description: Build a PII gold set. Local-only: synthetic data only, nothing leaves the machine.
description: Build and verify a PII gold set … Local-only: synthetic or consented data only; …
---
```

**After** (skval's finding applied — quote the description so the colon is safe):
**After** (skval's finding applied — wrap the description in quotes so the colon is safe):

```yaml
---
name: annotate
description: "Build a PII gold set. Local-only: synthetic data only, nothing leaves the machine. Use when labeling PII spans or measuring inter-annotator agreement (kappa)."
description: "Build and verify a PII gold set … Local-only: synthetic or consented data only; …"
---
```

Expand Down
4 changes: 2 additions & 2 deletions docs/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,8 @@ <h2>Benchmark</h2>
<h3 style="color:var(--white);font-size:19px;margin:28px 0 4px">The upgrade: score → fix → re-score</h3>
<p class="lead" style="margin:0 0 14px">skval doesn't just grade — its ranked findings drive the fix. The biggest real-world turnarounds:</p>
<div class="grid2">
<div class="card"><h3>web skill · invalid frontmatter · <span class="warn">50</span> → <span class="A">100</span></h3>
<p style="color:var(--dim);margin:0">A real published skill whose unquoted colon broke its YAML — it used to <em>crash</em> skval, and now scores. One-line fix: quote the description. <code>compare.py</code>: <b class="A">+50</b>, Revise → Ship. <a href="https://github.com/DCCA/skval/blob/main/docs/examples/skill-benchmark/from-the-web.md">case study →</a></p></div>
<div class="card"><h3><a href="https://github.com/glebis/claude-skills">annotate</a> · invalid frontmatter · <span class="warn">50</span> → <span class="A">100</span></h3>
<p style="color:var(--dim);margin:0">A real published skill (glebis/claude-skills) whose unquoted colon broke its YAML — it used to <em>crash</em> skval, and now scores. One-line fix: quote the description. <code>compare.py</code>: <b class="A">+50</b>, Revise → Ship. <a href="https://github.com/DCCA/skval/blob/main/docs/examples/skill-benchmark/from-the-web.md">case study →</a></p></div>
<div class="card"><h3>bad-skill · <span class="warn">73</span> → <span class="A">100</span></h3>
<p style="color:var(--dim);margin:0">Four findings fixed one-for-one — kebab name, drop <code>&lt;&gt;</code>, broken ref, stray key. <code>compare.py</code>: <b class="A">+27</b>, Revise → Ship. <a href="https://github.com/DCCA/skval/tree/main/docs/examples/skill-benchmark/improved">case study →</a></p></div>
</div>
Expand Down
Loading