Optimize the docs skill trigger description

[ckrough]

Optimize the description frontmatter for trigger accuracy using the skill-creator optimization loop.

Part of #15. Depends on #18 and #20.

The description is the primary triggering mechanism. This skill has an unusually high overtrigger risk: agents touch things they would call documentation constantly, and most of them (ADRs, commit messages, changelogs, code comments, CLAUDE.md) are out of scope. A pushy description without tested negatives will fire on all of them.

This task is the mitigation for that risk, not polish.

Subtasks

• Write 20 trigger eval queries, realistic and specific, in the format {"query": "...", "should_trigger": true|false}
• Weight the negatives toward near-misses: writing an ADR, drafting a changelog entry, wording a commit message, editing a README, a general "how do I make a markdown table" question
• Include should-trigger cases where the user never says "documentation": moving a file into docs/, "this guide has gotten bloated", "where should this go"
• Review the eval set before running the loop
• Run the skill-creator optimization loop against the current session model
• Apply best_description and record the before/after scores in this issue

Acceptance Criteria

• 20 queries, roughly balanced between should-trigger and should-not-trigger
• Negative queries are genuine near-misses, not obviously irrelevant prompts
• The applied description scores better on the held-out test split than the draft it replaces
• The final description stays under the 1024-character agentskills.io limit
• Before and after scores recorded in this issue

[ckrough]

daedalus lead claimed

[ckrough]

Optimization loop: null result, and why

The trigger eval set is written and shipping. The description optimization is deferred. Recording the measurement failure here so the next attempt does not repeat it.

What was run

skill-creator/scripts/run_loop.py against the 20-query set, --model sonnet --max-iterations 5 --runs-per-query 3 --holdout 0.4 (12 train / 8 held-out test).

Iteration	Train	Test
1	6/12	4/8
2	6/12	4/8
3	6/12	4/8
4	6/12	4/8
5	6/12	4/8

Five different candidate descriptions, identical scores, and best_description came back byte-identical to the original (819 chars). Those numbers are exactly the floor: all 10 positives failed and all 10 negatives passed by never firing. That is not a signal about the description, it is the instrument reading zero.

Two defects in the harness

1. The skill is installed as a slash command. run_single_query writes <project_root>/.claude/commands/<name>.md. Current Claude Code does not surface a project slash command in the model's skill list, so the Skill tool never fires no matter what the description says. Verified the counterfactual: copying the real SKILL.md to .claude/skills/docs/SKILL.md and running a positive query produced two Skill tool uses.

2. Detection aborts on the first non-Skill tool call. The stream handler does if tool_name in ("Skill", "Read") ... else: return False. Realistic multi-step queries make the model reach for Bash, Glob, or Grep first, so every substantive positive scores as a miss. This defect penalizes exactly the substantive queries that skill-creator's own guidance says to write.

Both were patched in a scratch copy of the harness outside this repo. The loop still returned a flat null result, so a third cause remains: a dynamically-named stub skill written into .claude/skills/<name>-skill-<uuid>/ does not enter the model's skill list the way a normal skill directory does.

Disposition

scriptorium/skills/docs/SKILL.md is unchanged. Applying a rewrite the harness scored identically to the original would have meant claiming a measured improvement that was never measured.

Acceptance criteria status:

• 20 queries, 10 should-trigger / 10 should-not-trigger
• Negatives are genuine near-misses (ADR, release notes, commit message, README trim, markdown table syntax, docstring rewrite, CLAUDE.md edit, a factual-accuracy fix, a naming scheme, an accessibility audit)
• Applied description beats the draft on the held-out split: blocked on the harness
• Description under 1024 chars (unchanged, 819)
• Before/after scores recorded: before is 4/8 on the held-out split, after does not exist

Criteria 3 and 5 need a working trigger harness. That is worth its own issue: fix the two defects above, find the third, then re-run this eval set unchanged.

[ckrough]

Parked: turn budget exhausted before the review gate returned

Blocking the pipeline run rather than shipping unreviewed. No PR was opened.

Parked work: scriptorium/skills/docs/evals/trigger-evals.json, stashed on branch issue/21 as 1b63886516902a647c789ad67e266f67180e0fd3 (worktree .claude/worktrees/issue-21). The stash lives in the shared object store and survives the worktree. Resume with /daedalus:run-pipeline resume 21; the checkpoint carries the same ref.

State at park:

Phase	Result
Eval set authored	20 entries, 10 positive / 10 negative, all unique, schema-conformant
Test gate	bash .daedalus/gates/structure.sh exit 0
Lint gate	claude plugin validate . exit 0 (6 warnings, identical count on main, pre-existing)
Optimization loop	Null result, diagnosed in the previous comment
Test-validator	Spawned, produced no output before the budget ran out
Review-gate	Spawned, produced no output before the budget ran out

Why blocked and not shipped: review-gate APPROVE is a hard precondition for the commit, and a deterministic gate result outranks my own read of the diff. The eval set looks correct to me and every mechanical check passes, but "looks correct to me" is not the verdict the pipeline requires. The two judges never reported, so there is no verdict to act on.

What a resume needs to do: restore the stash, re-run the test-validator and review-gate on the restored diff, then commit and open the PR. The scope decision is already settled: ship the eval set alone, leave SKILL.md unchanged, and open a follow-up for the trigger-harness defects.

[ckrough]

PR #34 is open with the eval set: #34

Branch issue/21, commit 87470f0. Review-gate APPROVE, test and lint both exit 0, /simplify NO-OP, security review clean. Two of the test-validator's four advisories were folded in before commit: one of a redundant positive pair was replaced with a never-scaffold-precondition query, and three queries that echoed SKILL.md's own worked examples were paraphrased into different domains so they test generalization rather than recall.

This issue stays open. Criteria 3 and 5 are carried by #35, which also picks up the SKILL.md README seam and the missing CONTRIBUTING.md note.

[ckrough]

Trigger-harness status from #35, relevant to acceptance criteria 3 and 5 here.

The harness that blocked this issue is fixed and lands in the #35 PR: scripts/evals/run_trigger_evals.py, stdlib-only, with a 19-file grounding fixture. A 2-query smoke scores 1.00 accuracy (positive fires, negative declines), against the previous floor where five different descriptions all scored identically at train 6/12 and test 4/8.

The scored runs are deferred to this issue, with a corrected budget. Measured cost is ~$0.80/query blended, not the $0.10-$0.15 the #35 plan projected, so five descriptions across 20 queries is ~$80. Positives are cheap (the harness kills the subprocess the moment the Skill call appears); negatives run the task to completion and dominate the bill. Budget accordingly, or cut per-query cost first by bounding max turns and dropping Bash from --tools.

No before/after description scores exist yet, so this issue's criteria 3 and 5 stay open. The incumbent description (810 chars after the README seam fix) and a deliberately vague control are checked in at scriptorium/evals/candidates/docs-2026-08.json as the starting pair.