AI Instruction Stress Test — Find Trigger and Behavior Failures Before Release

Find out how your SKILL.md fails before a real session pays for it.

Wind-tunnel testing for instruction text — skills, system prompts, CLAUDE.md rules, agent definitions, plugin commands. It casts a small set of personas from your real contexts, replays realistic utterances against the trigger surface alone, then walks the body line-by-line per scenario and reports what actually happens. Every finding is anchored to a line number; description problems come with a paste-ready rewrite.

What it does

• Predicts selection from the target harness's pre-load metadata, commonly the name and description. Always-loaded prompts have no selection step; unknown loader behavior is stated as an assumption.
• Hunts named failure classes: false-fires, missed triggers, sibling-skill collisions, stale-world assumptions, silent failures (flows that "succeed" and deliver a wrong result), and stranger-usability gaps.
• Reports verdict-first: the first paragraph answers "would you ship this today, and what is the one fix."
• Stays read-only: it reports on the artifact, never edits it.
• Optionally emits traces.jsonl so failure clusters can feed an eval pipeline.

When it fires

• "Probe this skill before I publish it."
• "Wind-tunnel this system prompt."
• "Will this description misfire on unrelated requests?"
• "I just rewrote my agent definition — stress-test it."
• 「帮我风洞一下这个 skill」
• 「这个 description 会不会误触?」

Install

npx skills add m1nga/scenario-probe

Reproducible simulation fixture

Authored illustration, not an independent agent run. Give the probe this complete miniature instruction:

description: Use when asked to summarize the provided meeting notes.
body: Summarize only the notes in three bullets. If notes are absent, request them.
      Do not invent owners, dates, or decisions.

Try “summarize these notes”, “uh, recap these meeting notes in three points”, “translate this sentence”, and “summarize my meeting” with no notes. Expected predictions: hit, hit, sleep, then request the missing input. Add any actual co-installed sibling to check overlap; without its metadata, collision status is unknown. If nothing else fails, zero findings is a valid bounded result.

Longer example (illustrative)

You: I rewrote the description of my deploy-checklist skill. Wind-tunnel it before I push.

Claude casts four personas (you dictating on your phone, you mid-deploy, a fresh machine with no config files, a stranger who installed only this skill), writes nine utterances, predicts fires/sleeps from the description alone, then walks the body for each scenario that fires. The report opens: "Not yet — 'deploy' is claimed so broadly that this collides with your release-notes skill on 'prep the release', and the rollback section reads a file that does not exist on a fresh install, with no defined fallback." Each finding cites a line; the description fix is ready to paste.

Works well with

• write-judge-prompt + validate-evaluator — the probe's traces.jsonl is trace supply for products with no production traffic yet: failure clusters become judge criteria, then the judge gets calibrated. Wind-tunnel → judge → calibration is a pipeline.
• product-experience-officer — the cousin seat. A runnable product with a UI gets PEO; instruction text gets scenario-probe. Both distinguish simulated predictions from observed execution; a supported zero-finding result is allowed.

Design notes

This methodology was extracted from a real audit, not designed in the abstract. Before publishing, it ran across a solo builder's private library of 23 skills — 140+ persona × scenario runs in one pass. That audit surfaced problems no single-skill review had caught: systemic trigger-layer debt (descriptions written for the author's own phrasing, so paraphrases slept), a family of silent failures (skills that reported "saved" with nowhere to save, or invented numbers to satisfy a mandated output schema), and split-brain copies (the same skill duplicated across two harnesses, drifted apart, each copy auditing clean on its own).

Three opinionated choices follow from that audit:

• Trigger surface first, body later. Most findings were first-5-seconds failures — the wrong skill fired, or none did. Reading the body before predicting triggers contaminates the prediction, so the phases are ordered to make that impossible.
• Personas come from the owner's real contexts, not a fixed checklist. The highest-yield persona in the original audit was "the owner at their messiest" — dictated input, goal buried in the last sentence. A generic cast would not have included it. The stranger persona is mandatory only when publishing, because that is the moment it can change the verdict.
• Check coverage without forcing defects. If a clean simulation missed a meaningful boundary, recast once. Otherwise report zero findings, with real trigger selection and tool execution still unverified.

Evidence

Historical scenario probes (simulated)

Probed 7 scenarios across 4 personas · 3 fired correctly · 3 correctly stayed quiet · the 7th run was the probe auditing itself.

"帮我 stress-test 一下这个上线计划" ("stress-test my launch plan") → stayed quiet. Plans are grilling's seat; this skill only takes instruction text — and the exclusion lives in the description, where triggering actually happens.

"I just rewrote my CLAUDE.md commit rules — stress-test them before I rely on it." → fired: cast four personas, predicted triggers from the description alone, then walked the rule body line-by-line per scenario.

"测一下这个产品" ("test this product for me") → stayed quiet. Runnable products belong to product-experience-officer; the probe practices the boundary it preaches.

Run on its own SKILL.md, it flagged that its description sits 17 characters under the 1024 loader cap — with the NOT-clauses last in line to be truncated. It got the same treatment as everything else.

Probe method: scenario-probe

Author

Built and maintained by Ming.