Proposal: "Audit an agent skill" play based on a USENIX Security 2026 detection methodology

[sumleo]

The playbook has skills for MCP server auditing and prompt-injection testing, but no play for auditing an agent skill (SKILL.md + bundled code) before installation.

I'm the first author of a USENIX Security 2026 measurement study ("Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild, https://arxiv.org/abs/2602.06547). We built and evaluated a static + dynamic detection funnel (SkillScan) that narrowed 98,380 marketplace skills to 157 confirmed malicious ones. The decision criteria generalize into a step-by-step audit procedure: metadata/instruction consistency checks, shadow-feature detection (declared vs. actual behavior), credential-access patterns, exfiltration indicators, and publisher-cluster signals.

Proposal: I draft a new play "audit-agent-skill" following your play template, with checklists and output format (severity, CWE mapping, evidence snippet, remediation). Maintainers' guidance on template/format expectations welcome before I start.

[Mayur021]

This is a useful gap to close, and the SkillScan funnel maps cleanly onto a play. One dimension worth building into the severity scoring is the reversibility of the actions the skill can reach. A skill is a different audit object depending on whether its declared and actual behavior tops out at read-only, reversible, externally reversible, or irreversible action. Your shadow-feature check (declared versus actual) is where this bites hardest: a declared read-only skill that actually holds a credential-access or exfiltration path is not just an undeclared feature, it is a reversibility-class escalation from reversible to irreversible or external, and that is the case that should dominate severity.

Two concrete additions to the output format you proposed:

• a "max reversibility class reachable" field (read-only / reversible / externally reversible / irreversible) recorded for both declared and actual behavior, so the gap between them is explicit;
• severity weighted by that worst-case class rather than by CWE alone, since the same CWE is a very different incident when the reachable action is a read versus a deletion or an external send.

This lines up with how OWASP AISVS v1.0 frames action-class verification (C9.2.3 reversibility classification, C9.2.10 worst-case across a chain), so the play would slot next to an existing OWASP anchor.

[arian-gogani]

The pre-installation detection in sumleo's SkillScan and the design-time reversibility classification Mayur021 added cover the audit lifecycle up to the moment a skill is invoked. The third audit object that this play could anchor cleanly against is the execution itself.

OWASP AST09 was extended last week with an Execution Receipts subsection (PR #35 in the agentic-skills-top-10 repo, merged June 25) that defines a bilateral receipt schema for skill execution: an admission record signed before the skill runs (capturing the manifest hash, requested scope, declared identity, and policy decision) and an outcome record signed after the skill completes (capturing actual side effects and the post-execution result). Both records share a content-derived action_ref so a third-party verifier can confirm "this exact skill execution was admitted and produced this exact outcome" without trusting the runtime that emitted the records.

Where this composes with the proposed play:

1. SkillScan's static + dynamic findings can be referenced in the admission record's policy decision context. If the funnel returned "confirmed malicious", the admission record signs a DENY verdict and no execution occurs. If it returned "permitted", the admission record signs an ALLOW verdict with the SkillScan version and finding hash bound to that decision. Either way, the decision is provable at audit time.
2. Mayur021's max-reversibility-class field is exactly the kind of declared-vs-actual gap the bilateral pair makes detectable. Declared reversibility goes in the admission record, actual reversibility (or the irreversibility the skill ended up reaching) goes in the outcome record. The gap becomes machine-checkable rather than asserted.
3. Denied-before-dispatch carries equal audit weight to allowed-and-executed in this model. A skill that SkillScan rejected still produces an admission receipt with verdict DENY, which is the strongest evidence the system caught the violation before damage occurred. That is often more valuable to a regulator than the audit trail of skills that actually ran.
   The play template's severity / CWE mapping / evidence snippet / remediation slots already align well. The "evidence snippet" slot is where the admission_record + outcome_record pair fits as the post-deployment evidence the auditor can independently verify.

Happy to draft a short Execution Receipts cross-reference paragraph for the play if useful. The normative material is already in OWASP AST09 PR #35; the play would just need a one-paragraph pointer.

[Mayur021]

This composes well, and the declared-vs-actual split is the right home for the max-reversibility-class field. Declared class in the admission record, reached class in the outcome record, gap machine-checkable rather than asserted. Agreed on that.

One thing worth pinning down. The bilateral pair makes the gap checkable for a single execution. The failure mode that survives per-execution checking is composition across a chain: skill A and skill B each honor their declared class in their own admission and outcome records, and the chain still reaches an irreversibility no single skill declared. Each receipt verifies clean in isolation.

Evaluating that needs a verifier to walk the chain and compute worst-case reachable reversibility across the linked sequence, not just per-skill admission against outcome. That depends on whether the schema carries a causal link from one skill's outcome to the next skill's admission, beyond the attempt_id that pairs a single admission with its own outcome. If PR #35 already has that cross-execution reference, worst-case-across-chain is computable from the receipts and this is a one-line note in the play. If it does not, chain linkage is the gap worth naming, and it is the same worst-case-across-the-chain rule the AISVS C9 reversibility controls settled on.

The three-object framing is right regardless: pre-install detection, design-time class, execution receipt.