Affected file: src/skillspector/nodes/analyzers/mcp_tool_poisoning.py
Summary
python
_TP4_EXECUTABLE_TYPES = frozenset(
{"python", "javascript", "typescript", "shell", "ruby", "go", "rust"}
)
...
executable_type_by_path = {
str(metadata.get("path")): str(metadata.get("type"))
for metadata in component_metadata
if isinstance(metadata, dict) and metadata.get("type") in _TP4_EXECUTABLE_TYPES
}
executable_paths = [
path
for path, content in file_cache.items()
if path in executable_type_by_path
...
]
TP4 gathers "actual code" for its declared-purpose-vs-behavior LLM check only from components whose type metadata is one of the seven executable language types above. Components typed markdown/text are excluded from executable_type_by_path entirely, so any code embedded in a .md file — including fenced code blocks meant to be copy-executed, which is a common and legitimate Agent Skills authoring pattern — is invisible to TP4. This is independent of file size, content, or any suppression config; the code never enters the batch-construction step.
Confirmed on two independent skills from the official anthropics/skills repo:
pdf — TP4 reported 8+ of 10 declared capabilities (merge, split, OCR, watermark, etc.) as having "zero implementation." In fact SKILL.md itself contains full, working inline Python for merge/split/rotate/ watermark/OCR using pypdf/pdfplumber/reportlab/pytesseract — the skill's actual delivery mechanism for most of its capabilities is documented, copy-executable snippets in the markdown, not pre-packaged .py scripts. TP4 only sees the small scripts/*.py helpers and reports the rest as unimplemented.
mcp-builder — TP4 reported "zero server-creation logic" for the skill's core purpose, counting only two small executable .py files. The skill's actual server-building example code lives in two markdown reference guides (970 and 719 lines), which TP4 never inspects.
Both fired as TP4/HIGH-or-MEDIUM findings and survived the LLM semantic layer which only adjudicates the code it's shown — it does not know about code TP4 never gathered in the first place.
Relation to existing issues
Related to, but distinct from, #135 ("Static pattern analyzers fire on markdown documentation and code blocks," closed via #140). #135 covers the 12 regex-based static analyzers (static_runner.py's is_code_example() / _NON_EXECUTABLE_FILE_TYPES) over-firing on markdown — the opposite direction of this issue. TP4 is a separate, LLM-based analyzer (mcp_tool_poisoning.py) with its own code-gathering path, and its problem is under-crediting markdown as "no implementation," not over-flagging it. Confirmed directly against the current main (well downstream of #135/#140) that _TP4_EXECUTABLE_TYPES still excludes markdown — #140 did not touch this code path.
Why this isn't a baseline/suppression problem
Unlike the PE3/EA1 false positives fixed in #415/#417, this can't be addressed with a suppression rule: a global id: TP4 rule would disable the check entirely, including its legitimate catches (a skill that claims a capability it implements nowhere — not even in markdown — is a real finding), and there's no stable message substring to scope a narrower rule to, since the mismatched-capability text differs per skill and per finding.
Proposed direction (not a full fix — wanted maintainer input on approach)
Extend the code-gathering step to also extract fenced code blocks from markdown-typed components (by declared language, e.g. ```python) and feed them into the same batching/token-budget pipeline TP4 already uses for .py/.js/etc. files, rather than gating on component type alone. Open questions worth maintainer input before implementation:
How to distinguish a fenced block meant to be copy-executed (this skill's actual delivery mechanism) from an illustrative/incomplete example not meant to run standalone — misclassifying the latter as "actual behavior" could introduce new false positives in the other direction.
Token-budget impact: markdown reference docs can be large (970+ lines in the mcp-builder case above); including their code fences changes TP4's cost/latency profile, which may need its own limits distinct from TP4_MAX_FILE_CODE_BYTES.
Happy to help scope/implement once there's agreement on direction — this one felt like it needed a design decision from the maintainers rather than a drop-in regex fix like #415/#417.
Reported by
Benedict Kwok, ZTAI Security Advisors LLC (benedictkwok@ztai.ai) — found while independently evaluating SkillSpector against several official anthropics/skills skills.
Affected file: src/skillspector/nodes/analyzers/mcp_tool_poisoning.py
Summary
python
_TP4_EXECUTABLE_TYPES = frozenset(
{"python", "javascript", "typescript", "shell", "ruby", "go", "rust"}
)
...
executable_type_by_path = {
str(metadata.get("path")): str(metadata.get("type"))
for metadata in component_metadata
if isinstance(metadata, dict) and metadata.get("type") in _TP4_EXECUTABLE_TYPES
}
executable_paths = [
path
for path, content in file_cache.items()
if path in executable_type_by_path
...
]
TP4 gathers "actual code" for its declared-purpose-vs-behavior LLM check only from components whose type metadata is one of the seven executable language types above. Components typed markdown/text are excluded from executable_type_by_path entirely, so any code embedded in a .md file — including fenced code blocks meant to be copy-executed, which is a common and legitimate Agent Skills authoring pattern — is invisible to TP4. This is independent of file size, content, or any suppression config; the code never enters the batch-construction step.
Confirmed on two independent skills from the official anthropics/skills repo:
pdf — TP4 reported 8+ of 10 declared capabilities (merge, split, OCR, watermark, etc.) as having "zero implementation." In fact SKILL.md itself contains full, working inline Python for merge/split/rotate/ watermark/OCR using pypdf/pdfplumber/reportlab/pytesseract — the skill's actual delivery mechanism for most of its capabilities is documented, copy-executable snippets in the markdown, not pre-packaged .py scripts. TP4 only sees the small scripts/*.py helpers and reports the rest as unimplemented.
mcp-builder — TP4 reported "zero server-creation logic" for the skill's core purpose, counting only two small executable .py files. The skill's actual server-building example code lives in two markdown reference guides (970 and 719 lines), which TP4 never inspects.
Both fired as TP4/HIGH-or-MEDIUM findings and survived the LLM semantic layer which only adjudicates the code it's shown — it does not know about code TP4 never gathered in the first place.
Relation to existing issues
Related to, but distinct from, #135 ("Static pattern analyzers fire on markdown documentation and code blocks," closed via #140). #135 covers the 12 regex-based static analyzers (static_runner.py's is_code_example() / _NON_EXECUTABLE_FILE_TYPES) over-firing on markdown — the opposite direction of this issue. TP4 is a separate, LLM-based analyzer (mcp_tool_poisoning.py) with its own code-gathering path, and its problem is under-crediting markdown as "no implementation," not over-flagging it. Confirmed directly against the current main (well downstream of #135/#140) that _TP4_EXECUTABLE_TYPES still excludes markdown — #140 did not touch this code path.
Why this isn't a baseline/suppression problem
Unlike the PE3/EA1 false positives fixed in #415/#417, this can't be addressed with a suppression rule: a global id: TP4 rule would disable the check entirely, including its legitimate catches (a skill that claims a capability it implements nowhere — not even in markdown — is a real finding), and there's no stable message substring to scope a narrower rule to, since the mismatched-capability text differs per skill and per finding.
Proposed direction (not a full fix — wanted maintainer input on approach)
Extend the code-gathering step to also extract fenced code blocks from markdown-typed components (by declared language, e.g. ```python) and feed them into the same batching/token-budget pipeline TP4 already uses for .py/.js/etc. files, rather than gating on component type alone. Open questions worth maintainer input before implementation:
How to distinguish a fenced block meant to be copy-executed (this skill's actual delivery mechanism) from an illustrative/incomplete example not meant to run standalone — misclassifying the latter as "actual behavior" could introduce new false positives in the other direction.
Token-budget impact: markdown reference docs can be large (970+ lines in the mcp-builder case above); including their code fences changes TP4's cost/latency profile, which may need its own limits distinct from TP4_MAX_FILE_CODE_BYTES.
Happy to help scope/implement once there's agreement on direction — this one felt like it needed a design decision from the maintainers rather than a drop-in regex fix like #415/#417.
Reported by
Benedict Kwok, ZTAI Security Advisors LLC (benedictkwok@ztai.ai) — found while independently evaluating SkillSpector against several official anthropics/skills skills.