What happens
scan() returns all matches, not the first, and the manifest has overlapping patterns. Measured on master after #126:
bare DashScope token 2 matches: dashscope_api_token, generic_provider_api_key
DashScope via env assign 4 matches: agent_env_override,
chinese_provider_api_key_assignment,
dashscope_api_token,
generic_provider_api_key
bare OpenAI-shaped token 1 match: generic_provider_api_key
OpenAI via env assign 1 match: generic_provider_api_key
One DashScope key in one command yields four critical findings.
This is not #126's fault
Three of those four overlapped before that PR. The DashScope token rule is sk- plus 32 hex; the new generic rule is sk- plus 20-or-more alphanumeric. Hex is a subset of alphanumeric and 32 is more than 20, so the fourth was added to an existing pile rather than creating one.
Every match is correct. None is a false positive. The problem is that a SOC receives four criticals for one secret, and the whole argument of this project is that criticals stop being read when they arrive in bulk. It is the alert-fatigue failure in a different coat from the one 0.7.0 fixed.
Reproduce
from agentmetry.core.audit.dlp.scanner import scan
key = "sk-" + "0123456789abcdef0123456789abcdef"
v = scan("run_command", {"command": f"export DASHSCOPE_API_KEY={key}"}, mode="log")
print(sorted({m.rule_id for m in v.matches}))
Options, none decided
- Precedence in the manifest. A rule declares what it supersedes, and the scanner drops superseded matches when a more specific one hits. Keeps all rules independently testable and needs a schema field.
- Negative lookaheads. Cheap and local:
(?i)\bsk-(?!ant-)(?![a-f0-9]{32}\b)[a-zA-Z0-9]{20,}\b. Does not generalise, and every new rule has to know about every older one.
- Deduplicate on the matched span. Two rules hitting the same characters collapse to the highest-severity or most-specific one. Fixes the token overlap and correctly leaves the env-var rules alone, since those match a different span for a different reason.
- Accept it, and dedupe in the SIEM. Defensible, and it pushes our noise onto the customer.
Option 3 looks closest to right, because agent_env_override firing alongside a token rule is genuinely two findings about two different things, while two token rules on the same 35 characters is one finding twice.
Not urgent
DLP defaults to log mode. Nothing pages on this today.
Source
Found reviewing #126, whose author had reasoned about ordering and reasonably assumed first-match wins. The scanner docstring says otherwise, which is worth making true or making obvious.
What happens
scan()returns all matches, not the first, and the manifest has overlapping patterns. Measured on master after #126:One DashScope key in one command yields four critical findings.
This is not #126's fault
Three of those four overlapped before that PR. The DashScope token rule is
sk-plus 32 hex; the new generic rule issk-plus 20-or-more alphanumeric. Hex is a subset of alphanumeric and 32 is more than 20, so the fourth was added to an existing pile rather than creating one.Every match is correct. None is a false positive. The problem is that a SOC receives four criticals for one secret, and the whole argument of this project is that criticals stop being read when they arrive in bulk. It is the alert-fatigue failure in a different coat from the one 0.7.0 fixed.
Reproduce
Options, none decided
(?i)\bsk-(?!ant-)(?![a-f0-9]{32}\b)[a-zA-Z0-9]{20,}\b. Does not generalise, and every new rule has to know about every older one.Option 3 looks closest to right, because
agent_env_overridefiring alongside a token rule is genuinely two findings about two different things, while two token rules on the same 35 characters is one finding twice.Not urgent
DLP defaults to
logmode. Nothing pages on this today.Source
Found reviewing #126, whose author had reasoned about ordering and reasonably assumed first-match wins. The scanner docstring says otherwise, which is worth making true or making obvious.