Skip to content

dlp: one secret produces up to four critical findings #130

Description

@blitzcrieg1

What happens

scan() returns all matches, not the first, and the manifest has overlapping patterns. Measured on master after #126:

bare DashScope token       2 matches: dashscope_api_token, generic_provider_api_key
DashScope via env assign   4 matches: agent_env_override,
                                      chinese_provider_api_key_assignment,
                                      dashscope_api_token,
                                      generic_provider_api_key
bare OpenAI-shaped token   1 match:   generic_provider_api_key
OpenAI via env assign      1 match:   generic_provider_api_key

One DashScope key in one command yields four critical findings.

This is not #126's fault

Three of those four overlapped before that PR. The DashScope token rule is sk- plus 32 hex; the new generic rule is sk- plus 20-or-more alphanumeric. Hex is a subset of alphanumeric and 32 is more than 20, so the fourth was added to an existing pile rather than creating one.

Every match is correct. None is a false positive. The problem is that a SOC receives four criticals for one secret, and the whole argument of this project is that criticals stop being read when they arrive in bulk. It is the alert-fatigue failure in a different coat from the one 0.7.0 fixed.

Reproduce

from agentmetry.core.audit.dlp.scanner import scan
key = "sk-" + "0123456789abcdef0123456789abcdef"
v = scan("run_command", {"command": f"export DASHSCOPE_API_KEY={key}"}, mode="log")
print(sorted({m.rule_id for m in v.matches}))

Options, none decided

  1. Precedence in the manifest. A rule declares what it supersedes, and the scanner drops superseded matches when a more specific one hits. Keeps all rules independently testable and needs a schema field.
  2. Negative lookaheads. Cheap and local: (?i)\bsk-(?!ant-)(?![a-f0-9]{32}\b)[a-zA-Z0-9]{20,}\b. Does not generalise, and every new rule has to know about every older one.
  3. Deduplicate on the matched span. Two rules hitting the same characters collapse to the highest-severity or most-specific one. Fixes the token overlap and correctly leaves the env-var rules alone, since those match a different span for a different reason.
  4. Accept it, and dedupe in the SIEM. Defensible, and it pushes our noise onto the customer.

Option 3 looks closest to right, because agent_env_override firing alongside a token rule is genuinely two findings about two different things, while two token rules on the same 35 characters is one finding twice.

Not urgent

DLP defaults to log mode. Nothing pages on this today.

Source

Found reviewing #126, whose author had reasoned about ordering and reasonably assumed first-match wins. The scanner docstring says otherwise, which is worth making true or making obvious.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions