Skip to content

Add alert closure evidence and tuning handoff gates #2233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Errordog2 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add alert closure evidence and tuning handoff gates #2233

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 88 additions & 2 deletions skills/secops/alert-triage/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ phase: [operate, respond]
frameworks: [MITRE-ATT&CK-v16, NIST-SP-800-61-Rev2]
difficulty: beginner
time_estimate: "10-20min per alert"
version: "1.0.0"
version: "1.0.1"
author: unitoneai
license: MIT
allowed-tools: Read, Grep, Glob
Expand Down Expand Up @@ -116,6 +116,54 @@ Assign a disposition and priority based on collected and correlated data.
| **Benign True Positive (BTP)** | BTP | The alert correctly identified the activity described in the rule, but the activity is authorized, expected, or part of legitimate operations. | Document the legitimate reason. If recurring, request a rule tuning (filter/exclusion). Close alert. |
| **False Positive (FP)** | FP | The alert fired incorrectly -- the underlying activity does not match what the rule intended to detect (rule logic error, data quality issue). | Document the false positive cause. Submit a tuning request to detection engineering. Close alert. |

#### BTP/FP Closure Evidence Gate

Do not close a **Benign True Positive** or **False Positive** alert solely
because the activity "looks expected" or has been seen before. Before closure,
record auditable evidence that supports the disposition and proves the closure
will not hide unresolved risk.

Required closure evidence:

- **Source of truth:** Link to the approved change ticket, maintenance window,
runbook, deployment record, asset inventory entry, HR/business request, or
other authoritative record that explains the activity.
- **Owner confirmation:** Identify the asset owner, service owner, customer
contact, or on-call lead who confirmed the business context. For production,
regulated, privileged, or customer-managed assets, do not close without owner
confirmation unless escalation documents the failed confirmation attempt.
- **Time and scope fit:** Confirm the observed user, host, process, command,
source/destination, and timestamp match the approved window and expected
activity. Emergency or break-glass activity may be legitimate but still
requires post-use review.
- **Temporary exception controls:** If an allowlist, suppression, exception, or
maintenance rule is used, record the expiry date, revalidation owner, and
rollback criteria.
- **Customer approval for MSSP/MDR:** For managed environments, record customer
approval before suppressing or tuning alerts that affect their tenant.

If the evidence is missing, stale, or only based on analyst memory, classify the
alert as **Unresolved / needs escalation** rather than BTP/FP closure.

#### Tuning Handoff Gate for Repeated BTP/FP Patterns

Repeated benign or false-positive alerts are detection-quality work, not just
queue noise. If the same alert pattern recurs above the team's threshold (for
example, repeated count over 7 days, repeated privileged account activity, or
recurring customer-visible alerts), create a detection-engineering handoff before
final closure.

The handoff must include:

- recurrence count and lookback window;
- affected rule, data source, entity pattern, and ATT&CK mapping;
- proposed tuning type such as threshold change, entity allowlist with expiry,
enrichment requirement, rule logic fix, suppression, or runbook update;
- safety constraints that preserve malicious variants;
- tuning ticket/link, owner, due date, and review/expiry date.

If no tuning ticket is created, explain why the pattern should continue to alert.

#### Priority Matrix

Assign a priority level based on the combination of asset criticality, threat severity, and confidence.
Expand Down Expand Up @@ -194,7 +242,7 @@ Produce the triage decision as a structured report:
```markdown
## Alert Triage Report
**Date:** [YYYY-MM-DD HH:MM UTC]
**Skill:** alert-triage v1.0.0
**Skill:** alert-triage v1.0.1
**Frameworks:** MITRE ATT&CK v16, NIST SP 800-61 Rev 2
**Analyst:** [Name or AI-assisted]

Expand Down Expand Up @@ -243,6 +291,26 @@ Produce the triage decision as a structured report:
[If disposition is BTP or FP, describe the recommended rule tuning
to prevent recurrence -- e.g., add filter for specific parent process,
exclude known-good IP range, adjust threshold.]

### Closure Evidence (Required for BTP/FP)
| Evidence Field | Value |
|---|---|
| Source of Truth | [Change ticket / maintenance window / runbook / owner request / N/A] |
| Owner Confirmation | [Name/role, timestamp, or escalation if unavailable] |
| Time and Scope Fit | [Observed activity matches approved user/host/process/window: Yes/No] |
| Temporary Exception Expiry | [Expiry/revalidation date or N/A] |
| Customer Approval (MSSP/MDR) | [Approval reference or N/A] |
| Closure Decision | [Close as BTP/FP / Keep unresolved / Escalate] |

### Detection Tuning Handoff (If Repeated BTP/FP)
| Field | Value |
|---|---|
| Recurrence Count / Window | [N alerts over X days] |
| Pattern | [Rule/entity/process/source/destination pattern] |
| Proposed Tuning | [Threshold / allowlist with expiry / rule logic fix / enrichment / runbook] |
| Safety Constraints | [Malicious variants that must still alert] |
| Tuning Ticket | [Ticket/link or reason not created] |
| Owner / Due Date / Review Date | [Detection owner, target date, expiry/revalidation] |
```

---
Expand Down Expand Up @@ -311,6 +379,14 @@ SIEM-assigned alert severity (Critical/High/Medium/Low) reflects the detection r

Marking an alert as "False Positive" or "Benign" without recording why leads to repeated investigation of the same alert pattern and prevents detection engineering from tuning the rule. Every closed alert should include the specific reason for the disposition, enabling trend analysis and rule improvement.

### Pitfall 3a: Treating Repeated Benign Alerts as Safe to Suppress Without Evidence

A repeated BTP/FP can still hide risk if the owner, approved window, scope, or
temporary exception expiry is missing. Analyst memory is not closure evidence.
For production, privileged, regulated, or customer-managed assets, require
source-of-truth evidence and owner confirmation before closure, and route
recurring patterns to detection engineering.

### Pitfall 4: Failing to Look for Kill Chain Progression

Investigating an alert in isolation without checking for activity before and after the detected event misses multi-stage attacks. An attacker who triggers one alert likely generated detectable activity at other stages of the kill chain. Always check for related events within a +/- 30 minute window on the same host and user, and look for lateral activity on other hosts.
Expand All @@ -333,6 +409,16 @@ This skill processes user-supplied content that may include alert payloads, log

---

## Changelog

- **1.0.1** -- Added BTP/FP closure evidence gates, temporary exception expiry,
owner/customer confirmation requirements, and detection tuning handoff fields
for recurring benign or false-positive alert patterns.
- **1.0.0** -- Initial release. Four-phase alert triage method with ATT&CK
correlation, NIST SP 800-61 escalation guidance, and structured triage output.

---

## 9. References

1. **NIST SP 800-61 Rev 2 -- Computer Security Incident Handling Guide** -- https://csrc.nist.gov/publications/detail/sp/800-61/rev-2/final
Expand Down