Skip to content

docs: require a scenario for every detection rule #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sepehr-safari merged 1 commit into from
Jul 30, 2026
Merged

docs: require a scenario for every detection rule #148

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,13 @@ trace with expected failure outcomes and optional assertions.

Detection rules identify failure patterns in traces.

**Every detection rule ships with a scenario that exercises it, added in the same
pull request as the rule.** A rule without a scenario has no regression guard, so
nothing in the corpus catches it breaking. This is the invariant that keeps the
scenario corpus honest as rules are added, and it means the scenario count in the
READMEs tracks real coverage rather than volume. See
[How to Add a Scenario](#how-to-add-a-scenario).

### How to Add a Detection Rule

1. Add the failure code to `FailureCode` in `packages/toolkit/src/core/types.ts`.
Expand Down Expand Up @@ -290,6 +297,11 @@ Detection rules identify failure patterns in traces.
6. **Audit all existing scenarios** — new rules may trigger on existing fixtures.
Fix false positives or add the new code to `expectedFailures`.

7. **Add a scenario that exercises the new rule**, in this same pull request,
following [How to Add a Scenario](#how-to-add-a-scenario). This is the positive side of
step 6: step 6 confirms the rule stays quiet where it should, and this confirms
it fires where it should. A rule shipped without one is incomplete.

## Contributing Good-First-Issues

Good-first-issues are labeled with `good-first-issue` and are designed for
Expand Down
Loading