From b246c4a4d239cd3bed050df82b8870c35a15760b Mon Sep 17 00:00:00 2001 From: eric-forte-elastic Date: Wed, 24 Jun 2026 14:13:52 -0400 Subject: [PATCH] Initial documentation --- ...s_and_data_from_elastic_security_to_vcs.md | 2 + docs/dac_quick_start_guide.md | 78 ++++++++++++++++++- docs/faq.md | 14 +++- docs/internals_of_the_detection_rules_repo.md | 11 +++ 4 files changed, 103 insertions(+), 2 deletions(-) diff --git a/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md b/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md index 42072ca..741ee54 100644 --- a/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md +++ b/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md @@ -126,6 +126,8 @@ jobs: Here **`kibana export-rules --directory` (`-d`)** is the **output** directory (unlike **`export-rules-from-repo`**, where **`-d`** loads **input** rules). Add **`--save-as-yaml` / `-sy`** to write YAML instead of TOML. For repo → NDJSON or per-rule YAML, see **`export-rules-from-repo`** in the [Quick Start](dac_quick_start_guide.md#syncing-with-kibana). +When syncing more than one rule set from the same Kibana space, use `--custom-rules-only` / `-cro` or `--export-query` / `-eq` to keep each export intentionally scoped. For example, custom rules can be exported with `-cro`, customized Elastic prebuilt rules can be filtered with `alert.attributes.params.ruleSource.isCustomized: true and alert.attributes.params.immutable: true`, and unmodified Elastic prebuilt rules can be filtered with `alert.attributes.params.ruleSource.isCustomized: false and alert.attributes.params.immutable: true`. Avoid using `enabled` to classify rule ownership because it only represents whether the rule is active in Kibana. For more detail and examples, see [Managing custom and prebuilt rules together](dac_quick_start_guide.md#managing-custom-and-prebuilt-rules-together). + ```bash # Export Rules from Elastic Security python -m detection_rules kibana export-rules -d test-export-rules --skip-errors diff --git a/docs/dac_quick_start_guide.md b/docs/dac_quick_start_guide.md index cd24641..f9b95bd 100644 --- a/docs/dac_quick_start_guide.md +++ b/docs/dac_quick_start_guide.md @@ -343,7 +343,83 @@ Optionally add `-d ` or `-f ` to limit to a directory or single --- -## 10. Version locking (optional) +## 10. Managing multiple rule sets + +Many DaC workflows eventually need to manage more than one collection of rules at the same time: team-owned custom rules, environment-specific rules, modified Elastic prebuilt rules, and unmodified Elastic prebuilt rules that should still be backed up or promoted between spaces. Treat each collection as a rule set with an explicit source of truth, then decide whether the sets should share one deployment config or move independently. + +**Recommended local patterns:** + +- **One deployment unit:** Use one custom rules directory and list multiple rule directories in its `_config.yaml`. This is best when the rule sets share the same schema map, test config, versioning strategy, exceptions, and release process. + ```yaml + rule_dirs: + - rules + - rules_team_a + - rules_team_b + ``` +- **Independent deployment units:** Use a separate custom rules directory for each rule set when different teams, spaces, schemas, tests, or release cadences need to stay isolated. Set `CUSTOM_RULES_DIR` to the rule set you are validating or syncing so the CLI loads the matching `_config.yaml`, `stack-schema-map.yaml`, `test_config.yaml`, and optional version files. + +For repository-to-artifact workflows, `export-rules-from-repo` can load more than one input directory by repeating `--directory` / `-d`. This is useful when you want one NDJSON handoff from several compatible local rule sets: + +```bash +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules export-rules-from-repo \ + --directory dac_custom_rules_dir/rules \ + --directory dac_custom_rules_dir/rules_team_a \ + --outfile combined-rules.ndjson \ + --include-exceptions \ + --include-action-connectors +``` + +If the rule sets have different configs, validate and export each one separately, then combine them at the Kibana import, CI/CD, or change-management layer instead of forcing them through a single `_config.yaml`. + +### Managing custom and prebuilt rules together + +`kibana export-rules` can export custom rules, Elastic prebuilt rules, and customized Elastic prebuilt rules from the same Kibana space. By default, it exports all matching rules. Use `--custom-rules-only` / `-cro` for custom rules, or `--export-query` / `-eq` when you need more precise Kibana-side filtering. + +Do not use `enabled` as the primary way to classify rule ownership. `enabled` only says whether a rule is active in Kibana. Some Elastic-authored rules are enabled by default for structural reasons, such as surfacing alerts from Elastic Defend or Elastic Cloud Defend, and a disabled rule can still be a custom or customized prebuilt rule. For ownership and customization, prefer `immutable` and `ruleSource` fields exposed in exported rule objects and queryable through the Kibana rule filter syntax. + +Useful categories: + +- **Custom rules:** use `--custom-rules-only` / `-cro`, or filter for internal/non-immutable rules. +- **Customized prebuilt rules:** `immutable: true` and `ruleSource.isCustomized: true`. +- **Unmodified prebuilt rules:** `immutable: true` and `ruleSource.isCustomized: false`. + +Examples: + +```bash +# Custom rules only +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules kibana export-rules \ + --directory exports/custom-rules \ + --custom-rules-only \ + --skip-errors +``` + +```bash +# Customized Elastic prebuilt rules +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules kibana export-rules \ + --directory exports/customized-prebuilt-rules \ + --export-query 'alert.attributes.params.ruleSource.isCustomized: true and alert.attributes.params.immutable: true' \ + --skip-errors +``` + +```bash +# Unmodified Elastic prebuilt rules +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules kibana export-rules \ + --directory exports/unmodified-prebuilt-rules \ + --export-query 'alert.attributes.params.ruleSource.isCustomized: false and alert.attributes.params.immutable: true' \ + --skip-errors +``` + +Kibana's rule filter syntax uses `alert.attributes.*` field paths. Some exported rule JSON fields use different casing or nesting than the filter path; for example, exported rule metadata may appear as `rule_source.is_customized`, while the filter path is `alert.attributes.params.ruleSource.isCustomized`. See the Kibana [Find rules API](https://www.elastic.co/docs/api/doc/kibana/operation/operation-findrules) for the `filter` parameter used by the Detection Engine rule search endpoint. + +When you need to audit what changed in a customized prebuilt rule, inspect the exported `rule_source.customized_fields` list. For example, a customized prebuilt rule may show `query` and `index` in `customized_fields`, which lets you distinguish a rule whose detection logic changed from one that only points to a different index pattern. + +--- + +## 11. Version locking (optional) Version locking can help when you sync in both directions (repo ↔ Kibana) or use overwrite workflows with multiple sources of truth at a time for rules: the lock file records which version of each rule is “current,” so the CLI can avoid overwriting newer changes or detect conflicts. If you only have a single source of truth at one time, you typically do not need it. diff --git a/docs/faq.md b/docs/faq.md index 9c530e5..7f64997 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -179,4 +179,16 @@ Running `view-rule` on this rule should look similar to the following. When the View Rule Custom Schema -For more information on how you can also automatically generate the custom schemas please see [Option 3: Custom Schema Validation](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-3-custom-schema-validation) \ No newline at end of file +For more information on how you can also automatically generate the custom schemas please see [Option 3: Custom Schema Validation](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-3-custom-schema-validation) + +#### **Q16**: How do I distinguish custom rules, customized prebuilt rules, and unmodified prebuilt rules when exporting from Kibana? + +**A16**: Use rule source metadata rather than the rule's `enabled` state. `enabled` only indicates whether the rule is active in Kibana; it does not reliably identify ownership or customization. + +Useful export categories are: + +- Custom rules: use `python -m detection_rules kibana export-rules --custom-rules-only`, or filter for internal/non-immutable rules. +- Customized Elastic prebuilt rules: filter with `alert.attributes.params.ruleSource.isCustomized: true and alert.attributes.params.immutable: true`. +- Unmodified Elastic prebuilt rules: filter with `alert.attributes.params.ruleSource.isCustomized: false and alert.attributes.params.immutable: true`. + +When reviewing exported customized prebuilt rules, inspect `rule_source.customized_fields` to see what changed. For example, `query` means detection logic was customized, while `index` means the rule's index patterns were customized. See [Managing custom and prebuilt rules together](dac_quick_start_guide.md#managing-custom-and-prebuilt-rules-together) for full command examples. \ No newline at end of file diff --git a/docs/internals_of_the_detection_rules_repo.md b/docs/internals_of_the_detection_rules_repo.md index 7b2296e..6fe361d 100644 --- a/docs/internals_of_the_detection_rules_repo.md +++ b/docs/internals_of_the_detection_rules_repo.md @@ -295,6 +295,17 @@ testing: config: etc/test_config.yaml ``` +For multiple local rule sets, choose whether they should be loaded by one custom rules config or by separate configs. If the rule sets share schema validation, tests, exceptions/actions directories, and versioning strategy, list multiple directories under `rule_dirs`: + +```yaml +rule_dirs: +- rules +- rules_team_a +- rules_team_b +``` + +If the rule sets belong to different teams, Kibana spaces, stack versions, schema maps, or release cadences, use separate custom rules directories and set `CUSTOM_RULES_DIR` to the directory being validated or synced. This keeps each rule set's `_config.yaml`, `stack-schema-map.yaml`, `test_config.yaml`, and optional version files independent. For Kibana export examples that separate custom rules from customized and unmodified Elastic prebuilt rules, see [Managing custom and prebuilt rules together](dac_quick_start_guide.md#managing-custom-and-prebuilt-rules-together). + ```bash (detection-rules-build) ➜ detection-rules git:(main) ✗ ❯ python -m detection_rules custom-rules setup-config dac_custom_rules_dir Loaded config file: /home/user/code/detection-rules/.detection-rules-cfg.json