Backport missing custom-documentation fields to 8.19#754
Open
nicholasberlin wants to merge 9 commits into
Open
Backport missing custom-documentation fields to 8.19#754nicholasberlin wants to merge 9 commits into
nicholasberlin wants to merge 9 commits into
Conversation
* add new security event metrics to custom documentation * add generated files (cherry picked from commit f74d96a)
This adds sources info to process events in the metrics document.
```
"process_events": {
"sent_bytes": 1478944,
"sent_count": 732,
"sources": [
{
"sent_bytes": 0,
"sent_count": 0,
"source": "shmget",
"suppressed_bytes": 0,
"suppressed_count": 1
}
],
"suppressed_bytes": 0,
"suppressed_count": 7
}
```
This change allows for data based diag->prod decisions for new Linux
process events: load_module, ptrace, and shmget
(cherry picked from commit fe393fa)
(cherry picked from commit 7d45502)
(cherry picked from commit e745043)
Contributor
Author
|
This is a work in progress. Please ignore for now. |
* add missing field * add generated file (cherry picked from commit 89c1a22)
…#740) Some rules emit populated threat.technique.subtechnique.{id,name,reference} fields. The Linux and macOS malicious_behavior_alert custom_documentation only listed the parent threat.technique.subtechnique path, causing the EAF custom-documentation validator to fail with undocumented keys. Mirrors the shape already present in the Windows alert (PR #614 era). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> (cherry picked from commit 332151d)
Endpoint.metrics.documents_volume.{process,security}_events.sources.*
were added to custom_documentation only (#652, #653), so the fields were
never mapped in the package. The metrics index template sets
dynamic: false, so these fields are currently not indexed on any stack.
Define them in custom_schemas/custom_endpoint.yml mirroring the existing
api_events.sources.* mappings, and regenerate with make.
(cherry picked from commit 322d76a)
Requested in review: sample values for the new
documents_volume.{process,security}_events.sources.* mappings so the
automated mapping tests exercise them. Source values reflect what the
endpoint emits: process event sources are opcode names (load_module),
security event sources are diagnostic_<event id> strings.
(cherry picked from commit 52535bc)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the custom documentation entries the endpoint-dev EAF suite found missing when endpoint-dev #20922 pinned this repo's
8.19branch as the endpoint-package submodule (build: https://buildkite.com/elastic/endpoint-dev/builds/23075).Cherry-picked from
main, in original order:Fixes EAF failures asserting missing custom documentation for:
Generated with Claude Code.
Update: Follow-up build #23099 (after the fixes above landed on this branch) flagged 3 more undocumented fields. Added:
5. 0068ff3 - Add process.Ext.trusted to custom docs for already-running process events (#697)
6. 89c1a22 - Add missing field in the ASM API events custom documentation (#726)
7. 332151d - Fix Linux and macOS behavior alert custom doc for subtechnique fields (#740)
Fixes additional EAF failures for:
Update 2 (mappings, not just docs): review feedback on the endpoint-dev side pointed out that
Endpoint.metrics.documents_volume.{process,security}_events.sources.*had only ever been added tocustom_documentation- the fields were never defined incustom_schemas, so they are missing from the actual package mappings (the metrics index template isdynamic: false, so they are not indexed). This was true onmainas well; fixed there by #762 and cherry-picked here (regeneration verified on this branch).The other fields in this PR did not need mapping changes:
subtechnique.*,security_descriptor,Ext.trusted{,_descendant}, andnetwork.directionwere already mapped on 8.19 (only the documentation lagged), andkernel_process_events.*lives undersystem_impactwhich isenabled: false(intentionally not indexed), matching main.