Context
Investigating a customer-reported analyzer issue (single-node/broken process trees) in
security-team#13301.
The root causes split cleanly into two groups:
- Five bugs that are entirely ours to fix in the Kibana resolver (blank/
"undefined" labels,
representative-selection on duplicate docs, label fallback, synthesized "ghost" nodes for orphaned
parents, and root-sentinel handling). Not the subject of this issue — already scoped and in
progress on our side.
- Three observations about the FDR data itself that we'd like Integrations' help understanding
or investigating, listed below. None of these are field-mapping problems — we've already confirmed
the integration correctly maps everything CrowdStrike sends (ContextProcessId → process.entity_id,
ContextBaseFileName → process.name, etc.). These are questions about the upstream data shape and
the health of the backing data stream.
We are not asking for a pipeline/mapping change. We're asking for confirmation on No.1, and
investigation on No.2 and No.3, since 2 and 3 look like they may be within Integrations' or the FDR
package's control (rollover policy, ingestion behavior) rather than a CrowdStrike-side limitation.
1. Confirm the process-event-type / name-coverage breakdown is expected (low priority — documentation ask)
Measured on one FDR backing index (.ds-logs-crowdstrike.fdr-default-2026.06.11-000010, live dev
stack, 2026-06-23):
- Total
event.category: process docs: 10,996,710
- With
process.name: 6,073,624 (55%)
- Without
process.name: 4,997,705 (45%)
Cross-checking event.action + event.id on a sample entity confirms the unnamed docs are
ProcessAncestryInformation and EndOfProcess event types — distinct FDR event types from
ProcessRollup2 (process start), which is the only one carrying process.name. This matches
what Chema described informally in Slack (2026-06-16): FDR emits multiple event types per process
lifecycle, and only ProcessRollup2 carries the name.
Ask: can someone confirm this ~45%-nameless split is the expected, general shape of FDR telemetry
(not specific to this environment), so we can document it? We're already handling it on the Kibana
side by preferring ProcessRollup2 when representing a node — this is a confirm-only ask, not a
blocker.
2. Investigate duplicate events across backing indices
Checked event.action + event.id for a sample entity (869661616536) across 3 backing indices.
The same event.id appears more than once, in different backing indices, under different _ids —
i.e., the same underlying event is stored more than once in the data stream. On top of that, some
copies of what should be the same document disagree with each other: one copy has
process.name + process.parent.entity_id populated, another copy of the same event has neither.
Ask:
- Is
_id generation for FDR documents deterministic, or could retries/re-ingestion produce a new
_id for an event that was already indexed?
- Is this duplication specific to this test/POC environment's ingestion setup, or a general
characteristic of the FDR integration?
This doesn't directly cause the blank-label bug we're fixing (a named copy of the event still exists
elsewhere), but it inflates the dataset and may be related to No.3 below.
3. Investigate index rollover / ILM configuration health
_cat/indices on logs-crowdstrike.fdr* in the same environment shows:
- 37 backing indices, ~6 billion documents total
- Recent indices at 3 primary + 1 replica = 6 shards each, 90M–300M docs per index
- Pathological rollover: 2026-06-11 rolled over 3× in a single day (
-000010/11/12);
2026-06-19 rolled over 4× (-000023/24/26/27)
- Wildly uneven index sizes: e.g. one day's backing index has ~1.1M docs, the next day's has
~265M docs
This shard/index topology is very likely the direct cause of query timeouts we're seeing on the
Kibana side — a 3-entity filter against a single recent backing index returns in milliseconds, but
the same filter against logs-crowdstrike.fdr* over a 15-day window (~90 shards) returns a 502
(gateway timeout), and a slightly wider aggregation trips the circuit breaker (parent breaker at
6.7gb against a 6.6gb limit).
Ask: could someone review whether the rollover policy (max_primary_shard_size/max_docs
thresholds, or whatever governs this data stream's ILM) is tuned appropriately for this ingest
volume? The multi-rollover-per-day pattern and the size unevenness both suggest it may not be. We'd
also like to know if this is expected for high-volume FDR ingestion generally, or specific to this
test environment.
Reproduction
load-crowdstrike-samples.sh (in the elastic/kibana repo) loads sample
FDR/alert docs for local repro of the labeling bugs (No.1 area, not directly relevant to #No.2/No.3 here,
which were observed on a live multi-billion-doc environment rather than the small sample set).
- Raw queries used to produce the numbers above are recorded in
13301-tree-trace-pup.md ("Verification queries used" section)
load-crowdstrike-samples.sh
13301-tree-trace-pup.md
Not blocking
None of No1. - No.3 block the Kibana-side fixes for the analyzer bugs, which are proceeding independently.
This issue is to get eyes on the data-shape/ingestion-health questions in parallel.
Context
Investigating a customer-reported analyzer issue (single-node/broken process trees) in
security-team#13301.
The root causes split cleanly into two groups:
"undefined"labels,representative-selection on duplicate docs, label fallback, synthesized "ghost" nodes for orphaned
parents, and root-sentinel handling). Not the subject of this issue — already scoped and in
progress on our side.
or investigating, listed below. None of these are field-mapping problems — we've already confirmed
the integration correctly maps everything CrowdStrike sends (
ContextProcessId→process.entity_id,ContextBaseFileName→process.name, etc.). These are questions about the upstream data shape andthe health of the backing data stream.
We are not asking for a pipeline/mapping change. We're asking for confirmation on No.1, and
investigation on No.2 and No.3, since 2 and 3 look like they may be within Integrations' or the FDR
package's control (rollover policy, ingestion behavior) rather than a CrowdStrike-side limitation.
1. Confirm the process-event-type / name-coverage breakdown is expected (low priority — documentation ask)
Measured on one FDR backing index (
.ds-logs-crowdstrike.fdr-default-2026.06.11-000010, live devstack, 2026-06-23):
event.category: processdocs: 10,996,710process.name: 6,073,624 (55%)process.name: 4,997,705 (45%)Cross-checking
event.action+event.idon a sample entity confirms the unnamed docs areProcessAncestryInformationandEndOfProcessevent types — distinct FDR event types fromProcessRollup2(process start), which is the only one carryingprocess.name. This matcheswhat Chema described informally in Slack (2026-06-16): FDR emits multiple event types per process
lifecycle, and only
ProcessRollup2carries the name.Ask: can someone confirm this ~45%-nameless split is the expected, general shape of FDR telemetry
(not specific to this environment), so we can document it? We're already handling it on the Kibana
side by preferring
ProcessRollup2when representing a node — this is a confirm-only ask, not ablocker.
2. Investigate duplicate events across backing indices
Checked
event.action+event.idfor a sample entity (869661616536) across 3 backing indices.The same
event.idappears more than once, in different backing indices, under different_ids —i.e., the same underlying event is stored more than once in the data stream. On top of that, some
copies of what should be the same document disagree with each other: one copy has
process.name+process.parent.entity_idpopulated, another copy of the same event has neither.Ask:
_idgeneration for FDR documents deterministic, or could retries/re-ingestion produce a new_idfor an event that was already indexed?characteristic of the FDR integration?
This doesn't directly cause the blank-label bug we're fixing (a named copy of the event still exists
elsewhere), but it inflates the dataset and may be related to No.3 below.
3. Investigate index rollover / ILM configuration health
_cat/indicesonlogs-crowdstrike.fdr*in the same environment shows:-000010/11/12);2026-06-19 rolled over 4× (
-000023/24/26/27)~265M docs
This shard/index topology is very likely the direct cause of query timeouts we're seeing on the
Kibana side — a 3-entity filter against a single recent backing index returns in milliseconds, but
the same filter against
logs-crowdstrike.fdr*over a 15-day window (~90 shards) returns a 502(gateway timeout), and a slightly wider aggregation trips the circuit breaker (parent breaker at
6.7gb against a 6.6gb limit).
Ask: could someone review whether the rollover policy (
max_primary_shard_size/max_docsthresholds, or whatever governs this data stream's ILM) is tuned appropriately for this ingest
volume? The multi-rollover-per-day pattern and the size unevenness both suggest it may not be. We'd
also like to know if this is expected for high-volume FDR ingestion generally, or specific to this
test environment.
Reproduction
load-crowdstrike-samples.sh(in theelastic/kibanarepo) loads sampleFDR/alert docs for local repro of the labeling bugs (No.1 area, not directly relevant to #No.2/No.3 here,
which were observed on a live multi-billion-doc environment rather than the small sample set).
13301-tree-trace-pup.md("Verification queries used" section)load-crowdstrike-samples.sh
13301-tree-trace-pup.md
Not blocking
None of No1. - No.3 block the Kibana-side fixes for the analyzer bugs, which are proceeding independently.
This issue is to get eyes on the data-shape/ingestion-health questions in parallel.