Skip to content

CrowdStrike FDR: confirm process-event name coverage and investigate duplicate events / rollover health #20081

Description

@tomsonpl

Context

Investigating a customer-reported analyzer issue (single-node/broken process trees) in
security-team#13301.

The root causes split cleanly into two groups:

  • Five bugs that are entirely ours to fix in the Kibana resolver (blank/"undefined" labels,
    representative-selection on duplicate docs, label fallback, synthesized "ghost" nodes for orphaned
    parents, and root-sentinel handling). Not the subject of this issue — already scoped and in
    progress on our side.
  • Three observations about the FDR data itself that we'd like Integrations' help understanding
    or investigating, listed below. None of these are field-mapping problems — we've already confirmed
    the integration correctly maps everything CrowdStrike sends (ContextProcessIdprocess.entity_id,
    ContextBaseFileNameprocess.name, etc.). These are questions about the upstream data shape and
    the health of the backing data stream.
    We are not asking for a pipeline/mapping change. We're asking for confirmation on No.1, and
    investigation on No.2 and No.3, since 2 and 3 look like they may be within Integrations' or the FDR
    package's control (rollover policy, ingestion behavior) rather than a CrowdStrike-side limitation.

1. Confirm the process-event-type / name-coverage breakdown is expected (low priority — documentation ask)

Measured on one FDR backing index (.ds-logs-crowdstrike.fdr-default-2026.06.11-000010, live dev
stack, 2026-06-23):

  • Total event.category: process docs: 10,996,710
  • With process.name: 6,073,624 (55%)
  • Without process.name: 4,997,705 (45%)
    Cross-checking event.action + event.id on a sample entity confirms the unnamed docs are
    ProcessAncestryInformation and EndOfProcess event types — distinct FDR event types from
    ProcessRollup2 (process start), which is the only one carrying process.name. This matches
    what Chema described informally in Slack (2026-06-16): FDR emits multiple event types per process
    lifecycle, and only ProcessRollup2 carries the name.
    Ask: can someone confirm this ~45%-nameless split is the expected, general shape of FDR telemetry
    (not specific to this environment), so we can document it? We're already handling it on the Kibana
    side by preferring ProcessRollup2 when representing a node — this is a confirm-only ask, not a
    blocker.

2. Investigate duplicate events across backing indices

Checked event.action + event.id for a sample entity (869661616536) across 3 backing indices.
The same event.id appears more than once, in different backing indices, under different _ids —
i.e., the same underlying event is stored more than once in the data stream. On top of that, some
copies of what should be the same document disagree with each other: one copy has
process.name + process.parent.entity_id populated, another copy of the same event has neither.
Ask:

  • Is _id generation for FDR documents deterministic, or could retries/re-ingestion produce a new
    _id for an event that was already indexed?
  • Is this duplication specific to this test/POC environment's ingestion setup, or a general
    characteristic of the FDR integration?
    This doesn't directly cause the blank-label bug we're fixing (a named copy of the event still exists
    elsewhere), but it inflates the dataset and may be related to No.3 below.

3. Investigate index rollover / ILM configuration health

_cat/indices on logs-crowdstrike.fdr* in the same environment shows:

  • 37 backing indices, ~6 billion documents total
  • Recent indices at 3 primary + 1 replica = 6 shards each, 90M–300M docs per index
  • Pathological rollover: 2026-06-11 rolled over 3× in a single day (-000010/11/12);
    2026-06-19 rolled over 4× (-000023/24/26/27)
  • Wildly uneven index sizes: e.g. one day's backing index has ~1.1M docs, the next day's has
    ~265M docs
    This shard/index topology is very likely the direct cause of query timeouts we're seeing on the
    Kibana side — a 3-entity filter against a single recent backing index returns in milliseconds, but
    the same filter against logs-crowdstrike.fdr* over a 15-day window (~90 shards) returns a 502
    (gateway timeout), and a slightly wider aggregation trips the circuit breaker (parent breaker at
    6.7gb against a 6.6gb limit).
    Ask: could someone review whether the rollover policy (max_primary_shard_size/max_docs
    thresholds, or whatever governs this data stream's ILM) is tuned appropriately for this ingest
    volume? The multi-rollover-per-day pattern and the size unevenness both suggest it may not be. We'd
    also like to know if this is expected for high-volume FDR ingestion generally, or specific to this
    test environment.

Reproduction

  • load-crowdstrike-samples.sh (in the elastic/kibana repo) loads sample
    FDR/alert docs for local repro of the labeling bugs (No.1 area, not directly relevant to #No.2/No.3 here,
    which were observed on a live multi-billion-doc environment rather than the small sample set).
  • Raw queries used to produce the numbers above are recorded in
    13301-tree-trace-pup.md ("Verification queries used" section)

load-crowdstrike-samples.sh

13301-tree-trace-pup.md

Not blocking

None of No1. - No.3 block the Kibana-side fixes for the analyzer bugs, which are proceeding independently.
This issue is to get eyes on the data-shape/ingestion-health questions in parallel.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions