CrowdStrike FDR: `enable_deduplication` fingerprint drops distinct events and fails to dedup re-ingested data (CSPM/IOA)

[kcreddy]

Summary

When enable_deduplication is enabled on the CrowdStrike FDR data stream, the fingerprint processor that generates _id produces wrong results for Cloud Security (CSPM) events:

1. Distinct events are silently dropped. Multiple different findings collapse to a single _id and overwrite each other.
2. Re-ingesting the same data creates duplicates instead of deduping — i.e. the feature fails at its one job.

Root cause is the position of the fingerprint processor in packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml: it runs before @timestamp (and other fields) reach their final values, and its field set does not uniquely identify these events.

Affected events

Events whose authoritative timestamp is set by a sub-pipeline that runs after the fingerprint:

• cspm_iom (IOM and CloudSecurityIOMEvaluation events) — sets @timestamp from crowdstrike.created.
• cspm_ioa (cloud IOA events) — sets @timestamp from crowdstrike.event_created.

Normal endpoint events (process/network/dns/file) are not affected — their @timestamp is set from UTCTimestamp/ContextTimeStamp before the fingerprint and is not overwritten later.

Root cause detail

The fingerprint at default.yml:540-551 hashes:

fields:
  - '@timestamp'
  - crowdstrike.id
  - crowdstrike.aid
  - crowdstrike.cid
  - _temp.type

For CSPM/IOA events at that point in the pipeline:

• crowdstrike.id and crowdstrike.aid are empty, crowdstrike.cid and _temp.type (data) are constant.
• @timestamp has not been set to the event's real time yet. The early @timestamp block falls back to _ingest.timestamp (processing time) because the source time field for these events (crowdstrike.created / crowdstrike.event_created) is only consumed later, inside cspm_iom / cspm_ioa.

Consequences:

• The _id is effectively keyed on processing time, which is non-deterministic. A batch of N distinct findings from one evaluation collapses to however many _ingest.timestamp values the batches happened to span (often 1–2), dropping the rest.
• Because processing time changes on every run, re-ingesting the same object yields a new _id → duplicates. So dedup neither preserves distinct findings nor prevents true duplicates.

Reproduction

1. Install CrowdStrike, enable Enable Data Deduplication on the FDR data stream.
2. Ingest an FDR object containing a CloudSecurityIOMEvaluation batch (one evaluation run yields several distinct findings for the same resource, sharing @timestamp/cid, with empty id/aid).
3. Observe fewer documents than distinct findings (e.g. 5 findings → 1–2 indexed). The survivors are timing-dependent.
4. Re-ingest the same object → duplicates appear instead of being deduped.

Proposed fix

1. Move the fingerprint processor to the end of default.yml, after all pipeline sub-pipeline calls and field mappings, and before the remove of _temp (currently line 2794). This ensures @timestamp and the other inputs are final.
2. Update the field set (some inputs are transformed before that point — crowdstrike.aid is renamed to host.id). Use the already-assembled event.id plus a content discriminator:

- fingerprint:
    description: When deduplication is enabled, fingerprint a set of fields to prevent the same event from being indexed more than once.
    tag: fingerprint_crowdstrike_fdr_0e5ffd3f
    if: ctx._conf?.enable_deduplication == true
    fields:
      - '@timestamp'
      - event.id        # encodes crowdstrike.id|aid|cid; persists to end
      - _temp.type
      - message         # discriminates distinct CSPM findings that share @timestamp/cid
    target_field: _id
    ignore_missing: true

Note: message is renamed to event.original early in the pipeline and only re-populated by the sub-pipelines, so it is meaningful only once the fingerprint is moved to the end. For a more durable key than free-text message, a stable finding identity (e.g. rule.id + resource id) could be used for CSPM events; message is the minimal fix.

Acceptance criteria

• With enable_deduplication on, all distinct CloudSecurityIOMEvaluation/IOM/IOA findings from one object are indexed (no silent loss).
• Re-ingesting the same object produces no duplicates.
• Normal endpoint event dedup behavior is unchanged.
• Pipeline tests cover a multi-finding CSPM object (distinct docs kept) and a re-ingest (deduped).

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)