o365: fix document_parsing_exception on Sender field

[chemamartinez]

Proposed commit message

o365: fix document_parsing_exception on Sender field

o365.audit.Sender is polymorphic across workloads: Exchange records
report it as a plain email address, while Microsoft Teams
impersonation-detection records report it as a structured identity
object ({UPN, MRI, DisplayName, OrganizationId, UserObjectId}).

The 3.10.3 fix addressed the Teams shape by mapping Sender as
`type: object` with `subobjects: false`. That resolved ingestion for
Teams events, but Elasticsearch only supports a single static type
per field path across a data stream. Since the Exchange string shape
is the dominant case across almost every other workload, locking the
field to `object` caused it to reject the plain-string form instead,
reproducing the same class of document_parsing_exception in the
opposite direction.

This change reverts o365.audit.Sender to `keyword` and adds an ingest
pipeline processor that renames Sender to a new SenderEntity field
whenever it arrives as a Map, before the document reaches the
mapping. This mirrors the existing pattern already used for
Parameters, ModifiedProperties, and ExtendedProperties, where a field
that can arrive as either a scalar or an object is normalized to one
consistent shape ahead of indexing rather than relying on the mapping
to absorb both shapes. Sender stays a keyword for every event that
reports it as a string, and the structured Teams identity object is
preserved in full under SenderEntity.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[chemamartinez]

/test

[github-actions]

TL;DR

The failed Buildkite step did not reach the o365 package checks. CI failed while installing yq for the Buildkite scripts unit tests: the downloaded yq.tar.gz was not a tar archive, so tar exited with status 2.

Remediation

• Retry the Buildkite build; this looks like a transient external download/setup failure rather than a PR code issue.
• If this recurs, harden .buildkite/scripts/common.sh with_yq() to use curl -f (or equivalent response validation) before extracting the downloaded archive.

Investigation details

Root Cause

The failing command was .buildkite/scripts/run_buildkite_scripts_tests.sh. The script installs yq before running test_trigger_backport.sh when yq is not already on PATH (.buildkite/scripts/run_buildkite_scripts_tests.sh:49-56). That calls with_yq(), which downloads https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/${binary}.tar.gz and immediately extracts it (.buildkite/scripts/common.sh:199-207).

The log shows extraction failed before any trigger_backport_lib.sh assertions ran, which means the unit test suite failed during tool setup, not because of the PR's packages/o365 changes.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/45828
• Job/step: :bash: :snake: Buildkite scripts unit tests
• Key log excerpt:

=== Running trigger_backport_lib.sh tests ===
Adding PATH to the environment variables...
tar: This does not look like a tar archive

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
🚨 Error: The command exited with status 2

The PR diff is limited to packages/o365 files (manifest.yml, changelog.yml, audit pipeline/fields/docs/test expectation), while the failure happens in shared Buildkite script setup before package-specific checks are run.

Verification

• Reviewed the failed Buildkite log and traced the setup path to .buildkite/scripts/run_buildkite_scripts_tests.sh and .buildkite/scripts/common.sh.
• Compared the PR file list; no .buildkite/ files are modified in this PR.

Follow-up

• None if a retry succeeds. If repeated builds hit the same failure, update with_yq() to fail on non-2xx downloads before tar runs, matching the failure mode shown here.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[efd6]

This will need this patch to allow the upgrade test to pass.

0001-o365-make-upgrade-script-test-robust-to-index-roll-o.patch

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[vera-review-bot]

No issues across the latest commits 40a6852.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package o365 👍(0) 💚(0) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
audit	2364.07	1510.57	-853.5 (-36.1%)	💔
audit	2364.07	1893.94	-470.13 (-19.89%)	💔

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 40a6852

History

• 💔 Build #45830 failed 8b9ea6d
• 💔 Build #45828 failed d5a648f
• 💔 Build #45824 failed d5a648f

cc @chemamartinez

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[elastic-vault-github-plugin-prod]

Package o365 - 3.10.4 containing this change is available at https://epr.elastic.co/package/o365/3.10.4/