[Fleet] Strip remote-cluster transform source indices on Serverless#275649
Merged
szwarckonrad merged 6 commits intoJul 2, 2026
Merged
Conversation
szwarckonrad
added
release_note:skip
Contributor
Author
|
@elasticmachine merge upstream |
Contributor
Author
|
/ci |
Contributor
💛 Build succeeded, but was flaky
Failed CI Steps
Test Failures
Metrics [docs]
History
|
|
Pinging @elastic/fleet (Team:Fleet) |
juliaElastic
approved these changes
Jul 2, 2026
MykhailoKondrat
pushed a commit
to MykhailoKondrat/kibana
that referenced
this pull request
Jul 2, 2026
…lastic#275649) ## Summary Installing the **Elastic Defend** integration fails on Serverless **with cross-project search (CPS) enabled** with a `400`. The `metadata_united` transform carries a cross-cluster (`*:`) source so the managing cluster can read endpoint metadata from a remote Elasticsearch output ([elastic/endpoint-package#747](elastic/endpoint-package#747)). CCS is **stateful-only**, so a Serverless project running in cross-project mode rejects any transform whose source targets remote indices (`action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested`), which aborts the whole package install. This strips the remote-cluster (`*:`) entries from **the Defend `metadata_united` transform's** source on Serverless, before `putTransform`, so it installs and runs against its local indices. Stateful is untouched (`*:` there resolves to empty when no remote is connected). The strip is gated on `isServerlessEnabled` (all Serverless), **not** on CPS specifically: CCS is unavailable on Serverless regardless, so a `*:` source is never valid there — stripping it is harmless when CPS is off (it would just resolve to empty) and prevents the failure if CPS is enabled later. ## Deliberately narrow + temporary Per team decision (feature-freeze timing), this is a **minimal, temporary bridge**: - **Scoped to one transform by name** — the strip only touches transforms whose `installationName` contains `endpoint.metadata_united-` (matches v1 + v2 ids). No other package's transforms are affected, even if they carried a `*:` source. - **Isolated for easy removal** — the logic lives in its own `ccs_transform_source.ts` util (+ test), called from `install.ts`'s `handleTransformInstall`. When the long-term approach lands, removal is a clean delete of that file plus one import/call. - The **longer-term** approach is per-environment transform variants selected declaratively at install (`_meta.environments`), so no source is mutated at install: **elastic#275669** (Fleet gate) + **elastic/endpoint-package#750** (the two variants). Longer still, the plan is to remove these transforms entirely. ## What this looks like Installing endpoint `9.5.0-prerelease.1` (the build carrying the `*:` source): | Scenario | Before | After | |---|---|---| | Serverless, cross-project **enabled** | ❌ 400 | ✅ 200 — `*:` stripped, transform `started` | | Serverless, cross-project **disabled** | ✅ 200 (`*:` resolves empty) | ✅ 200 — `*:` stripped (harmless) | | Stateful | ✅ 200 | ✅ 200 (`*:` kept) | <details> <summary>Testing</summary> - **Unit** (`ccs_transform_source.test.ts`): strips the Defend transform's `*:` on serverless (logged); a no-op without a `*:`, on stateful, and for **any non-`metadata_united` transform**. The `isRemoteIndexExpression` classifier mirrors ES `RemoteClusterAware#isRemoteIndexName`, so date-math (e.g. a `+12:00` timezone inside `<...>`) and `::` selectors are not misread as cluster separators. - **Install wiring** (`transforms.test.ts`): the strip runs during `installTransforms` on the legacy JSON path — serverless strips, stateful preserves. - **Manual E2E** on local serverless (cross-project enabled): endpoint install `400 → 200`, installed transform source reduced to local indices, transform `started`. </details> <details> <summary>Root cause (ES, not Fleet)</summary> ES `TransformConfig.validateNoCrossProjectWhenCrossProjectIsDisabled` rejects remote source indices when `CrossProjectModeDecider.crossProjectEnabled()` (node setting `serverless.cross_project.enabled=true`, Serverless only) is true and the `TRANSFORM_CROSS_PROJECT` feature flag is off. `defer_validation:true` doesn't skip it (request-level validation). Fleet's `putTransform` catch only swallows `security_exception`/`resource_already_exists_exception`, so the 400 propagates and fails the install. </details> --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
szwarckonrad
added a commit
that referenced
this pull request
Jul 6, 2026
…with remote ES output (#271559) Depends on: elastic/elasticsearch#152211 elastic/endpoint-package#747 #275649 When an Elastic Defend agent uses a Fleet remote ES output, its endpoint data (metadata, events, policy responses, action responses) is written to the remote cluster rather than the managing cluster. The managing cluster's Kibana then can't read it, so endpoint management surfaces like host list, host details, response actions, policy responses and suggestions come up empty or broken. This PR makes the Security Solution endpoint read paths cross-cluster-search (CCS) aware: when remote clusters are connected, endpoint index patterns are prefixed with `*:` so the managing cluster reads the data wherever the agent shipped it. Same approach osquery shipped in #260518. The entire pathway is gated behind a new experimental feature flag `defendRemoteOutputCcs`, off by default. With the flag off, or when no remote clusters are connected, every prefix is a no-op and behavior is unchanged. What's included - New `ccs_utils.ts` helper (`hasConnectedRemoteClusters` + `prefixIndexPatternsWithCcs`), modeled on osquery's. - ccsEnabled threaded through the endpoint read paths, applied at the DSL/index-pattern build sites: - metadata + agent status (host list, host details, capability validation) - response actions (reads / validation / status / history) - policy responses - suggestions + endpoint fields - workflow insights - The `defendRemoteOutputCcs` flag is checked inside `hasConnectedRemoteClusters`, so it short-circuits before any `_remote/info` call when off. Testing Validated end-to-end on a local two-cluster setup (managing + remote-output, CCS configured): host list, host details, all response action types completing over CCS, policy responses, and suggestions/endpoint fields. Also verified on a single-cluster (no-remotes) deployment that flag-on and flag-off behave identically to today. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Installing the Elastic Defend integration fails on Serverless with cross-project search (CPS) enabled with a
400. Themetadata_unitedtransform carries a cross-cluster (*:) source so the managing cluster can read endpoint metadata from a remote Elasticsearch output (elastic/endpoint-package#747). CCS is stateful-only, so a Serverless project running in cross-project mode rejects any transform whose source targets remote indices (action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested), which aborts the whole package install.This strips the remote-cluster (
*:) entries from the Defendmetadata_unitedtransform's source on Serverless, beforeputTransform, so it installs and runs against its local indices. Stateful is untouched (*:there resolves to empty when no remote is connected).The strip is gated on
isServerlessEnabled(all Serverless), not on CPS specifically: CCS is unavailable on Serverless regardless, so a*:source is never valid there — stripping it is harmless when CPS is off (it would just resolve to empty) and prevents the failure if CPS is enabled later.Deliberately narrow + temporary
Per team decision (feature-freeze timing), this is a minimal, temporary bridge:
installationNamecontainsendpoint.metadata_united-(matches v1 + v2 ids). No other package's transforms are affected, even if they carried a*:source.ccs_transform_source.tsutil (+ test), called frominstall.ts'shandleTransformInstall. When the long-term approach lands, removal is a clean delete of that file plus one import/call._meta.environments), so no source is mutated at install: [Fleet] Install package transforms by stack environment (_meta.environments) #275669 (Fleet gate) + Split metadata_united transform into stateful and serverless variants endpoint-package#750 (the two variants). Longer still, the plan is to remove these transforms entirely.What this looks like
Installing endpoint
9.5.0-prerelease.1(the build carrying the*:source):*:stripped, transformstarted*:resolves empty)*:stripped (harmless)*:kept)Testing
ccs_transform_source.test.ts): strips the Defend transform's*:on serverless (logged); a no-op without a*:, on stateful, and for any non-metadata_unitedtransform. TheisRemoteIndexExpressionclassifier mirrors ESRemoteClusterAware#isRemoteIndexName, so date-math (e.g. a+12:00timezone inside<...>) and::selectors are not misread as cluster separators.transforms.test.ts): the strip runs duringinstallTransformson the legacy JSON path — serverless strips, stateful preserves.400 → 200, installed transform source reduced to local indices, transformstarted.Root cause (ES, not Fleet)
ES
TransformConfig.validateNoCrossProjectWhenCrossProjectIsDisabledrejects remote source indices whenCrossProjectModeDecider.crossProjectEnabled()(node settingserverless.cross_project.enabled=true, Serverless only) is true and theTRANSFORM_CROSS_PROJECTfeature flag is off.defer_validation:truedoesn't skip it (request-level validation). Fleet'sputTransformcatch only swallowssecurity_exception/resource_already_exists_exception, so the 400 propagates and fails the install.