[Fleet] Install package transforms by stack environment (_meta.environments)#275669
Draft
szwarckonrad wants to merge 1 commit into
Draft
[Fleet] Install package transforms by stack environment (_meta.environments)#275669szwarckonrad wants to merge 1 commit into
szwarckonrad wants to merge 1 commit into
Conversation
szwarckonrad
added
release_note:skip
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
This was referenced Jun 30, 2026
Draft
szwarckonrad
added a commit
that referenced
this pull request
Jul 2, 2026
…275649) ## Summary Installing the **Elastic Defend** integration fails on Serverless **with cross-project search (CPS) enabled** with a `400`. The `metadata_united` transform carries a cross-cluster (`*:`) source so the managing cluster can read endpoint metadata from a remote Elasticsearch output ([elastic/endpoint-package#747](elastic/endpoint-package#747)). CCS is **stateful-only**, so a Serverless project running in cross-project mode rejects any transform whose source targets remote indices (`action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested`), which aborts the whole package install. This strips the remote-cluster (`*:`) entries from **the Defend `metadata_united` transform's** source on Serverless, before `putTransform`, so it installs and runs against its local indices. Stateful is untouched (`*:` there resolves to empty when no remote is connected). The strip is gated on `isServerlessEnabled` (all Serverless), **not** on CPS specifically: CCS is unavailable on Serverless regardless, so a `*:` source is never valid there — stripping it is harmless when CPS is off (it would just resolve to empty) and prevents the failure if CPS is enabled later. ## Deliberately narrow + temporary Per team decision (feature-freeze timing), this is a **minimal, temporary bridge**: - **Scoped to one transform by name** — the strip only touches transforms whose `installationName` contains `endpoint.metadata_united-` (matches v1 + v2 ids). No other package's transforms are affected, even if they carried a `*:` source. - **Isolated for easy removal** — the logic lives in its own `ccs_transform_source.ts` util (+ test), called from `install.ts`'s `handleTransformInstall`. When the long-term approach lands, removal is a clean delete of that file plus one import/call. - The **longer-term** approach is per-environment transform variants selected declaratively at install (`_meta.environments`), so no source is mutated at install: **#275669** (Fleet gate) + **elastic/endpoint-package#750** (the two variants). Longer still, the plan is to remove these transforms entirely. ## What this looks like Installing endpoint `9.5.0-prerelease.1` (the build carrying the `*:` source): | Scenario | Before | After | |---|---|---| | Serverless, cross-project **enabled** | ❌ 400 | ✅ 200 — `*:` stripped, transform `started` | | Serverless, cross-project **disabled** | ✅ 200 (`*:` resolves empty) | ✅ 200 — `*:` stripped (harmless) | | Stateful | ✅ 200 | ✅ 200 (`*:` kept) | <details> <summary>Testing</summary> - **Unit** (`ccs_transform_source.test.ts`): strips the Defend transform's `*:` on serverless (logged); a no-op without a `*:`, on stateful, and for **any non-`metadata_united` transform**. The `isRemoteIndexExpression` classifier mirrors ES `RemoteClusterAware#isRemoteIndexName`, so date-math (e.g. a `+12:00` timezone inside `<...>`) and `::` selectors are not misread as cluster separators. - **Install wiring** (`transforms.test.ts`): the strip runs during `installTransforms` on the legacy JSON path — serverless strips, stateful preserves. - **Manual E2E** on local serverless (cross-project enabled): endpoint install `400 → 200`, installed transform source reduced to local indices, transform `started`. </details> <details> <summary>Root cause (ES, not Fleet)</summary> ES `TransformConfig.validateNoCrossProjectWhenCrossProjectIsDisabled` rejects remote source indices when `CrossProjectModeDecider.crossProjectEnabled()` (node setting `serverless.cross_project.enabled=true`, Serverless only) is true and the `TRANSFORM_CROSS_PROJECT` feature flag is off. `defer_validation:true` doesn't skip it (request-level validation). Fleet's `putTransform` catch only swallows `security_exception`/`resource_already_exists_exception`, so the 400 propagates and fails the install. </details> --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
MykhailoKondrat
pushed a commit
to MykhailoKondrat/kibana
that referenced
this pull request
Jul 2, 2026
…lastic#275649) ## Summary Installing the **Elastic Defend** integration fails on Serverless **with cross-project search (CPS) enabled** with a `400`. The `metadata_united` transform carries a cross-cluster (`*:`) source so the managing cluster can read endpoint metadata from a remote Elasticsearch output ([elastic/endpoint-package#747](elastic/endpoint-package#747)). CCS is **stateful-only**, so a Serverless project running in cross-project mode rejects any transform whose source targets remote indices (`action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested`), which aborts the whole package install. This strips the remote-cluster (`*:`) entries from **the Defend `metadata_united` transform's** source on Serverless, before `putTransform`, so it installs and runs against its local indices. Stateful is untouched (`*:` there resolves to empty when no remote is connected). The strip is gated on `isServerlessEnabled` (all Serverless), **not** on CPS specifically: CCS is unavailable on Serverless regardless, so a `*:` source is never valid there — stripping it is harmless when CPS is off (it would just resolve to empty) and prevents the failure if CPS is enabled later. ## Deliberately narrow + temporary Per team decision (feature-freeze timing), this is a **minimal, temporary bridge**: - **Scoped to one transform by name** — the strip only touches transforms whose `installationName` contains `endpoint.metadata_united-` (matches v1 + v2 ids). No other package's transforms are affected, even if they carried a `*:` source. - **Isolated for easy removal** — the logic lives in its own `ccs_transform_source.ts` util (+ test), called from `install.ts`'s `handleTransformInstall`. When the long-term approach lands, removal is a clean delete of that file plus one import/call. - The **longer-term** approach is per-environment transform variants selected declaratively at install (`_meta.environments`), so no source is mutated at install: **elastic#275669** (Fleet gate) + **elastic/endpoint-package#750** (the two variants). Longer still, the plan is to remove these transforms entirely. ## What this looks like Installing endpoint `9.5.0-prerelease.1` (the build carrying the `*:` source): | Scenario | Before | After | |---|---|---| | Serverless, cross-project **enabled** | ❌ 400 | ✅ 200 — `*:` stripped, transform `started` | | Serverless, cross-project **disabled** | ✅ 200 (`*:` resolves empty) | ✅ 200 — `*:` stripped (harmless) | | Stateful | ✅ 200 | ✅ 200 (`*:` kept) | <details> <summary>Testing</summary> - **Unit** (`ccs_transform_source.test.ts`): strips the Defend transform's `*:` on serverless (logged); a no-op without a `*:`, on stateful, and for **any non-`metadata_united` transform**. The `isRemoteIndexExpression` classifier mirrors ES `RemoteClusterAware#isRemoteIndexName`, so date-math (e.g. a `+12:00` timezone inside `<...>`) and `::` selectors are not misread as cluster separators. - **Install wiring** (`transforms.test.ts`): the strip runs during `installTransforms` on the legacy JSON path — serverless strips, stateful preserves. - **Manual E2E** on local serverless (cross-project enabled): endpoint install `400 → 200`, installed transform source reduced to local indices, transform `started`. </details> <details> <summary>Root cause (ES, not Fleet)</summary> ES `TransformConfig.validateNoCrossProjectWhenCrossProjectIsDisabled` rejects remote source indices when `CrossProjectModeDecider.crossProjectEnabled()` (node setting `serverless.cross_project.enabled=true`, Serverless only) is true and the `TRANSFORM_CROSS_PROJECT` feature flag is off. `defer_validation:true` doesn't skip it (request-level validation). Fleet's `putTransform` catch only swallows `security_exception`/`resource_already_exists_exception`, so the 400 propagates and fails the install. </details> --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets a package declare, per transform, which stack flavor it applies to via a new
_meta.environmentsfield, and has Fleet install only the matching variant — verbatim, with no install-time mutation.In
installTransforms, transform assets whose_meta.environmentsexcludes the current flavor (stateful/serverless, fromisServerlessEnabled) are filtered out before install. A transform with no_meta.environmentsinstalls everywhere, so every existing package is unaffected (back-compat).Why
The endpoint
metadata_unitedtransform gained a cross-cluster (*:) source for remote-output support (elastic/endpoint-package#747). CCS is stateful-only, so on serverless ES rejects any transform with a remote source (action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested), which aborts the whole package install — it had to be reverted (elastic/endpoint-package#749).This is the declarative fix: the package ships a stateful variant (with
*:) and a serverless variant (local-only), each tagged with_meta.environments, and Fleet installs the right one. Nothing is mutated at install, so the runtime transform matches the package definition.Pairs with elastic/endpoint-package#750 (the two variants).
Changes
transform/install.ts—installTransformsfilters transform paths by_meta.environmentsvs the current stack flavor before delegating to the (untouched) install functions. Skips are logged.security_solution/common/endpoint/constants.ts—METADATA_UNITED_TRANSFORMwidened to the variant-agnostic prefixendpoint.metadata_united-, so the endpoint data-loader/cypress/evals helpers match either the-default(stateful) or-serverlessid. Every consumer uses it as a prefix (startsWith/${...}*), never as an exact id.Testing
transforms.test.ts): serverless installs only the serverless variant; stateful only the stateful variant; a field-less transform installs everywhere (back-compat).started; the field-less published package (9.5.0-prerelease.2) installs unchanged.Follow-up
x-pack/platform/test/fleet_api_integration/apis/epm/install_endpoint.tshardcodes the-defaulttransform id (it tests the published package, green today). It will need a deployment-aware assertion once the endpoint-package variants ship to the registry and that FTR runs on serverless.