Split metadata_united transform into stateful and serverless variants#750
Draft
szwarckonrad wants to merge 2 commits into
Draft
Split metadata_united transform into stateful and serverless variants#750szwarckonrad wants to merge 2 commits into
szwarckonrad wants to merge 2 commits into
Conversation
This was referenced Jun 30, 2026
Draft
szwarckonrad
added a commit
to elastic/kibana
that referenced
this pull request
Jul 2, 2026
…275649) ## Summary Installing the **Elastic Defend** integration fails on Serverless **with cross-project search (CPS) enabled** with a `400`. The `metadata_united` transform carries a cross-cluster (`*:`) source so the managing cluster can read endpoint metadata from a remote Elasticsearch output ([elastic/endpoint-package#747](elastic/endpoint-package#747)). CCS is **stateful-only**, so a Serverless project running in cross-project mode rejects any transform whose source targets remote indices (`action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested`), which aborts the whole package install. This strips the remote-cluster (`*:`) entries from **the Defend `metadata_united` transform's** source on Serverless, before `putTransform`, so it installs and runs against its local indices. Stateful is untouched (`*:` there resolves to empty when no remote is connected). The strip is gated on `isServerlessEnabled` (all Serverless), **not** on CPS specifically: CCS is unavailable on Serverless regardless, so a `*:` source is never valid there — stripping it is harmless when CPS is off (it would just resolve to empty) and prevents the failure if CPS is enabled later. ## Deliberately narrow + temporary Per team decision (feature-freeze timing), this is a **minimal, temporary bridge**: - **Scoped to one transform by name** — the strip only touches transforms whose `installationName` contains `endpoint.metadata_united-` (matches v1 + v2 ids). No other package's transforms are affected, even if they carried a `*:` source. - **Isolated for easy removal** — the logic lives in its own `ccs_transform_source.ts` util (+ test), called from `install.ts`'s `handleTransformInstall`. When the long-term approach lands, removal is a clean delete of that file plus one import/call. - The **longer-term** approach is per-environment transform variants selected declaratively at install (`_meta.environments`), so no source is mutated at install: **#275669** (Fleet gate) + **elastic/endpoint-package#750** (the two variants). Longer still, the plan is to remove these transforms entirely. ## What this looks like Installing endpoint `9.5.0-prerelease.1` (the build carrying the `*:` source): | Scenario | Before | After | |---|---|---| | Serverless, cross-project **enabled** | ❌ 400 | ✅ 200 — `*:` stripped, transform `started` | | Serverless, cross-project **disabled** | ✅ 200 (`*:` resolves empty) | ✅ 200 — `*:` stripped (harmless) | | Stateful | ✅ 200 | ✅ 200 (`*:` kept) | <details> <summary>Testing</summary> - **Unit** (`ccs_transform_source.test.ts`): strips the Defend transform's `*:` on serverless (logged); a no-op without a `*:`, on stateful, and for **any non-`metadata_united` transform**. The `isRemoteIndexExpression` classifier mirrors ES `RemoteClusterAware#isRemoteIndexName`, so date-math (e.g. a `+12:00` timezone inside `<...>`) and `::` selectors are not misread as cluster separators. - **Install wiring** (`transforms.test.ts`): the strip runs during `installTransforms` on the legacy JSON path — serverless strips, stateful preserves. - **Manual E2E** on local serverless (cross-project enabled): endpoint install `400 → 200`, installed transform source reduced to local indices, transform `started`. </details> <details> <summary>Root cause (ES, not Fleet)</summary> ES `TransformConfig.validateNoCrossProjectWhenCrossProjectIsDisabled` rejects remote source indices when `CrossProjectModeDecider.crossProjectEnabled()` (node setting `serverless.cross_project.enabled=true`, Serverless only) is true and the `TRANSFORM_CROSS_PROJECT` feature flag is off. `defer_validation:true` doesn't skip it (request-level validation). Fleet's `putTransform` catch only swallows `security_exception`/`resource_already_exists_exception`, so the 400 propagates and fails the install. </details> --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
MykhailoKondrat
pushed a commit
to MykhailoKondrat/kibana
that referenced
this pull request
Jul 2, 2026
…lastic#275649) ## Summary Installing the **Elastic Defend** integration fails on Serverless **with cross-project search (CPS) enabled** with a `400`. The `metadata_united` transform carries a cross-cluster (`*:`) source so the managing cluster can read endpoint metadata from a remote Elasticsearch output ([elastic/endpoint-package#747](elastic/endpoint-package#747)). CCS is **stateful-only**, so a Serverless project running in cross-project mode rejects any transform whose source targets remote indices (`action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested`), which aborts the whole package install. This strips the remote-cluster (`*:`) entries from **the Defend `metadata_united` transform's** source on Serverless, before `putTransform`, so it installs and runs against its local indices. Stateful is untouched (`*:` there resolves to empty when no remote is connected). The strip is gated on `isServerlessEnabled` (all Serverless), **not** on CPS specifically: CCS is unavailable on Serverless regardless, so a `*:` source is never valid there — stripping it is harmless when CPS is off (it would just resolve to empty) and prevents the failure if CPS is enabled later. ## Deliberately narrow + temporary Per team decision (feature-freeze timing), this is a **minimal, temporary bridge**: - **Scoped to one transform by name** — the strip only touches transforms whose `installationName` contains `endpoint.metadata_united-` (matches v1 + v2 ids). No other package's transforms are affected, even if they carried a `*:` source. - **Isolated for easy removal** — the logic lives in its own `ccs_transform_source.ts` util (+ test), called from `install.ts`'s `handleTransformInstall`. When the long-term approach lands, removal is a clean delete of that file plus one import/call. - The **longer-term** approach is per-environment transform variants selected declaratively at install (`_meta.environments`), so no source is mutated at install: **elastic#275669** (Fleet gate) + **elastic/endpoint-package#750** (the two variants). Longer still, the plan is to remove these transforms entirely. ## What this looks like Installing endpoint `9.5.0-prerelease.1` (the build carrying the `*:` source): | Scenario | Before | After | |---|---|---| | Serverless, cross-project **enabled** | ❌ 400 | ✅ 200 — `*:` stripped, transform `started` | | Serverless, cross-project **disabled** | ✅ 200 (`*:` resolves empty) | ✅ 200 — `*:` stripped (harmless) | | Stateful | ✅ 200 | ✅ 200 (`*:` kept) | <details> <summary>Testing</summary> - **Unit** (`ccs_transform_source.test.ts`): strips the Defend transform's `*:` on serverless (logged); a no-op without a `*:`, on stateful, and for **any non-`metadata_united` transform**. The `isRemoteIndexExpression` classifier mirrors ES `RemoteClusterAware#isRemoteIndexName`, so date-math (e.g. a `+12:00` timezone inside `<...>`) and `::` selectors are not misread as cluster separators. - **Install wiring** (`transforms.test.ts`): the strip runs during `installTransforms` on the legacy JSON path — serverless strips, stateful preserves. - **Manual E2E** on local serverless (cross-project enabled): endpoint install `400 → 200`, installed transform source reduced to local indices, transform `started`. </details> <details> <summary>Root cause (ES, not Fleet)</summary> ES `TransformConfig.validateNoCrossProjectWhenCrossProjectIsDisabled` rejects remote source indices when `CrossProjectModeDecider.crossProjectEnabled()` (node setting `serverless.cross_project.enabled=true`, Serverless only) is true and the `TRANSFORM_CROSS_PROJECT` feature flag is off. `defer_validation:true` doesn't skip it (request-level validation). Fleet's `putTransform` catch only swallows `security_exception`/`resource_already_exists_exception`, so the 400 propagates and fails the install. </details> --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the
metadata_unitedtransform into two deployment-specific variants, selected at install time via a new_meta.environmentsfield that Fleet honors:default.json— stateful: keeps the cross-cluster (*:) source so the managing cluster can read endpoint metadata from a remote Elasticsearch output._meta.environments: ["stateful"]serverless.json— serverless: local-only source (CCS is unsupported on serverless; ES rejects remote sources in a transform there)._meta.environments: ["serverless"]Both write to the same destination index, so exactly one installs per environment.
Background
The cross-cluster
*:source (#747) broke the endpoint package install on serverless QA projects —action_request_validation_exception: Cross-project calls are not supported, but remote indices were requested— and was reverted in #749 (9.5.0-prerelease.2).This re-lands the CCS source for stateful while keeping serverless on a local-only transform, with no install-time mutation: Fleet installs whichever variant matches the deployment and skips the other.
Paired Kibana change
Requires the Fleet install gate that reads
_meta.environments(links the two variants to the deployment flavor): elastic/kibana#275669.Testing
Validated locally on both stateful and serverless (cross-project enabled): the matching variant installs verbatim (HTTP 200), the other is skipped, and the transform starts healthy. A package without
_meta.environmentsinstalls unchanged (back-compat).