[Fleet] Enable experimental toggles in Fleet UI from Kibana config

[kpollich]

Enable the experimental data stream toggles (synthetic source, TSDB, doc-values-only) in Fleet's policy editor by toggling a feature flag in the Kibana config used for launching the local Kibana instance.

Ref elastic/kibana#148418
Ref elastic/kibana#148317

Not sure if this should also be turned on in the other Kibana config files:

• kibana_config_8x.yml
• kibana_config_80.yml

If they should be edited too, let me know and I'll happily make the changes.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-23T16:01:55.977+0000

• Duration: 13 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	464
Skipped	0
Total	464

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

Check

• Took 3 min 21 sec . View more details here
• Description: make install test-check-packages-with-kind check-git-clean

Check

• Took 3 min 58 sec . View more details here
• Description: make install test-check-packages-other check-git-clean

Check

• Took 3 min 52 sec . View more details here
• Description: make install test-check-packages-with-custom-agent check-git-clean

Check

• Took 3 min 52 sec . View more details here
• Description: make install test-check-packages-benchmarks check-git-clean

Check

• Took 3 min 26 sec . View more details here
• Description: make install test-build-zip check-git-clean

Build elastic-package

• Took 3 min 2 sec . View more details here
• Description: make PACKAGE_UNDER_TEST=apache test-check-packages-parallel

Build elastic-package

• Took 3 min 2 sec . View more details here
• Description: make PACKAGE_UNDER_TEST=aws test-check-packages-parallel

Build elastic-package

• Took 2 min 57 sec . View more details here
• Description: make PACKAGE_UNDER_TEST=gcp test-check-packages-parallel

Build elastic-package

• Took 3 min 31 sec . View more details here
• Description: make PACKAGE_UNDER_TEST=nginx test-check-packages-parallel

Build elastic-package

• Took 2 min 55 sec . View more details here
• Description: make PACKAGE_UNDER_TEST=sql_input test-check-packages-parallel

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (27/27)	💚
Files	65.833% (79/120)	👎 -0.076
Classes	61.017% (108/177)	👎 -0.359
Methods	43.825% (291/664)	👎 -4.817
Lines	28.083% (2908/10355)	👎 -3.624
Conditionals	100.0% (0/0)	💚

[ruflin]

LGTM

[jsoriano]

Failures in CI seem related to this change, should this setting be applied only for versions >= 8.7? (We don't have an easy way to do it).

�[36mkibana_1                     |�[0m [2023-01-23T16:12:56.735+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
�[36mkibana_1                     |�[0m [2023-01-23T16:13:10.753+00:00][FATAL][root] Error: [config validation of [xpack.fleet].enableExperimental]: [experimentalDataStreamSettings] is not allowed. Allowed values are: createPackagePolicyMultiPageLayout, packageVerification, showDevtoolsRequest, diagnosticFileUploadEnabled

[kpollich]

Failures in CI seem related to this change, should this setting be applied only for versions >= 8.7? (We don't have an easy way to do it).

Yes, this option is only available in 8.7+. Due to the nature of Kibana's config validator, the setting will result in an invalid config for older versions.

[ruflin]

@jsoriano @kpollich For our current testing, is there a manual way to overwrite it by setting some environment variables or similar?

[jsoriano]

For our current testing, is there a manual way to overwrite it by setting some environment variables or similar?

I don't think so.

Can these flags be changed on runtime through the API?

[joshdover]

Kibana supports setting configs via env vars in the Docker images: https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config

Probably something like XPACK_FLEET_ENABLEEXPERIMENTAL=true would work.

I think this isn't automatic though, so there might be some Docker entrypoint script that needs to be updated for the image to support this. Kibana operations team would be the one to ask.

[ruflin]

As a manual workaround what I did for now is add xpack.fleet.enableExperimental: ["experimentalDataStreamSettings"] as an entry into .elastic-package/profiles/default/stack/kibana.config.8x.yml and run elastic-package stack up --version=8.7.0-SNAPSHOT -v and it works. This is a manual way to make sure we don't get blocked for testing.

[joshdover]

Ah found it, think we'd have to add the configs here: https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker#L277

[ruflin]

Can these flags be changed on runtime through the API?

@jsoriano Unfortunately no.

[ruflin]

Can we get this moving? Most of these flags were introduced so we can test it with packages. Not having it available in elastic-package is problematic as this is where most of us tests packages.

[jsoriano]

Can we get this moving? Most of these flags were introduced so we can test it with packages. Not having it available in elastic-package is problematic as this is where most of us tests packages.

Let me take a quick look to adding a new config file for Kibana 8.7.

[jsoriano]

#1128

[jsoriano]

Change merged as part of #1128, closing this one.

[ruflin]

@jsoriano Thanks!