diff --git a/.backportrc.json b/.backportrc.json index 24ccaa6b7..1ba6c84b7 100644 --- a/.backportrc.json +++ b/.backportrc.json @@ -1,7 +1,7 @@ { "repoOwner": "elastic", "repoName": "ingest-docs", - "targetBranchChoices": ["main", "8.15","8.14","8.13","8.12", "8.11", "8.10", "8.9", "8.8", "8.7", "8.6", "7.17"], + "targetBranchChoices": ["8.18","8.17","8.16","8.15","8.14","8.13","8.12", "8.11", "8.10", "8.9", "8.8", "8.7", "8.6", "7.17"], "autoMerge": true, "autoMergeMethod": "squash", "branchLabelMapping": { diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 8751696d8..84a111847 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1 +1 @@ -* @elastic/platform-docs +* @elastic/ingest-docs diff --git a/docs/en/ingest-management/agent-policies.asciidoc b/docs/en/ingest-management/agent-policies.asciidoc index 1fd6a1cfc..e5dc29ed1 100644 --- a/docs/en/ingest-management/agent-policies.asciidoc +++ b/docs/en/ingest-management/agent-policies.asciidoc @@ -96,7 +96,11 @@ The following table illustrates the {fleet} user actions available to different |{y} |{n} -|<> +|<> +|{y} +|{n} + +|<> |{y} |{n} @@ -116,10 +120,6 @@ The following table illustrates the {fleet} user actions available to different |{y} |{n} -|<> -|{y} -|{n} - |<> |{y} |{n} @@ -162,6 +162,9 @@ Also see <>. == Add an integration to a policy An {agent} policy consists of one or more integrations that are applied to the agents enrolled in that policy. +When you add an integration, the policy created for that integration can be shared with multiple {agent} policies. +This reduces the number of integrations policies that you need to actively manage. + To add a new integration to one or more {agent} policies: . In {fleet}, click **Agent policies**. @@ -184,6 +187,11 @@ You can update the settings for an installed integration at any time: . In {kib}, go to the **Integrations** page. . On the **Integration policies** tab, for the integration that you like to update open the **Actions** menu and select **Edit integration**. . On the **Edit ** page you can update any configuration settings and also update the list of {agent} polices to which the integration is added. ++ +If you clear the **Agent policies** field, the integration will be removed from any {agent} policies to which it had been added. ++ +To identify any integrations that have been "orphaned", that is, not associated with any {agent} policies, check the **Agent polices** column on the **Integration policies** tab. +Any integrations that are installed but not associated with an {agent} policy are as labeled as `No agent policies`. [discrete] [[apply-a-policy]] @@ -301,23 +309,76 @@ Note that adding custom tags is not supported for a small set of inputs: * `cloud-defend` * `fleet-server` * `pf-elastic-collector`, `pf-elastic-symbolizer`, and `pf-host-agent` +* `endpoint` inputs. Instead, use the advanced settings (`*.advanced.document_enrichment.fields`) of the {elastic-defend} Integration. [discrete] [[change-policy-enable-agent-monitoring]] -== Enable agent monitoring +== Configure agent monitoring -Use this setting to collect monitoring logs and metrics from {agent}. All monitoring data will be written to the specified **Default namespace**. +Use these settings to collect monitoring logs and metrics from {agent}. All monitoring data will be written to the specified **Default namespace**. . In {fleet}, click **Agent policies**. Select the name of the policy you want to edit. -. Click the **Settings** tab and scroll to **Enable agent monitorings**. +. Click the **Settings** tab and scroll to **Agent monitoring**. . Select whether to collect agent logs, agent metrics, or both, from the {agents} that use the policy. - ++ When this setting is enabled an {agent} integration is created automatically. +. Expand the **Advanced monitoring options** section to access <>. + +. Save your changes for the updated monitoring settings to take effect. + +[discrete] +[[advanced-agent-monitoring-settings]] +=== Advanced agent monitoring settings + +**HTTP monitoring endpoint** + +Enabling this setting exposes a `/liveness` API endpoint that you can use to monitor {agent} health according to the following HTTP codes: + +* `200`: {agent} is healthy. The endpoint returns a `200` OK status as long as {agent} is responsive and can process configuration changes. +* `500`: A component or unit is in a failed state. +* `503`: The agent coordinator is unresponsive. + +You can pass a `failon` parameter to the `/liveness` endpoint to determine what component state will result in a `500` status. For example, `curl 'localhost:6792/liveness?failon=degraded'` will return `500` if a component is in a degraded state. + +The possible values for `failon` are: + +* `degraded`: Return an error if a component is in a degraded state or failed state, or if the agent coordinator is unresponsive. +* `failed`: Return an error if a unit is in a failed state, or if the agent coordinator is unresponsive. +* `heartbeat`: Return an error only if the agent coordinator is unresponsive. + +If no `failon` parameter is provided, the default `failon` behavior is `heartbeat`. + +The HTTP monitoring endpoint can also be link:https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-http-request[used with Kubernetes], to restart the container for example. + +When you enable this setting, you need to provide the host URL and port where the endpoint can be accessed. Using the default `localhost` is recommended. + +When the HTTP monitoring endpoint is enabled you can also select to **Enable profiling at `/debug/pprof`**. This controls whether the {agent} exposes the `/debug/pprof/` endpoints together with the monitoring endpoints. + +The heap profiles available from `/debug/pprof/` are included in <> by default. CPU profiles are also included when the `--cpu-profile` option is included. For full details about the profiles exposed by `/debug/pprof/` refer to the link:https://pkg.go.dev/net/http/pprof[pprof package documentation]. + +Profiling at `/debug/pprof` is disabled by default. Data produced by these endpoints can be useful for debugging but present a security risk. It's recommended to leave this option disabled if the monitoring endpoint is accessible over a network. + +**Diagnostics rate limiting** + +You can set a rate limit for the action handler for diagnostics requests coming from {fleet}. The setting affects only {fleet}-managed {agents}. By default, requests are limited to an interval of `1m` and a burst value of `1`. This setting does not affect diagnostics collected through the CLI. + +**Diagnostics file upload** + +This setting configures retries for the file upload client handling diagnostics requests coming from {fleet}. The setting affects only {fleet}-managed {agents}. By default, a maximum of `10` retries are allowed with an initial duration of `1s` and a backoff duration of `1m`. The client may retry failed requests with exponential backoff. + +[discrete] +[[agent-policy-automatic-agent-upgrade]] +== Configure an automatic {agent} upgrade + +For a high-scale deployment of {fleet}, you can configure an automatic, gradual rollout of a new minor or patch version to a percentage of the {agents} in your policy. For more information, refer to <>. + +NOTE: This feature is only available for certain subscription levels. For more information, refer to {subscriptions}[{stack} subscriptions]. + [discrete] [[change-policy-output]] == Change the output of a policy @@ -409,22 +470,6 @@ Select the name of the policy you want to edit. . Set **Limit CPU usage** as needed. For example, to limit Go processes supervised by {agent} to two operating system threads each, set this value to `2`. -[discrete] -[[agent-policy-http-monitoring]] -== Override the default monitoring port - -You can override the default port that {agent} uses to send monitoring data. It's useful to be able to adjust this setting if you have an application running on the machine on which the agent is deployed, and that is using the same port. - -. In {fleet}, click **Agent policies**. -Select the name of the policy you want to edit. - -. Click the **Settings** tab and scroll to **Advanced settings**. - -//. Set **Agent HTTP monitoring** setting to enabled, and then specify a host and port for the monitoring data output. -. Specify a host and port for the monitoring data output. - -//. Enable **buffer.enabled** if you'd like {agent} and {beats} to collect metrics into an in-memory buffer and expose these through a `/buffer` endpoint. This data can be useful for debugging or if the {agent} has issues communicating with {es}. Enabling this option may slightly increase process memory usage. - [discrete] [[agent-policy-log-level]] == Set the {agent} log level @@ -524,4 +569,6 @@ The unenroll task runs every ten minutes, and it unenrolls a maximum of one thou [[agent-policy-scale]] == Policy scaling recommendations -A single instance of {fleet} supports a maximum of 500 {agent} policies. If more policies are configured, UI performance might be impacted. +A single instance of {fleet} supports a maximum of 1000 {agent} policies. If more policies are configured, UI performance might be impacted. The maximum number of policies is not affected by the number of spaces in which the policies are used. + +If you are using {agent} with link:{serverless-docs}[{serverless-full}], the maximum supported number of {agent} policies is 500. diff --git a/docs/en/ingest-management/beats-agent-comparison.asciidoc b/docs/en/ingest-management/beats-agent-comparison.asciidoc index cad8123cf..b90c8d23f 100644 --- a/docs/en/ingest-management/beats-agent-comparison.asciidoc +++ b/docs/en/ingest-management/beats-agent-comparison.asciidoc @@ -271,9 +271,9 @@ The following table shows a comparison of capabilities supported by {beats} and |Run without root on host |{y} -|{n} -|{n} -|{fleet}-managed {agent}s require root permission, in particular for {elastic-defend}. Standalone {agent}s also require root permissions. {beats} requires root permission only if it's configured to capture data that requires that level of permission. +|{y} +|{y} +|{fleet}-managed {agent}s, Standalone {agent}s, and {beats} require root permission only if they're configured to capture data that requires that level of permission. |Multiple outputs |{y} diff --git a/docs/en/ingest-management/commands.asciidoc b/docs/en/ingest-management/commands.asciidoc index b8ff95146..a56492a4d 100644 --- a/docs/en/ingest-management/commands.asciidoc +++ b/docs/en/ingest-management/commands.asciidoc @@ -28,15 +28,15 @@ invoking the wrong binary. * <> * <> * <> -* <> [technical preview] -* <> [technical preview] +* <> preview:[] +* <> * <> * <> * <> * <> * <> * <> -* <> [technical preview] +* <> * <> //* <> @@ -77,7 +77,7 @@ This command is intended for debugging purposes only. The output format and stru [source,shell] ---- elastic-agent diagnostics [--file ] - [-p] + [--cpu-profile] [--exclude-events] [--help] [global-flags] @@ -92,9 +92,12 @@ Specifies the output archive name. Defaults to `elastic-agent-diagnostics- [--delay-enroll] [--elastic-agent-cert ] [--elastic-agent-cert-key ] + [--elastic-agent-cert-key-passphrase ] [--force] [--header ] [--help] @@ -188,6 +192,7 @@ elastic-agent enroll --fleet-server-es [--delay-enroll] [--elastic-agent-cert ] [--elastic-agent-cert-key ] + [--elastic-agent-cert-key-passphrase ] [--fleet-server-cert ] <1> [--fleet-server-cert-key ] [--fleet-server-cert-key-passphrase ] @@ -204,7 +209,6 @@ elastic-agent enroll --fleet-server-es [--force] [--header ] [--help] - [--non-interactive] [--proxy-disabled] [--proxy-header ] [--proxy-url ] @@ -248,6 +252,12 @@ Certificate to use as the client certificate for the {agent}'s connections to {f `--elastic-agent-cert-key`:: Private key to use as for the {agent}'s connections to {fleet-server}. +`--elastic-agent-cert-key-passphrase`:: +The path to the file that contains the passphrase for the mutual TLS private key that {agent} will use to connect to {fleet-server}. +The file must only contain the characters of the passphrase, no newline or extra non-printing characters. ++ +This option is only used if the `--elastic-agent-cert-key` is encrypted and requires a passphrase to use. + `--enrollment-token `:: Enrollment token to use to enroll {agent} into {fleet}. You can use the same enrollment token for multiple agents. @@ -341,11 +351,6 @@ verified. The content is encrypted, but the certificate is not verified. + We strongly recommend that you use a secure connection. -`--non-interactive`:: -Install {agent} in a non-interactive mode. This flag is helpful when -using automation software or scripted deployments. If {agent} is -already installed on the host, the installation will terminate. - `--proxy-disabled`:: Disable proxy support including environment variables. @@ -521,8 +526,6 @@ elastic-agent inspect components log-default [[elastic-agent-privileged-command]] == elastic-agent privileged -preview::[] - Run {agent} with full superuser privileges. This is the usual, default running mode for {agent}. The `privileged` command allows you to switch back to running an agent with full administrative privileges when you have been running it in `unprivileged` mode. @@ -579,6 +582,7 @@ elastic-agent install --url [--delay-enroll] [--elastic-agent-cert ] [--elastic-agent-cert-key ] + [--elastic-agent-cert-key-passphrase ] [--force] [--header ] [--help] @@ -610,6 +614,7 @@ elastic-agent install --fleet-server-es [--delay-enroll] [--elastic-agent-cert ] [--elastic-agent-cert-key ] + [--elastic-agent-cert-key-passphrase ] [--fleet-server-cert ] <1> [--fleet-server-cert-key ] [--fleet-server-cert-key-passphrase ] @@ -678,6 +683,12 @@ Certificate to use as the client certificate for the {agent}'s connections to {f `--elastic-agent-cert-key`:: Private key to use as for the {agent}'s connections to {fleet-server}. +`--elastic-agent-cert-key-passphrase`:: +The path to the file that contains the passphrase for the mutual TLS private key that {agent} will use to connect to {fleet-server}. +The file must only contain the characters of the passphrase, no newline or extra non-printing characters. ++ +This option is only used if the `--elastic-agent-cert-key` is encrypted and requires a passphrase to use. + `--enrollment-token `:: Enrollment token to use to enroll {agent} into {fleet}. You can use the same enrollment token for multiple agents. @@ -806,8 +817,22 @@ tags, you must unenroll the {agent}, then re-enroll it using new tags. Run {agent} without full superuser privileges. This option is useful in organizations that limit `root` access on Linux or macOS systems, or `admin` access on Windows systems. For details and limitations for running {agent} in this mode, refer to {fleet-guide}/elastic-agent-unprivileged.html[Run {agent} without administrative privileges]. - ++ Note that changing to `unprivileged` mode is prevented if the agent is currently enrolled in a policy that includes an integration that requires administrative access, such as the {elastic-defend} integration. ++ +preview:[] To run {agent} without superuser privileges as a pre-existing user or group, for instance under an Active Directory account, you can specify the user or group, and the password to use. ++ +For example: ++ +[source,shell] +---- +elastic-agent install --unprivileged --user="my.path\username" --password="mypassword" +---- ++ +[source,shell] +---- +elastic-agent install --unprivileged --group="my.path\groupname" --password="mypassword" +---- `--url `:: {fleet-server} URL to use to enroll the {agent} into {fleet}. @@ -1105,16 +1130,19 @@ to remove files. [IMPORTANT] ==== -Be sure to run the `uninstall` command from the directory where {agent} is installed and not from the directory where you previously ran the `install` command. +Be sure to run the `uninstall` command from a directory outside of where {agent} is installed. --- - -include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/uninstall-widget.asciidoc[] +For example, on a Windows system the install location is `C:\Program Files\Elastic\Agent`. Run the uninstall command from `C:\Program Files\Elastic` or `\tmp`, or even your default home directory: --- +[source,shell] +---- +C:\"Program Files"\Elastic\Agent\elastic-agent.exe uninstall +---- ==== +include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/uninstall-widget.asciidoc[] + [discrete] === Synopsis @@ -1130,6 +1158,9 @@ elastic-agent uninstall [--force] [--help] [global-flags] Uninstall {agent} and do not prompt for confirmation. This flag is helpful when using automation software or scripted deployments. +`--skip-fleet-audit`:: +Skip auditing with the {fleet-server}. + `--help`:: Show help for the `uninstall` command. @@ -1151,22 +1182,38 @@ elastic-agent uninstall [[elastic-agent-unprivileged-command]] == elastic-agent unprivileged -preview::[] - Run {agent} without full superuser privileges. This is useful in organizations that limit `root` access on Linux or macOS systems, or `admin` access on Windows systems. For details and limitations for running {agent} in this mode, refer to {fleet-guide}/elastic-agent-unprivileged.html[Run {agent} without administrative privileges]. Note that changing a running {agent} to `unprivileged` mode is prevented if the agent is currently enrolled with a policy that contains the {elastic-defend} integration. +preview:[] To run {agent} without superuser privileges as a pre-existing user or group, for instance under an Active Directory account, add either a `--user` or `--group` parameter together with a `--password` parameter. + [discrete] === Examples +Run {agent} without administrative privileges: + [source,shell] ---- elastic-agent unprivileged ---- +preview:[] Run {agent} without administrative privileges, as a pre-existing user: + +[source,shell] +---- +elastic-agent unprivileged --user="my.pathl\username" --password="mypassword" +---- + +preview:[] Run {agent} without administrative privileges, as a pre-existing group: + +[source,shell] +---- +elastic-agent unprivileged --group="my.pathl\groupname" --password="mypassword" +---- + ++++
++++ diff --git a/docs/en/ingest-management/data-streams.asciidoc b/docs/en/ingest-management/data-streams.asciidoc index 8182bc0c1..a113548d1 100644 --- a/docs/en/ingest-management/data-streams.asciidoc +++ b/docs/en/ingest-management/data-streams.asciidoc @@ -103,12 +103,18 @@ For data streams, the index template configures the stream's backing indices as These templates are loaded when the integration is installed, and are used to configure the integration's data streams. [discrete] +[[data-streams-index-templates-edit]] == Edit the {es} index template WARNING: Custom index mappings may conflict with the mappings defined by the integration and may break the integration in {kib}. Do not change or customize any default mappings. -When you install an integration, {fleet} creates a default `@custom` component template for each data stream. The Custom component templates are named following the pattern: `@custom`. +When you install an integration, {fleet} creates two default `@custom` component templates: + +* A `@custom` component template allowing customization across all documents of a given data stream type, named following the pattern: `@custom`. +* A `@custom` component template for each data stream, named following the pattern: `@custom`. + +The `@custom` component template specific to a datastream has higher precedence over the data stream type `@custom` component template. You can edit a `@custom` component template to customize your {es} indices: @@ -182,6 +188,8 @@ this pipeline can be used for custom data processing, adding fields, sanitizing Starting in version 8.12, ingest pipelines can be configured to process events at various levels of customization. +NOTE: If you create a custom index pipeline, Elastic is not responsible for ensuring that it indexes and behaves as expected. Creating a custom pipeline involves custom processing of the incoming data, which should be done with caution and tested carefully. + `global@custom`:: Apply processing to all events + @@ -289,17 +297,240 @@ Refer to the breaking change in the 8.12.0 Release Notes for more detail and wor See <> to get started. [[data-streams-ilm-tutorial]] -== Tutorial: Customize data retention policies +== Tutorials: Customize data retention policies + +These tutorials explain how to apply a custom {ilm-init} policy to an integration's data stream. + +[discrete] +[[data-streams-general-info]] +== Before you begin + +For certain features you'll need to use a slightly different procedure to manage the index lifecycle: + +* APM: For verions 8.15 and later, refer to {observability-guide}/apm-ilm-how-to.html[Index lifecycle management]. +* Synthetic monitoring: Refer to {observability-guide}/synthetics-manage-retention.html[Manage data retention]. +* Universal Profiling: Refer to {observability-guide}/profiling-index-lifecycle-management.html[Universal Profiling index life cycle management]. + +[discrete] +[[data-streams-scenarios]] +== Identify your scenario + +How you apply an ILM policy depends on your use case. Choose a scenario for the detailed steps. + +* **<>**: You want to apply an ILM policy to all logs or metrics data streams across all namespaces. + +* **<>**: You want to apply an ILM policy to selected data streams in an integration. + +* **<>**: You want apply an ILM policy for data streams in a selected namespace in an integration. + + +[[data-streams-scenario1]] +== Scenario 1: Apply an ILM policy to all data streams generated from Fleet integrations across all namespaces + +++++ +Scenario 1: All data streams in all namespaces +++++ + +NOTE: This tutorial uses a `logs@custom` and a `metrics@custom` component template which are available in versions 8.13 and later. +For versions later than 8.4 and earlier than 8.13, you instead need to use the `@custom component template` and add the ILM policy to that template. +This needs to be done for every newly added integration. + +Mappings and settings for data streams can be customized through the creation of `*@custom` component templates, which are referenced by the index templates created by each integration. +The easiest way to configure a custom index lifecycle policy per data stream is to edit this template. + +This tutorial explains how to apply a custom index lifecycle policy to all of the data streams associated with the `System` integration, as an example. +Similar steps can be used for any other integration. +Setting a custom index lifecycle policy must be done separately for all logs and for all metrics, as described in the following steps. + +[discrete] +[id="data-streams-scenario1-step1"] +== Step 1: Create an index lifecycle policy + +. To open **Lifecycle Policies**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field]. +. Click **Create policy**. + +Name your new policy. +For this tutorial, you can use `my-ilm-policy`. +Customize the policy to your liking, and when you're done, click **Save policy**. + +[discrete] +[id="data-streams-scenario1-step2"] +== Step 2: Create a component template for the `logs` index templates + +The **Index Templates** view in {kib} shows you all of the index templates available to automatically apply settings, mappings, and aliases to indices: + +. To open **Index Management**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field]. +. Select **Index Templates**. +. Search for `system` to see all index templates associated with the System integration. +. Select any `logs-*` index template to view the associated component templates. For example, you can select the `logs-system.application` index template. ++ +[role="screenshot"] +image::images/component-templates-list.png[List of component templates available for the index template] + +. Select `logs@custom` in the list to view the component template properties. +. For a newly added integration, the component template won't exist yet. +Select **Create component template** to create it. +If the component template already exists, click **Manage** to update it. +. On the **Logistics** page, keep all defaults and click **Next**. +. On the **Index settings** page, in the **Index settings** field, specify the ILM policy that you created. For example: ++ +[source,json] +---- +{ + "index": { + "lifecycle": { + "name": "my-ilm-policy" + } + } +} +---- + +. Click **Next**. +. For both the **Mappings** and **Aliases** pages, keep all defaults and click **Next**. +. Finally, on the **Review** page, review the summary and request. If everything looks good, select **Create component template**. ++ +[role="screenshot"] +image::images/review-component-template01.png[Review details for the new component template] + +[discrete] +[id="data-streams-scenario1-step3"] +== Step 3: Roll over the data streams (optional) + +To confirm that the index template is using the `logs@custom` component template with your custom ILM policy: + +. Reopen the **Index Management** page and open the **Component Templates** tab. +. Search for `logs@` and select the `logs@custom` component template. +. The **Summary** shows the list of all data streams that use the component template, and the **Settings** view shows your newly configured ILM policy. + +New ILM policies only take effect when new indices are created, +so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), +or force a rollover of each data stream using the {ref}/indices-rollover-index.html[{es} rollover API. + +For example: + +[source,bash] +---- +POST /logs-system.auth/_rollover/ +---- + +[discrete] +[id="data-streams-scenario1-step4"] +== Step 4: Repeat these steps for the metrics data streams + +You've now applied a custom index lifecycle policy to all of the `logs-*` data streams in the `System` integration. +For the metrics data streams, you can repeat steps 2 and 3, using a `metrics-*` index template and the `metrics@custom` component template. + + + +[[data-streams-scenario2]] +== Scenario 2: Apply an ILM policy to specific data streams generated from Fleet integrations across all namespaces + +++++ +Scenario 2: Selected data streams in all namespaces +++++ + +Mappings and settings for data streams can be customized through the creation of `*@custom` component templates, +which are referenced by the index templates created by the {es} apm-data plugin. +The easiest way to configure a custom index lifecycle policy per data stream is to edit this template. + +This tutorial explains how to apply a custom index lifecycle policy to the `logs-system.auth` data stream. + +[discrete] +[id="data-streams-scenario2-step1"] +== Step 1: Create an index lifecycle policy -This tutorial explains how to apply a custom {ilm-init} policy to an integration's data stream. +. To open **Lifecycle Policies**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field]. +. Click **Create policy**. -**Scenario:** You have {agent}s collecting system metrics with the System integration in two environments--one with the namespace `development`, and one with `production`. +Name your new policy. +For this tutorial, you can use `my-ilm-policy`. +Customize the policy to your liking, and when you're done, click **Save policy**. + +[discrete] +[id="data-streams-scenario2-step2"] +== Step 2: View index templates + +The **Index Templates** view in {kib} shows you all of the index templates available to automatically apply settings, mappings, and aliases to indices: + +. To open **Index Management**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field]. +. Select **Index Templates**. +. Search for `system` to see all index templates associated with the System integration. +. Select the index template that matches the data stream for which you want to set up an ILM policy. For this example, you can select the `logs-system.auth` index template. ++ +[role="screenshot"] +image::images/index-template-system-auth.png[List of component templates available for the logs-system.auth index template] + +. In the **Summary**, select `logs-system.auth@custom` from the list to view the component template properties. +. For a newly added integration, the component template won't exist yet. +Select **Create component template** to create it. +If the component template already exists, click **Manage** to update it. +.. On the **Logistics** page, keep all defaults and click **Next**. +.. On the **Index settings** page, in the **Index settings** field, specify the ILM policy that you created. For example: ++ +[source,json] +---- +{ + "index": { + "lifecycle": { + "name": "my-ilm-policy" + } + } +} +---- + +.. Click **Next**. +.. For both the **Mappings** and **Aliases** pages, keep all defaults and click **Next**. +.. Finally, on the **Review** page, review the summary and request. If everything looks good, select **Create component template**. ++ +[role="screenshot"] +image::images/review-component-template02.png[Review details for the new component template] + +[discrete] +[id="data-streams-scenario2-step3"] +== Step 3: Roll over the data streams (optional) + +To confirm that the index template is using the `logs@custom` component template with your custom ILM policy: + +. Reopen the **Index Management** page and open the **Component Templates** tab. +. Search for `system` and select the `logs-system.auth@custom` component template. +. The **Summary** shows the list of all data streams that use the component template, and the **Settings** view shows your newly configured ILM policy. + +New ILM policies only take effect when new indices are created, +so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), +or force a rollover of the data stream using the {ref}/indices-rollover-index.html[{es} rollover API: + +[source,bash] +---- +POST /logs-system.auth/_rollover/ +---- + +[discrete] +[id="data-streams-scenario2-step4"] +== Step 4: Repeat these steps for other data streams + +You've now applied a custom index lifecycle policy to the `logs-system.auth` data stream in the `System` integration. +Repeat these steps for any other data streams for which you'd like to configure a custom ILM policy. + + + +[[data-streams-scenario3]] +== Scenario 3: Apply an ILM policy with integrations using multiple namespaces + +++++ +Scenario 3: Selected integrations and namespaces +++++ + +In this scenario, you have {agent}s collecting system metrics with the System integration in two environments--one with the namespace `development`, and one with `production`. **Goal:** Customize the {ilm-init} policy for the `system.network` data stream in the `production` namespace. Specifically, apply the built-in `90-days-default` {ilm-init} policy so that data is deleted after 90 days. -NOTE: If you cloned an index template to customize the data retention policy on an {es} version prior to 8.13, you must update the index template in the clone to use the `ecs@mappings` component template on {es} version 8.13 or later. See <> for the step-by-step instructions. +[NOTE] +==== +* This scenario involves cloning an index template. We strongly recommend repeating this procedure on every minor {stack} upgrade in order to avoid missing any possible changes to the structure of the managed index template(s) that are shipped with integrations. +* If you cloned an index template to customize the data retention policy on an {es} version prior to 8.13, you must update the index template in the clone to use the `ecs@mappings` component template on {es} version 8.13 or later. See <> for the step-by-step instructions. +==== [discrete] [[data-streams-ilm-one]] @@ -365,7 +596,14 @@ Now that you've created a component template, you need to create an index template to apply the changes to the correct data stream. The easiest way to do this is to duplicate and modify the integration's existing index template. -WARNING: When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading. +[WARNING] +==== +Please note the following: +* When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading. Cloning the index template of an integration package involves some risk as any changes made to the original index template when it is upgraded will not be propagated to the cloned version. +* These steps assume that you want to have a namespace specific ILM policy, which requires index template cloning. Cloning the index template of an integration package involves some risk because any changes made to the original index template as part of package upgrades are not propagated to the cloned version. See <> for details. ++ +If you want to change the ILM Policy, the number of shards, or other settings for the datastreams of one or more integrations, but **the changes do not need to be specific to a given namespace**, it's strongly recommended to use a `@custom` component template, as described in <> and <>, so as to avoid the problems mentioned above. See the <> section for details. +==== . Navigate to **{stack-manage-app}** > **Index Management** > **Index Templates**. . Find the index template you want to clone. The index template will have the `` and `` in its name, @@ -388,7 +626,7 @@ image::images/create-index-template.png[Create index template] == Step 4: Roll over the data stream (optional) To confirm that the data stream is now using the new index template and {ilm-init} policy, -you can either repeat <>, or navigate to **{dev-tools-app}** and run the following: +you can either repeat Step 1, or navigate to **{dev-tools-app}** and run the following: [source,bash] ---- @@ -686,22 +924,6 @@ You can modify your pipeline API request as needed to apply custom processing at Refer to <> to learn more. - - - - - - - - - - - - - - - - [[data-streams-advanced-features]] == Enabling and disabling advanced indexing features for {fleet}-managed data streams @@ -715,6 +937,7 @@ Refer to <> to learn more. * link:{ref}/mapping-source-field.html#synthetic-source[Synthetic `_source`] These features can be enabled and disabled for {fleet}-managed data streams by using the index template API and a few key settings. +Note that in versions 8.17.0 and later, Synthetic `_source` requires an Enterprise license. NOTE: If you are already making use of `@custom` component templates for ingest or retention customization (as shown for example in <>), exercise care to ensure you don't overwrite your customizations when making these requests. @@ -876,16 +1099,18 @@ For example, the following payload disables TSDS on `nginx.stubstatus`: [source,json] ---- PUT _component_template/@custom - { - "template": { - "mappings": { - "_source": { - "mode": "synthetic" + "settings": { + "index": { + "mapping": { + "source": { + "mode": "synthetic" + } } } } } + ---- [discrete] @@ -895,11 +1120,12 @@ PUT _component_template/@custom [source,json] ---- PUT _component_template/@custom - { - "template": { - "mappings": { - "_source": {} + "settings": { + "index": { + "mapping": { + "source": {"mode": "stored"} + } } } } diff --git a/docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc b/docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc index a3c564635..7ba89744f 100644 --- a/docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc +++ b/docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc @@ -73,12 +73,12 @@ Inside the downloaded manifest, update the Daemonset resource: ------------------------------------------------ containers: - name: elastic-agent - image: docker.elastic.co/beats/elastic-agent: + image: docker.elastic.co/elastic-agent/elastic-agent: args: ["-c", "/etc/elastic-agent/agent.yml", "-e"] ------------------------------------------------ Notes:: -The is just a placeholder for the elastic-agent image version that you will download in your manifest: eg. `image: docker.elastic.co/beats/elastic-agent: 8.11.0` +The is just a placeholder for the elastic-agent image version that you will download in your manifest: eg. `image: docker.elastic.co/elastic-agent/elastic-agent: 8.11.0` Important thing is to update your manifest with args details [source,yaml] diff --git a/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/elastic-agent-kubernetes-autodiscovery.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/elastic-agent-kubernetes-autodiscovery.asciidoc index 5252cc4f5..d46f81e44 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/elastic-agent-kubernetes-autodiscovery.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/elastic-agent-kubernetes-autodiscovery.asciidoc @@ -3,9 +3,9 @@ When you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running. -To use autodiscovery, you will need to modify the manifest file of the {agent}. Refer to <> to learn how to retrieve and configure it. +To use autodiscover, you will need to modify the manifest file of the {agent}. Refer to <> to learn how to retrieve and configure it. -There are two different ways to use autodiscovery: +There are two different ways to use autodiscover: * <> @@ -13,13 +13,13 @@ There are two different ways to use autodiscovery: [discrete] -== How to configure autodiscovery +== How to configure autodiscover -`Conditions Based Autodiscovery` is more suitable for scenarios when users know the different group of containers they want to monitor in advance. It is advisable to choose conditions-based configuration when administrators can configure specific conditions that match their needs. Conditions are supported in both Managed and Standalone {agent}. +`Conditions Based Autodiscover` is more suitable for scenarios when users know the different group of containers they want to monitor in advance. It is advisable to choose conditions-based configuration when administrators can configure specific conditions that match their needs. Conditions are supported in both Managed and Standalone {agent}. -`Hints Based Autodiscovry` is suitable for more generic scenarios, especially when users don't know the exact configuration of the system to monitor and can not create in advance conditions. Additionally a big advantage of Hints Autodiscovery is the ability to offer dynamic configuration of inputs based on annotations from Pods/Containers. If dynamic configuration is needed, then Hints should be enabled. Hints are supported only in Standalone {agent} mode. +`Hints Based Autodiscover` is suitable for more generic scenarios, especially when users don't know the exact configuration of the system to monitor and can not create in advance conditions. Additionally a big advantage of Hints Autodiscover is the ability to offer dynamic configuration of inputs based on annotations from Pods/Containers. If dynamic configuration is needed, then Hints should be enabled. Hints are supported only in Standalone {agent} mode. -*Best Practises when you configure autodiscovery:* +*Best Practises when you configure autodiscover:* - Always define alternatives and default values to your variables that are used in conditions or [hint templates](eg. See `auth.basic` set as `auth.basic.user: ${kubernetes.hints.nginx.access.username|kubernetes.hints.nginx.username|''}`` in [nginx.yml](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/nginx.yml#L8)) @@ -27,4 +27,4 @@ IMPORTANT: When an input uses a variable substitution that is not present in the - To debug configurations that include variable substitution and conditions, use the inspect command of {agent}. (See more information in <> in *Debugging* Section) -- In Condition Based autodiscovery is advisable to define a generic last condition that will act as your default condition and will be validated when all others fail or don't apply. If applicable, such conditions might help to identify processing and troubleshoot possible problems. +- In Condition Based autodiscover is advisable to define a generic last condition that will act as your default condition and will be validated when all others fail or don't apply. If applicable, such conditions might help to identify processing and troubleshoot possible problems. diff --git a/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/kubernetes-hints-autodiscover.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/kubernetes-hints-autodiscover.asciidoc index 27ef3b057..1adc1ae89 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/kubernetes-hints-autodiscover.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/autodiscovery/kubernetes-hints-autodiscover.asciidoc @@ -182,7 +182,7 @@ An init container is required to download the hints templates. ---- initContainers: - name: k8s-templates-downloader - image: docker.elastic.co/beats/elastic-agent:{branch} + image: docker.elastic.co/elastic-agent/elastic-agent:{branch} command: ['bash'] args: - -c diff --git a/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-standalone-logging.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-standalone-logging.asciidoc index fbfe68524..fbf5debd3 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-standalone-logging.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-standalone-logging.asciidoc @@ -141,7 +141,7 @@ Default: `true` | `agent.logging.event_data.path` | The directory that log files is written to. -include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/logging-widget.asciidoc[] +include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/event-logging-widget.asciidoc[] Logs file names end with a date and optional number: log-date.ndjson, log-date-1.ndjson, and so on as new files are created during rotation. diff --git a/docs/en/ingest-management/elastic-agent/configuration/env/container-envs.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/env/container-envs.asciidoc index 74d9e4cda..1d7115a21 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/env/container-envs.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/env/container-envs.asciidoc @@ -66,8 +66,6 @@ Settings used to prepare the {fleet} plugin in {kib}. |=== | Settings | Description -include::shared-env.asciidoc[tag=kibana-fleet-setup] - include::shared-env.asciidoc[tag=kibana-fleet-host] include::shared-env.asciidoc[tag=kibana-fleet-username] @@ -98,6 +96,12 @@ include::shared-env.asciidoc[tag=fleet-server-elasticsearch-host] include::shared-env.asciidoc[tag=fleet-server-elasticsearch-ca] +include::shared-env.asciidoc[tag=fleet-server-es-cert] + +include::shared-env.asciidoc[tag=fleet-server-es-cert-key] + +include::shared-env.asciidoc[tag=fleet-server-insecure-http] + include::shared-env.asciidoc[tag=fleet-server-service-token] include::shared-env.asciidoc[tag=fleet-server-service-token-path] @@ -116,8 +120,14 @@ include::shared-env.asciidoc[tag=fleet-server-cert-key] include::shared-env.asciidoc[tag=fleet-server-cert-key-passphrase] +include::shared-env.asciidoc[tag=fleet-server-client-auth] + include::shared-env.asciidoc[tag=fleet-server-es-ca-trusted-fingerprint] +include::shared-env.asciidoc[tag=fleet-daemon-timeout] + +include::shared-env.asciidoc[tag=fleet-server-timeout] + |=== [discrete] @@ -133,10 +143,18 @@ Settings used to enroll an {agent} into a {fleet-server}. |=== | Settings | Description +include::shared-env.asciidoc[tag=elastic-agent-cert] + +include::shared-env.asciidoc[tag=elastic-agent-cert-key] + +include::shared-env.asciidoc[tag=elastic-agent-cert-key-passphrase] + include::shared-env.asciidoc[tag=elastic-agent-tag] include::shared-env.asciidoc[tag=fleet-enroll] +include::shared-env.asciidoc[tag=fleet-force] + include::shared-env.asciidoc[tag=fleet-url] include::shared-env.asciidoc[tag=fleet-enrollment-token] diff --git a/docs/en/ingest-management/elastic-agent/configuration/env/shared-env.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/env/shared-env.asciidoc index 3a6174dfb..6548b66a9 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/env/shared-env.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/env/shared-env.asciidoc @@ -17,30 +17,49 @@ OPTIONAL INFO AND EXAMPLE // ============================================================================= -// tag::elastic-agent-tag[] +// tag::elastic-agent-cert[] | -[id="env-{type}-elastic-agent-tag"] -`ELASTIC_AGENT_TAGS` +[id="env-{type}-elastic-agent-cert"] +`ELASTIC_AGENT_CERT` -| (string) A comma-separated list of tags to apply to {fleet}-managed {agent}s. -You can use these tags to filter the list of agents in {fleet}. +| (string) The path to the mutual TLS client certificate that {agent} will use to connect to {fleet-server}. -// end::elastic-agent-tag[] +// end::elastic-agent-cert[] // ============================================================================= -// tag::kibana-fleet-setup[] +// tag::elastic-agent-cert-key[] | -[id="env-{type}-kibana-fleet-setup"] -`KIBANA_FLEET_SETUP` +[id="env-{type}-elastic-agent-cert-key"] +`ELASTIC_AGENT_CERT_KEY` -| (int) Set to `1` to enable {fleet} setup. -Enabling {fleet} is required before {fleet-server} will start. -When this action is not performed, a user must manually log in to {kib} and visit the {fleet} page to enable setup. +| (string) The path to the mutual TLS private key that {agent} will use to connect to {fleet-server}. -*Default:* none +// end::elastic-agent-cert-key[] + +// tag::elastic-agent-cert-key-passphrase[] +| +[id="env-{type}-elastic-agent-cert-key-passphrase"] +`ELASTIC_AGENT_CERT_KEY_PASSPHRASE` + +| (string) The path to the file that contains the passphrase for the mutual TLS private key that {agent} will use to connect to {fleet-server}. +The file must only contain the characters of the passphrase, no newline or extra non-printing characters. + +This option is only used if the `--elastic-agent-cert-key` is encrypted and requires a passphrase to use. + +// end::elastic-agent-cert-key-passphrase[] + +// ============================================================================= + +// tag::elastic-agent-tag[] +| +[id="env-{type}-elastic-agent-tag"] +`ELASTIC_AGENT_TAGS` -// end::kibana-fleet-setup[] +| (string) A comma-separated list of tags to apply to {fleet}-managed {agent}s. +You can use these tags to filter the list of agents in {fleet}. + +// end::elastic-agent-tag[] // ============================================================================= @@ -105,6 +124,20 @@ contains your CA's certificate. // ============================================================================= +// tag::fleet-force[] +| +[id="env-{type}-fleet-force"] +`FLEET_FORCE` + +| (bool) Set to `true` to force overwrite of the current {agent} configuration without prompting for confirmation. +This flag is helpful when using automation software or scripted deployments. + +*Default:* `false` + +// end::fleet-force[] + +// ============================================================================= + // tag::fleet-server-enable[] | [id="env-{type}-fleet-server-enable"] @@ -273,6 +306,21 @@ Overrides the port defined in the policy. // ============================================================================= +// tag::fleet-server-client-auth[] +| +[id="env-{type}-fleet-server-client-auth"] +`FLEET_SERVER_CLIENT_AUTH` + +| (string) One of `none`, `optional`, or `required`. +{fleet-server}'s client authentication option for client mTLS connections. +If `optional` or `required` is specified, client certificates are verified using CAs. + +*Default:* `none` + +// end::fleet-server-client-auth[] + +// ============================================================================= + // tag::fleet-server-es-ca-trusted-fingerprint[] | [id="env-{type}-fleet-server-es-ca-trusted-fingerprint"] @@ -288,6 +336,68 @@ by {agent} for communication. This flag is required when using self-signed certi // ============================================================================= +// tag::fleet-server-es-cert[] +| +[id="env-{type}-fleet-server-es-cert"] +`FLEET_SERVER_ES_CERT` + +| (string) The path to the mutual TLS client certificate that {fleet-server} will use to connect to {es}. + +*Default:* `""` + +// end::fleet-server-es-cert[] + +// ============================================================================= + +// tag::fleet-server-es-cert-key[] +| +[id="env-{type}-fleet-server-es-cert-key"] +`FLEET_SERVER_ES_CERT_KEY` + +| (string) The path to the mutual TLS private key that {fleet-server} will use to connect to {es}. + +*Default:* `""` + +// end::fleet-server-es-cert-key[] + +// ============================================================================= + +// tag::fleet-server-insecure-http[] +| +[id="env-{type}-fleet-server-insecure-http"] +`FLEET_SERVER_INSECURE_HTTP` + +| (bool) When `true`, {fleet-server} is exposed over insecure or unverified HTTP. +Setting this to `true` is not recommended. + +*Default:* `false` + +// end::fleet-server-insecure-http[] + +// ============================================================================= + +// tag::fleet-daemon-timeout[] +| +[id="env-{type}-fleet-daemon-timeout"] +`FLEET_DAEMON_TIMEOUT` + +| (duration) Set to indicate how long {fleet-server} will wait during the bootstrap process for {elastic-agent}. + +// end::fleet-daemon-timeout[] + +// ============================================================================= + +// tag::fleet-server-timeout[] +| +[id="env-{type}-fleet-server-timeout"] +`FLEET_SERVER_TIMEOUT` + +| (duration) Set to indicate how long {agent} will wait for {fleet-server} to check in as healthy. + +// end::fleet-server-timeout[] + +// ============================================================================= + // tag::fleet-enroll[] | [id="env-{type}-fleet-enroll"] diff --git a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc index 7eea9a6d8..b1967e13d 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc @@ -102,6 +102,8 @@ outputs: <1> In this example, the {es} nodes are available at `https://10.45.3.2:9220/elasticsearch` and `https://10.45.3.1:9230/elasticsearch`. + +Note that Elasticsearch Nodes in the link:{serverless-docs}[{serverless-full}] environment are exposed on port 443. // end::hosts-setting[] // ============================================================================= diff --git a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc index 839a2c043..c8c7c605d 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc @@ -44,6 +44,31 @@ outputs: verification_mode: full ---- +== Kafka output and using {ls} to index data to {es} + +If you are considering using {ls} to ship the data from `kafka` to {es}, please be aware the +structure of the documents sent from {agent} to `kafka` must not be modified by {ls}. +We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec in order +to make sure the input doesn't edit the fields and their contents. + +The data streams set up by the integrations expect to receive events having the same structure and +field names as they were sent directly from an {agent}. + +Refer to <> documentation for more details. + +[source,yaml] +---- +inputs { + kafka { + ... + ecs_compatibility => "disabled" + codec => json { ecs_compatibility => "disabled" } + ... + } +} +... +---- + == Kafka output configuration settings The `kafka` output supports the following settings, grouped by category. @@ -502,4 +527,4 @@ Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silentl // ============================================================================= -|=== \ No newline at end of file +|=== diff --git a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc index aec36d059..d64c76d3c 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc @@ -32,8 +32,14 @@ To receive the events in {ls}, you also need to create a {ls} configuration pipe The {ls} configuration pipeline listens for incoming {agent} connections, processes received events, and then sends the events to {es}. -The following example configures a {ls} pipeline that listens on port `5044` for -incoming {agent} connections and routes received events to {es}: +Please be aware that the structure of the documents sent from {agent} to {ls} must not be modified by the pipeline. +We recommend that the pipeline doesn’t edit or remove the fields and their contents. +Editing the structure of the documents coming from {agent} can prevent the {es} ingest pipelines associated to the integrations in use to work correctly. +We cannot guarantee that the {es} ingest pipelines associated to the integrations using {agent} can work with missing or modified fields. + +This {ls} pipeline definition example configures a pipeline that listens on port `5044` for +incoming {agent} connections and routes received events to {es}. + [source,yaml] ---- @@ -41,19 +47,28 @@ input { elastic_agent { port => 5044 enrich => none # don't modify the events' schema at all - # or minimal change, add only ssl and source metadata - # enrich => [ssl_peer_metadata, source_metadata] + ssl => true + ssl_certificate_authorities => [""] + ssl_certificate => "" + ssl_key => "" + ssl_verify_mode => "force_peer" } } output { elasticsearch { hosts => ["http://localhost:9200"] <1> + # cloud_id => "..." data_stream => "true" + api_key => "" <2> + data_stream => true + ssl => true + # cacert => "" } } ---- <1> The {es} server and the port (`9200`) where {es} is running. +<2> The API Key used by {ls} to ship data to the destination data streams. For more information about configuring {ls}, refer to {logstash-ref}/configuration.html[Configuring {ls}] and diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/elastic-agent-providers.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/elastic-agent-providers.asciidoc index 9bedeb67b..396682c44 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/elastic-agent-providers.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/elastic-agent-providers.asciidoc @@ -35,8 +35,9 @@ providers: - item: key2 ---- -Explicitly disable a provider by setting `enabled: false`. All providers -are prefixed without name collisions. The name of the provider is in the key in the configuration. +Providers are enabled automatically if a provider is referenced in an {agent} policy. +All providers are prefixed without name collisions. +The name of the provider is in the key in the configuration. [source,yaml] ---- @@ -92,9 +93,11 @@ configuration. [[disable-providers-by-default]] === Disabling Providers By Default -All registered providers are enabled by default. +All registered providers are disabled by default until they are referenced in a policy. -Disable all providers by default and only enable explicitly configured providers by setting `agent.providers.initial_default: false`. +You can disable all providers even if they are referenced in a policy by setting `agent.providers.initial_default: false`. + +The following configuration disables all providers from running except for the docker provider, if it becomes referenced in the policy: [source,yaml] ---- diff --git a/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.asciidoc index 77e468a77..0b08d36dd 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.asciidoc @@ -4,7 +4,10 @@ The {agent} installation includes an `elastic-agent.reference.yml` file that describes all the settings available in a standalone configuration. -This file is included here for your convenience. +To ensure that you're accessing the latest version, refer to the original link:https://github.com/elastic/elastic-agent/blob/main/elastic-agent.reference.yml[`elastic-agent.reference.yml` file] in the `elastic/elastic-agent` repository. +A copy is included here for your convenience. + +Each section of the file and available settings are also described in <>. [source,yaml] ---- diff --git a/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.yml b/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.yml index 646a10d19..392146289 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.yml +++ b/docs/en/ingest-management/elastic-agent/configuration/yaml/elastic-agent-reference-yaml.yml @@ -11,30 +11,51 @@ outputs: default: type: elasticsearch hosts: [127.0.0.1:9200] - api-key: "example-key" + api_key: "example-key" # username: "elastic" # password: "changeme" -inputs: - # Each input in a policy must have a unique value for the id key. - - id: unique-system-metrics-id - type: system/metrics + # Performance preset for elasticsearch outputs. One of "balanced", "throughput", + # "scale", "latency" and "custom". + # The default if unspecified is "custom". + preset: balanced +inputs: + - type: system/metrics + # Each input must have a unique ID. + id: unique-system-metrics-input # Namespace name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes # For index naming restrictions, see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params data_stream.namespace: default use_output: default streams: - - metricset: cpu + - metricsets: + - cpu # Dataset name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes data_stream.dataset: system.cpu - - metricset: memory + - metricsets: + - memory data_stream.dataset: system.memory - - metricset: network + - metricsets: + - network data_stream.dataset: system.network - - metricset: filesystem + - metricsets: + - filesystem data_stream.dataset: system.filesystem +# # Collecting log files +# - type: filestream +# # Input ID allowing Elastic Agent to track the state of this input. Must be unique. +# id: your-input-id +# streams: +# # Stream ID for this data stream allowing Filebeat to track the state of the ingested files. Must be unique. +# # Each filestream data stream creates a separate instance of the Filebeat filestream input. +# - id: your-filestream-stream-id +# data_stream: +# dataset: generic +# paths: +# - /var/log/*.log + # management: # # Mode of management, the Elastic Agent support two modes of operation: # # @@ -72,12 +93,12 @@ inputs: # target_directory: "${path.data}/downloads" # # timeout for downloading package # timeout: 120s -# # file path to a public key used for verifying downloaded artifacts -# # if not file is present agent will try to load public key from elastic.co website. -# pgpfile: "${path.data}/elastic.pgp" # # install_path describes the location of installed packages/programs. It is also used # # for reading program specifications. # install_path: "${path.data}/install" +# # retry_sleep_init_duration is the duration to sleep for before the first retry attempt. This +# # duration will increase for subsequent retry attempts in a randomized exponential backoff manner. +# retry_sleep_init_duration: 30s # agent.process: # # timeout for creating new processes. when process is not successfully created by this timeout @@ -92,6 +113,9 @@ inputs: # address: localhost # # port for the GRPC server that spawned processes connect back to. # port: 6789 +# # max_message_size limits the message size in agent internal communication +# # default is 100MB +# max_message_size: 104857600 # agent.retry: # # Enabled determines whether retry is possible. Default is false. @@ -108,6 +132,12 @@ inputs: # # Default is false # exponential: false +# agent.limits: +# # limits the number of operating system threads that can execute user-level Go code simultaneously. +# # Translates into the GOMAXPROCS runtime parameter for each Go process started by the agent and the agent itself. +# # By default is set to `0` which means using all available CPUs. +# go_max_procs: 0 + # agent.monitoring: # # enabled turns on monitoring of running processes # enabled: false @@ -115,10 +145,27 @@ inputs: # logs: false # # enables metrics monitoring # metrics: false +# # metrics_period defines how frequent we should sample monitoring metrics. Default is 60 seconds. +# metrics_period: 60s # # exposes /debug/pprof/ endpoints # # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost # pprof.enabled: false -# # exposes agent metrics using http, by default sockets and named pipes are used +# # The name of the output to use for monitoring data. +# use_output: monitoring +# # Exposes agent metrics using http, by default sockets and named pipes are used. +# # +# # `http` Also exposes a /liveness endpoint that will return an HTTP code depending on agent status: +# # 200: Agent is healthy +# # 500: A component or unit is in a failed state +# # 503: The agent coordinator is unresponsive +# # +# # You can pass a `failon` parameter to the /liveness endpoint to determine what component state will result in a 500. +# # For example: `curl 'localhost:6792/liveness?failon=degraded'` will return 500 if a component is in a degraded state. +# # The possible values for `failon` are: +# # `degraded`: return an error if a component is in a degraded state or failed state, or if the agent coordinator is unresponsive. +# # `failed`: return an error if a unit is in a failed state, or if the agent coordinator is unresponsive. +# # `heartbeat`: return an error only if the agent coordinator is unresponsive. +# # If no `failon` parameter is provided, the default behavior is `failon=heartbeat` # http: # # enables http endpoint # enabled: false @@ -129,6 +176,22 @@ inputs: # port: 6791 # # Metrics buffer endpoint # buffer.enabled: false +# # Configuration for the diagnostics action handler +# diagnostics: +# # Rate limit for the action handler. Does not affect diagnostics collected through the CLI. +# limit: +# # Rate limit interval. +# interval: 1m +# # Rate limit burst. +# burst: 1 +# # Configuration for the file-upload client. Client may retry failed requests with an exponential backoff. +# uploader: +# # Max retries allowed when uploading a chunk. +# max_retries: 10 +# # Initial duration of the backoff. +# init_dur: 1s +# # Max duration of the backoff. +# max_dur: 1m # # Allow fleet to reload its configuration locally on disk. # # Notes: Only specific process configuration and external input configurations will be reloaded. @@ -141,6 +204,13 @@ inputs: # # period define how frequent we should look for changes in the configuration. # period: 10s +# Feature Flags + +# This section enables or disables feature flags supported by Agent and its components. +#agent.features: +# fqdn: +# enabled: false + # Logging # There are four options for the log output: file, stderr, syslog, eventlog @@ -187,7 +257,7 @@ agent.logging.to_stderr: true # Configure log file size limit. If limit is reached, log file will be # automatically rotated - #rotateeverybytes: 10485760 # = 10MB + #rotateeverybytes: 20971520 # = 20MB # Number of rotated log files to keep. Oldest files will be deleted first. #keepfiles: 7 @@ -210,12 +280,65 @@ agent.logging.to_stderr: true # Set to true to log messages in JSON format. #agent.logging.json: false +#=============================== Events Logging =============================== +# Some outputs will log raw events on errors like indexing errors in the +# Elasticsearch output, to prevent logging raw events (that may contain +# sensitive information) together with other log messages, a different +# log file, only for log entries containing raw events, is used. It will +# use the same level, selectors and all other configurations from the +# default logger, but it will have it's own file configuration. +# +# Having a different log file for raw events also prevents event data +# from drowning out the regular log files. +# +# IMPORTANT: No matter the default logger output configuration, raw events +# will **always** be logged to a file configured by `agent.logging.event_data.files`. + +# agent.logging.event_data: +# Logging to rotating files. Set agent.logging.to_files to false to disable logging to +# files. +#agent.logging.event_data.to_files: true +#agent.logging.event_data: + # Configure the path where the logs are written. The default is the logs directory + # under the home path (the binary location). + #path: /var/log/filebeat + + # The name of the files where the logs are written to. + #name: filebeat-event-data + + # Configure log file size limit. If the limit is reached, log file will be + # automatically rotated. + #rotateeverybytes: 5242880 # = 5MB + + # Number of rotated log files to keep. The oldest files will be deleted first. + #keepfiles: 2 + + # The permissions mask to apply when rotating log files. The default value is 0600. + # Must be a valid Unix-style file permissions mask expressed in octal notation. + #permissions: 0600 + + # Enable log file rotation on time intervals in addition to the size-based rotation. + # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h + # are boundary-aligned with minutes, hours, days, weeks, months, and years as + # reported by the local system clock. All other intervals are calculated from the + # Unix epoch. Defaults to disabled. + #interval: 0 + + # Rotate existing logs on startup rather than appending them to the existing + # file. Defaults to false. + # rotateonstartup: false + # Providers # Providers supply the key/values pairs that are used for variable substitution # and conditionals. Each provider's keys are automatically prefixed with the name # of the provider. +# All registered providers are enabled by default. + +# Disable all providers by default and only enable explicitly configured providers. +# agent.providers.initial_default: false + #providers: # Agent provides information about the running agent. diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent-container.asciidoc b/docs/en/ingest-management/elastic-agent/elastic-agent-container.asciidoc index da0489754..8d7a45b66 100644 --- a/docs/en/ingest-management/elastic-agent/elastic-agent-container.asciidoc +++ b/docs/en/ingest-management/elastic-agent/elastic-agent-container.asciidoc @@ -1,7 +1,9 @@ [[elastic-agent-container]] = Run {agent} in a container -You can run {agent} inside a container -- either with {fleet-server} or standalone. Docker images for all versions of {agent} are available from the https://www.docker.elastic.co/r/beats/elastic-agent[Elastic Docker registry]. If you are running in Kubernetes, refer to {eck-ref}/k8s-elastic-agent.html[run {agent} on ECK]. +You can run {agent} inside a container -- either with {fleet-server} or standalone. Docker images for all versions of {agent} are available from the https://www.docker.elastic.co/r/elastic-agent/elastic-agent[Elastic Docker registry]. If you are running in Kubernetes, refer to {eck-ref}/k8s-elastic-agent.html[run {agent} on ECK]. + +Note that running {elastic-agent} in a container is supported only in Linux environments. For this reason we don't currently provide {agent} container images for Windows. Considerations: @@ -31,14 +33,32 @@ Run the `docker pull` command against the Elastic Docker registry: [source,terminal,subs="attributes"] ---- -docker pull docker.elastic.co/beats/elastic-agent:{version} +docker pull docker.elastic.co/elastic-agent/elastic-agent:{version} +---- + +Alternately, you can use the hardened link:https://wolfi.dev/[Wolfi] image. +Using Wolfi images requires Docker version 20.10.10 or later. +For details about why the Wolfi images have been introduced, refer to our article +link:https://www.elastic.co/blog/reducing-cves-in-elastic-container-images[Reducing CVEs in Elastic container images]. + + +[source,terminal,subs="attributes"] +---- +docker pull docker.elastic.co/elastic-agent/elastic-agent-wolfi:{version} ---- If you want to run Synthetics tests, run the `docker pull` command to fetch the *elastic-agent-complete* image: [source,terminal,subs="attributes"] ---- -docker pull docker.elastic.co/beats/elastic-agent-complete:{version} +docker pull docker.elastic.co/elastic-agent/elastic-agent-complete:{version} +---- + +To run Synthetics tests using the hardened link:https://wolfi.dev/[Wolfi] image, run: + +[source,terminal,subs="attributes"] +---- +docker pull docker.elastic.co/elastic-agent/elastic-agent-complete-wolfi:{version} ---- [discrete] @@ -54,7 +74,7 @@ Run the following commands to verify the *elastic-agent* container image signatu ["source","sh",subs="attributes"] -------------------------------------------- wget https://artifacts.elastic.co/cosign.pub <1> -cosign verify --key cosign.pub docker.elastic.co/beats/elastic-agent:{version} <2> +cosign verify --key cosign.pub docker.elastic.co/elastic-agent/elastic-agent:{version} <2> -------------------------------------------- <1> Download the Elastic public key to verify container signature <2> Verify the container against the Elastic public key @@ -64,14 +84,14 @@ If you're using the *elastic-agent-complete* image, run the commands as follows: ["source","sh",subs="attributes"] -------------------------------------------- wget https://artifacts.elastic.co/cosign.pub <1> -cosign verify --key cosign.pub docker.elastic.co/beats/elastic-agent-complete:{version} <2> +cosign verify --key cosign.pub docker.elastic.co/elastic-agent/elastic-agent-complete:{version} <2> -------------------------------------------- The command prints the check results and the signature payload in JSON format, for example: ["source","sh",subs="attributes"] -------------------------------------------- -Verification for docker.elastic.co/beats/elastic-agent-complete:{version} -- +Verification for docker.elastic.co/elastic-agent/elastic-agent-complete:{version} -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline @@ -87,7 +107,7 @@ To see the full list, run: [source,terminal,subs="attributes"] ---- -docker run --rm docker.elastic.co/beats/elastic-agent:{version} elastic-agent container -h +docker run --rm docker.elastic.co/elastic-agent/elastic-agent:{version} elastic-agent container -h ---- [discrete] @@ -95,6 +115,18 @@ docker run --rm docker.elastic.co/beats/elastic-agent:{version} elastic-agent co include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/run-agent-image/widget.asciidoc[] +If you need to run {fleet-server} as well, adjust the `docker run` command above by adding these environment variables: + +[source,yaml] +---- + --env FLEET_SERVER_ENABLE=true \ <1> + --env FLEET_SERVER_ELASTICSEARCH_HOST= \ <2> + --env FLEET_SERVER_SERVICE_TOKEN= <3> +---- +<1> Set to `true` to bootstrap {fleet-server} on this {agent}. This automatically forces {fleet} enrollment as well. +<2> The Elasticsearch host for Fleet Server to communicate with, for example `http://elasticsearch:9200`. +<3> Service token to use for communication with {es} and {kib}. + [TIP] .Running {agent} on a read-only file system ==== @@ -108,7 +140,7 @@ For example: [source,terminal,subs="attributes"] ---- -docker run --rm --mount source=$(pwd)/state,destination=/state -e {STATE_PATH}=/state --read-only docker.elastic.co/beats/elastic-agent:{version} <1> +docker run --rm --mount source=$(pwd)/state,destination=/state -e {STATE_PATH}=/state --read-only docker.elastic.co/elastic-agent/elastic-agent:{version} <1> ---- Where {STATE_PATH} is the path to a stateful directory to mount where {agent} application data can be stored. @@ -133,7 +165,7 @@ The example below shows how to enroll an {agent}: version: "3" services: elastic-agent: - image: docker.elastic.co/beats/elastic-agent:{version} <1> + image: docker.elastic.co/elastic-agent/elastic-agent:{version} <1> container_name: elastic-agent restart: always user: root # note, synthetic browser monitors require this set to `elastic-agent` diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent-unprivileged-mode.asciidoc b/docs/en/ingest-management/elastic-agent/elastic-agent-unprivileged-mode.asciidoc index ee35d6973..f4ebb351b 100644 --- a/docs/en/ingest-management/elastic-agent/elastic-agent-unprivileged-mode.asciidoc +++ b/docs/en/ingest-management/elastic-agent/elastic-agent-unprivileged-mode.asciidoc @@ -1,8 +1,6 @@ [[elastic-agent-unprivileged]] = Run {agent} without administrative privileges -preview::[] - Beginning with {stack} version 8.15, {agent} is no longer required to be run by a user with superuser privileges. You can now run agents in an `unprivileged` mode that does not require `root` access on Linux or macOS, or `admin` access on Windows. Being able to run agents without full administrative privileges is often a requirement in organizations where this kind of access is often very limited. In general, agents running without full administrative privileges will perform and behave exactly as those run by a superuser. There are certain integrations and datastreams that are not available, however. If an integration requires root access, this is <>. @@ -18,6 +16,7 @@ Refer to <> and <> for the * <> * <> * <> +* <> [discrete] [[unprivileged-running]] @@ -229,3 +228,41 @@ For example: . When you install {agent} with the `--unprivileged` setting, the `elastic-agent-user` user and the `elastic-agent` group are created automatically. . If you then want your user `myuser` to be able to run an {agent} command such as `elastic-agent status`, add the `myuser` user to the `elastic-agent` group. . Then, once added to the group, the `elastic-agent status` command will work. Prior to that, the user `myuser` running the command will result in a permission error that indicates a problem communicating with the control socket. + +[discrete] +[[unprivileged-preexisting-user]] +== Using `unprivileged` mode with a pre-existing user and group + +preview::[] + +In certain cases you may want to install {agent} in `unprivileged` mode, with the agent running as a pre-existing user or as part of a pre-existing group. +For example, on a Windows system you may have a service account in Active Directory and you'd like {agent} to run under that account. + +To install {agent} in `unprivileged` mode as a specific user, add the `--user` and `--password` parameters to the install command: + +[source,shell] +---- +elastic-agent install --unprivileged --user="my.path\username" --password="mypassword" +---- + +To install {agent} in `unprivileged` mode as part of a specific group, add the `--group` and `--password` parameters to the install command: + +[source,shell] +---- +elastic-agent install --unprivileged --group="my.path\groupname" --password="mypassword" +---- + +Alternatively, if you have {agent} already installed with administrative privileges, you can change the agent to use `unprivileged` mode and to run as a specific user or in a specific group. +For example: + +[source,shell] +---- +elastic-agent unprivileged --user="my.path\username" --password="mypassword" +---- + +[source,shell] +---- +elastic-agent unprivileged --group="my.path\groupname" --password="mypassword" +---- + + diff --git a/docs/en/ingest-management/elastic-agent/example-kubernetes-fleet-managed-agent-helm.asciidoc b/docs/en/ingest-management/elastic-agent/example-kubernetes-fleet-managed-agent-helm.asciidoc new file mode 100644 index 000000000..f7f78b9b3 --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/example-kubernetes-fleet-managed-agent-helm.asciidoc @@ -0,0 +1,152 @@ +[[example-kubernetes-fleet-managed-agent-helm]] += Example: Install {fleet}-managed {agent} on {k8s} using Helm + +preview::[] + +This example demonstrates how to install {fleet}-managed {agent} on a {k8s} system using a Helm chart, gather {k8s} metrics and send them to an {es} cluster in {ecloud}, and then view visualizations of those metrics in {kib}. + +For an overview of the {agent} Helm chart and its benefits, refer to <>. + +This guide takes you through these steps: + +* <> +* <> +* <> + + +[discrete] +[[agent-fleet-managed-helm-example-prereqs]] +=== Prerequisites + +To get started, you need: + +* A local install of the link:https://helm.sh/[Helm] {k8s} package manager. +* An link:{ess-trial}[{ecloud}] hosted {es} cluster on version 8.16 or higher. +* An active {k8s} cluster. +* A local clone of the link:https://github.com/elastic/elastic-agent/tree/8.16[elastic/elastic-agent] GitHub repository. Make sure to use the `8.16` branch to ensure that {agent} has full compatibility with the Helm chart. + +[discrete] +[[agent-fleet-managed-helm-example-install-agent]] +=== Install {agent} + +. Open your {ecloud} deployment, and from the navigation menu select **Fleet**. +. From the **Agents** tab, select **Add agent**. +. In the **Add agent** UI, specify a policy name and select **Create policy**. Leave the **Collect system logs and metrics** option selected. +. Scroll down in the **Add agent** flyout to the **Install Elastic Agent on your host** section. +. Select the **Linux TAR** tab and copy the values for `url` and `enrollment-token`. You'll use these when you run the `helm install` command. +. Open a terminal shell and change into a directory in your local clone of the `elastic-agent` repo. +. Copy this command. ++ +[source,sh] +---- +helm install demo ./deploy/helm/elastic-agent \ +--set agent.fleet.enabled=true \ +--set agent.fleet.url= \ +--set agent.fleet.token= \ +--set agent.fleet.preset=perNode +---- ++ +Note that the command has these properties: + +* `helm install` runs the Helm CLI install tool. +* `demo` gives a name to the installed chart. You can choose any name. +* `./deploy/helm/elastic-agent` is a local path to the Helm chart to install (in time it's planned to have a public URL for the chart). +* `--set agent.fleet.enabled=true` enables {fleet}-managed {agent}. The CLI parameter overrides the default `false` value for `agent.fleet.enabled` in the {agent} link:https://github.com/elastic/elastic-agent/blob/main/deploy/helm/elastic-agent/values.yaml[values.yaml] file. +* `--set agent.fleet.url=` sets the address where {agent} will connect to {fleet} in your {ecloud} deployment, over port 443 (again, overriding the value set by default in the {agent} link:https://github.com/elastic/elastic-agent/blob/main/deploy/helm/elastic-agent/values.yaml[values.yaml] file). +* `--set agent.fleet.token=` sets the enrollment token that {agent} uses to authenticate with {fleet}. +* `--set agent.fleet.preset=perNode` enables {k8s} metrics on `per node` basis. You can alternatively set cluster wide metrics (`clusterWide`) or kube-state-metrics (`ksmSharded`). ++ +-- +TIP: For a full list of all available YAML settings and descriptions, refer to the link:https://github.com/elastic/elastic-agent/tree/main/deploy/helm/elastic-agent[{agent} Helm Chart Readme]. +-- +. Update the command to replace: +.. `` with the URL that you copied earlier. +.. `` with the enrollment token that you copied earlier. ++ +After your updates, the command should look something like this: ++ +[source,sh] +---- +helm install demo ./deploy/helm/elastic-agent \ +--set agent.fleet.enabled=true \ +--set agent.fleet.url=https://256575858845283fxxxxxxxd5265d2b4.fleet.us-central1.gcp.foundit.no:443 \ +--set agent.fleet.token=eSVvFDUvSUNPFldFdhhZNFwvS5xxxxxxxxxxxxFEWB1eFF1YedUQ1NWFXwr== \ +--set agent.fleet.preset=perNode +---- + +. Run the command. ++ +The command output should confirm that {agent} has been installed: ++ +[source,sh] +---- +... +Installed agent: + - perNode [daemonset - managed mode] +... +---- + +. Run the `kubectl get pods -n default` command to confirm that the {agent} pod is running: ++ +[source,sh] +---- +NAME READY STATUS RESTARTS AGE +agent-pernode-demo-86mst 1/1 Running 0 12s +---- + +. In the **Add agent** flyout, wait a minute or so for confirmation that {agent} has successfully enrolled with {fleet} and that data is flowing: ++ +[role="screenshot"] +image::images/helm-example-nodes-enrollment-confirmation.png[Screen capture of Add Agent UI showing that the agent has enrolled in Fleet] + +. In {fleet}, open the **Agents** tab and see that an **Agent-pernode-demo-#####** agent is running. + +. Select the agent to view its details. + +. On the **Agent details** tab, on the **Integrations** pane, expand `system-1` to confirm that logs and metrics are incoming. You can click either the `Logs` or `Metrics` link to view details. ++ +[role="screenshot"] +image::images/helm-example-nodes-logs-and-metrics.png[Screen capture of the Logs and Metrics view on the Integrations pane] + + +[discrete] +[[agent-fleet-managed-helm-example-install-integration]] +=== Install the Kubernetes integration + +Now that you've {agent} and data is flowing, you can set up the {k8s} integration. + +. In your {ecloud} deployment, from the {kib} menu open the **Integrations** page. +. Run a search for `Kubernetes` and then select the {k8s} integration card. +. On the {k8s} integration page, click **Add Kubernetes** to add the integration to your {agent} policy. +. Scroll to the bottom of **Add Kubernetes integration** page. Under **Where to add this integration?** select the **Existing hosts** tab. On the **Agent policies** menu, select the agent policy that you created previously in the <> steps. ++ +You can leave all of the other integration settings at their default values. +. Click **Save and continue**. When prompted, select to **Add Elastic Agent later** since you've already added it using Helm. +. On the {k8s} integration page, open the **Assets** tab and select the **[Metrics Kubernetes] Pods** dashboard. ++ +On the dashboard, you can view the status of your {k8s} pods, including metrics on memory usage, CPU usage, and network throughput. ++ +[role="screenshot"] +image::images/helm-example-fleet-metrics-dashboard.png[Screen capture of the Metrics Kubernetes pods dashboard] + +You've successfully installed {agent} using Helm, and your {k8s} metrics data is available for viewing in {kib}. + +[discrete] +[[agent-fleet-managed-helm-example-tidy-up]] +=== Tidy up + +After you've run through this example, run the `helm uninstall` command to uninstall {agent}. + +[source,sh] +---- +helm uninstall demo +---- + +The uninstall should be confirmed as shown: + +[source,sh] +---- +release "demo" uninstalled +---- + +As a reminder, for full details about using the {agent} Helm chart refer to the link:https://github.com/elastic/elastic-agent/tree/main/deploy/helm/elastic-agent[{agent} Helm Chart Readme]. diff --git a/docs/en/ingest-management/elastic-agent/example-kubernetes-standalone-agent-helm.asciidoc b/docs/en/ingest-management/elastic-agent/example-kubernetes-standalone-agent-helm.asciidoc new file mode 100644 index 000000000..3dba3f63a --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/example-kubernetes-standalone-agent-helm.asciidoc @@ -0,0 +1,290 @@ +[[example-kubernetes-standalone-agent-helm]] += Example: Install standalone {agent} on Kubernetes using Helm + +preview::[] + +This example demonstrates how to install standalone {agent} on a Kubernetes system using a Helm chart, gather Kubernetes metrics and send them to an {es} cluster in {ecloud}, and then view visualizations of those metrics in {kib}. + +For an overview of the {agent} Helm chart and its benefits, refer to <>. + +This guide takes you through these steps: + +* <> +* <> +* <> +* <> + +[discrete] +[[agent-standalone-helm-example-prereqs]] +=== Prerequisites + +To get started, you need: + +* A local install of the link:https://helm.sh/[Helm] {k8s} package manager. +* An link:{ess-trial}[{ecloud}] hosted {es} cluster on version 8.16 or higher. +* An <>. +* An active {k8s} cluster. +* A local clone of the link:https://github.com/elastic/elastic-agent/tree/8.16[elastic/elastic-agent] GitHub repository. Make sure to use the `8.16` branch to ensure that {agent} has full compatibility with the Helm chart. + +[discrete] +[[agent-standalone-helm-example-install]] +=== Install {agent} + +. Open your {ecloud} deployment, and from the navigation menu select **Manage this deployment**. +. In the **Applications** section, copy the {es} endpoint and make a note of the endpoint value. +. Open a terminal shell and change into a directory in your local clone of the `elastic-agent` repo. +. Copy this command. ++ +[source,sh] +---- +helm install demo ./deploy/helm/elastic-agent \ +--set kubernetes.enabled=true \ +--set outputs.default.type=ESPlainAuthAPI \ +--set outputs.default.url=:443 \ +--set outputs.default.api_key="API_KEY" +---- ++ +Note that the command has these properties: + +* `helm install` runs the Helm CLI install tool. +* `demo` gives a name to the installed chart. You can choose any name. +* `./deploy/helm/elastic-agent` is a local path to the Helm chart to install (in time it's planned to have a public URL for the chart). +* `--set kubernetes.enabled=true` enables the {k8s} integration. The CLI parameter overrides the default `false` value for `kubernetes.enabled` in the {agent} link:https://github.com/elastic/elastic-agent/blob/main/deploy/helm/elastic-agent/values.yaml[values.yaml] file. +* `--set outputs.default.type=ESPlainAuthAPI` sets the authentication method for the {es} output to require an API key (again, overriding the value set by default in the {agent} link:https://github.com/elastic/elastic-agent/blob/main/deploy/helm/elastic-agent/values.yaml[values.yaml] file). +* `--set outputs.default.url=:443` sets the address of your {ecloud} deployment, where {agent} will send its output over port 443. +* `--set outputs.default.api_key="API_KEY"` sets the API key that {agent} will use to authenticate with your {es} cluster. ++ +-- +TIP: For a full list of all available YAML settings and descriptions, refer to the link:https://github.com/elastic/elastic-agent/tree/main/deploy/helm/elastic-agent[{agent} Helm Chart Readme]. +-- +. Update the command to replace: +.. `` with the {es} endpoint value that you copied earlier. +.. `` with your API key name. ++ +After your updates, the command should look something like this: ++ +[source,sh] +---- +helm install demo ./deploy/helm/elastic-agent \ +--set kubernetes.enabled=true \ +--set outputs.default.type=ESPlainAuthAPI \ +--set outputs.default.url=https://demo.es.us-central1.gcp.foundit.no:443 \ +--set outputs.default.api_key="A6ecaHNTJUFFcJI6esf4:5HJPxxxxxxxPS4KwSBeVEs" +---- + +. Run the command. ++ +The command output should confirm that three {agents} have been installed as well as the {k8s} integration: ++ +[source,sh] +---- +... +Installed agents: + - clusterWide [deployment - standalone mode] + - ksmSharded [statefulset - standalone mode] + - perNode [daemonset - standalone mode] + +Installed integrations: + - kubernetes [built-in chart integration] +... +---- + +. Run the `kubectl get pods -n default` command to confirm that the {agent} pods are running: ++ +[source,sh] +---- +NAME READY STATUS RESTARTS AGE +agent-clusterwide-demo-77c65f6c7b-trdms 1/1 Running 0 5m18s +agent-ksmsharded-demo-0 2/2 Running 0 5m18s +agent-pernode-demo-c7d75 1/1 Running 0 5m18s +---- + +. In your {ecloud} deployment, from the {kib} menu open the **Integrations** page. +. Run a search for `Kubernetes` and then select the {k8s} integration card. +. On the {k8s} integration page, select **Install Kubernetes assets**. This installs the dashboards, {es} indexes, and other assets used to monitor your {k8s} cluster. +. On the {k8s} integration page, open the **Assets** tab and select the **[Metrics Kubernetes] Nodes** dashboard. ++ +On the dashboard, you can view the status of your {k8s} nodes, including metrics on memory, CPU, and filesystem usage, network throughput, and more. ++ +[role="screenshot"] +image::images/helm-example-nodes-metrics-dashboard.png[Screen capture of the Metrics Kubernetes nodes dashboard] + +. On the {k8s} integration page, open the **Assets** tab and select the **[Metrics Kubernetes] Pods** dashboard. As with the nodes dashboard, on this dashboard you can view the status of your {k8s} pods, including various metrics on memory, CPU, and network throughput. ++ +[role="screenshot"] +image::images/helm-example-pods-metrics-dashboard.png[Screen capture of the Metrics Kubernetes pods dashboard] + +[discrete] +[[agent-standalone-helm-example-upgrade]] +=== Upgrade your {agent} configuration + +Now that you have {agent} installed, collecting, and sending data successfully, let's try changing the agent configuration settings. + +In the previous install example, three {agent} nodes were installed. One of these nodes, `agent-ksmsharded-demo-0`, is installed to enable the link:https://github.com/kubernetes/kube-state-metrics[kube-state-metrics] service. Let's suppose that you don't need those metrics and would like to upgrade your configuration accordingly. + +. Copy the command that you used earlier to install {agent}: ++ +[source,sh] +---- +helm install demo ./deploy/helm/elastic-agent \ +--set kubernetes.enabled=true \ +--set outputs.default.type=ESPlainAuthAPI \ +--set outputs.default.url=:443 \ +--set outputs.default.api_key="API_KEY" +---- + +. Update the command as follows: +.. Change `install` to upgrade. +.. Add a parameter `--set kubernetes.state.enabled=false`. This will override the default `true` value for the setting `kubernetes.state` in the {agent} link:https://github.com/elastic/elastic-agent/blob/main/deploy/helm/elastic-agent/values.yaml[values.yaml] file. ++ +[source,sh] +---- +helm upgrade demo ./deploy/helm/elastic-agent \ +--set kubernetes.enabled=true \ +--set kubernetes.state.enabled=false \ +--set outputs.default.type=ESPlainAuthAPI \ +--set outputs.default.url=:443 \ +--set outputs.default.api_key="API_KEY" +---- + +. Run the command. ++ +The command output should confirm that now only two {agents} are installed together with the {k8s} integration: ++ +[source,sh] +---- +... +Installed agents: + - clusterWide [deployment - standalone mode] + - perNode [daemonset - standalone mode] + +Installed integrations: + - kubernetes [built-in chart integration] +... +---- + +You've upgraded your configuration to run only two {agents}, without the kube-state-metrics service. You can similarly upgrade your agent to change other settings defined in the in the {agent} link:https://github.com/elastic/elastic-agent/blob/main/deploy/helm/elastic-agent/values.yaml[values.yaml] file. + +[discrete] +[[agent-standalone-helm-example-change-mode]] +=== Change {agent}'s running mode + +By default {agent} runs under the `elastic` user account. For some use cases you may want to temporarily change an agent to run with higher privileges. + +. Run the `kubectl get pods -n default` command to view the running {agent} pods: ++ +[source,sh] +---- +NAME READY STATUS RESTARTS AGE +agent-clusterwide-demo-77c65f6c7b-trdms 1/1 Running 0 5m18s +agent-pernode-demo-c7d75 1/1 Running 0 5m18s +---- + +. Now, run the `kubectl exec` command to enter one of the running {agents}, substituting the correct pod name returned from the previous command. For example: ++ +[source,sh] +---- +kubectl exec -it pods/agent-pernode-demo-c7d75 -- bash +---- + +. From inside the pod, run the Linux `ps aux` command to view the running processes. ++ +[source,sh] +---- +ps aux +---- ++ +The results should be similar to the following: ++ +[source,sh] +---- +USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND +elastic+ 1 0.0 0.0 1936 416 ? Ss 21:04 0:00 /usr/bin/tini -- /usr/local/bin/docker-entrypoint -c /etc/elastic-agent/agent.yml -e +elastic+ 10 0.2 1.3 2555252 132804 ? Sl 21:04 0:13 elastic-agent container -c /etc/elastic-agent/agent.yml -e +elastic+ 37 0.6 2.0 2330112 208468 ? Sl 21:04 0:37 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat metricbeat -E +elastic+ 38 0.2 1.7 2190072 177780 ? Sl 21:04 0:13 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat filebeat -E se +elastic+ 56 0.1 1.7 2190136 175896 ? Sl 21:04 0:11 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat metricbeat -E +elastic+ 68 0.1 1.8 2190392 184140 ? Sl 21:04 0:12 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat metricbeat -E +elastic+ 78 0.7 2.0 2330496 204964 ? Sl 21:04 0:48 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat filebeat -E se +elastic+ 535 0.0 0.0 3884 3012 pts/0 Ss 22:47 0:00 bash +elastic+ 543 0.0 0.0 5480 2360 pts/0 R+ 22:47 0:00 ps aux +---- + +. In the command output, note that {agent} is currently running as the `elastic` user: ++ +[source,sh] +---- +elastic+ 10 0.2 1.3 2555252 132804 ? Sl 21:04 0:13 elastic-agent container -c /etc/elastic-agent/agent.yml -e +---- + +. Run `exit` to leave the {agent} pod. + +. Run the `helm upgrade` command again, this time adding the parameter `--set agent.unprivileged=false` to override the default `true` value for that setting. ++ +[source,sh] +---- +helm upgrade demo ./deploy/helm/elastic-agent \ +--set kubernetes.enabled=true \ +--set kubernetes.state.enabled=false \ +--set outputs.default.type=ESPlainAuthAPI \ +--set outputs.default.url=:443 \ +--set outputs.default.api_key="API_KEY" \ +--set agent.unprivileged=false +---- + +. Run the `kubectl get pods -n default` command to view the running {agent} pods: ++ +[source,sh] +---- +NAME READY STATUS RESTARTS AGE +agent-clusterwide-demo-77c65f6c7b-trdms 1/1 Running 0 5m18s +agent-pernode-demo-s6s7z 1/1 Running 0 5m18s +---- + +. Re-run the `kubectl exec` command to enter one of the running {agents}, substituting the correct pod name. For example: ++ +[source,sh] +---- +kubectl exec -it pods/agent-pernode-demo-s6s7z -- bash +---- + +. From inside the pod, run the Linux `ps aux` command to view the running processes. ++ +[source,sh] +---- +USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND +root 1 0.0 0.0 1936 452 ? Ss 23:10 0:00 /usr/bin/tini -- /usr/local/bin/docker-entrypoint -c /etc/elastic-agent/agent.yml -e +root 9 0.9 1.3 2488368 135920 ? Sl 23:10 0:01 elastic-agent container -c /etc/elastic-agent/agent.yml -e +root 27 0.9 1.9 2255804 203128 ? Sl 23:10 0:01 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat metricbeat -E +root 44 0.3 1.8 2116148 187432 ? Sl 23:10 0:00 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat metricbeat -E +root 64 0.3 1.8 2263868 188892 ? Sl 23:10 0:00 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat metricbeat -E +root 76 0.4 1.8 2190136 190972 ? Sl 23:10 0:00 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat filebeat -E se +root 100 1.2 2.0 2256316 207692 ? Sl 23:10 0:01 /usr/share/elastic-agent/data/elastic-agent-d99b09/components/agentbeat filebeat -E se +root 142 0.0 0.0 3752 3068 pts/0 Ss 23:12 0:00 bash +root 149 0.0 0.0 5480 2376 pts/0 R+ 23:13 0:00 ps aux +---- + +. Run `exit` to leave the {agent} pod. + +You've upgraded the {agent} privileges to run as `root`. To change the settings back, you can re-run the `helm upgrade` command with `--set agent.unprivileged=true` to return to the default `unprivileged` mode. + + +[discrete] +[[agent-standalone-helm-example-tidy-up]] +=== Tidy up + +After you've run through this example, run the `helm uninstall` command to uninstall {agent}. + +[source,sh] +---- +helm uninstall demo +---- + +The uninstall should be confirmed as shown: + +[source,sh] +---- +release "demo" uninstalled +---- + +As a reminder, for full details about using the {agent} Helm chart refer to the link:https://github.com/elastic/elastic-agent/tree/main/deploy/helm/elastic-agent[{agent} Helm Chart Readme]. diff --git a/docs/en/ingest-management/elastic-agent/images/helm-example-fleet-metrics-dashboard.png b/docs/en/ingest-management/elastic-agent/images/helm-example-fleet-metrics-dashboard.png new file mode 100644 index 000000000..f0f3ae7fa Binary files /dev/null and b/docs/en/ingest-management/elastic-agent/images/helm-example-fleet-metrics-dashboard.png differ diff --git a/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-enrollment-confirmation.png b/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-enrollment-confirmation.png new file mode 100644 index 000000000..c55a50bd6 Binary files /dev/null and b/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-enrollment-confirmation.png differ diff --git a/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-logs-and-metrics.png b/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-logs-and-metrics.png new file mode 100644 index 000000000..4d57e979d Binary files /dev/null and b/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-logs-and-metrics.png differ diff --git a/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-metrics-dashboard.png b/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-metrics-dashboard.png new file mode 100644 index 000000000..9322eb818 Binary files /dev/null and b/docs/en/ingest-management/elastic-agent/images/helm-example-nodes-metrics-dashboard.png differ diff --git a/docs/en/ingest-management/elastic-agent/images/helm-example-pods-metrics-dashboard.png b/docs/en/ingest-management/elastic-agent/images/helm-example-pods-metrics-dashboard.png new file mode 100644 index 000000000..fa894a1de Binary files /dev/null and b/docs/en/ingest-management/elastic-agent/images/helm-example-pods-metrics-dashboard.png differ diff --git a/docs/en/ingest-management/elastic-agent/install-agent-msi.asciidoc b/docs/en/ingest-management/elastic-agent/install-agent-msi.asciidoc index b3b882d4b..c67a1e18a 100644 --- a/docs/en/ingest-management/elastic-agent/install-agent-msi.asciidoc +++ b/docs/en/ingest-management/elastic-agent/install-agent-msi.asciidoc @@ -10,13 +10,24 @@ The MSI package installer must be run by an administrator account. The installer . Download the latest Elastic Agent MSI binary from the link:https://www.elastic.co/downloads/elastic-agent[{agent} download page]. -. Run the installer: +. Run the installer. The command varies slightly depending on whether you're using the default Windows command prompt or PowerShell. ++ +==== +** Using the default command prompt: + [source,shell] ---- elastic-agent--windows-x86_64.msi INSTALLARGS="--url= --enrollment-token=" ---- + +** Using PowerShell: ++ +[source,shell] +---- +./elastic-agent--windows-x86_64.msi --% INSTALLARGS="--url= --enrollment-token=" +---- +==== ++ Where: * `VERSION` is the {stack} version you're installing, indicated in the MSI package name. For example, `8.13.2`. @@ -47,7 +58,7 @@ Installing using an MSI package has the following behaviors: [discrete] == Upgrading -Upgrades are not supported and are prevented by the MSI itself. Instead, all of your {agent} upgrades can be managed in {fleet}. +The {agent} version can be upgraded via {fleet}, but the registered MSI version will display the initially installed version (this shortcoming will be addressed in future releases). Attempts to upgrade outside of {fleet} via the MSI will require an uninstall and reinstall procedure to upgrade. Also note that this MSI implementation relies on the tar {agent} binary to upgrade the installation. Therefore if the {agent} is installed in an air-gapped environment, you must ensure that the tar image is available before an upgrade request is issued. [discrete] == Installing in a custom location diff --git a/docs/en/ingest-management/elastic-agent/install-elastic-agent-in-container.asciidoc b/docs/en/ingest-management/elastic-agent/install-elastic-agent-in-container.asciidoc index 7be52ecb1..2bedfd51d 100644 --- a/docs/en/ingest-management/elastic-agent/install-elastic-agent-in-container.asciidoc +++ b/docs/en/ingest-management/elastic-agent/install-elastic-agent-in-container.asciidoc @@ -30,3 +30,5 @@ To learn how to run {agent}s in a containerized environment, see: * {eck-ref}/k8s-elastic-agent.html[Run {agent} on ECK] -- for {eck} users +NOTE: Enrollment handling for {agent} in a containerized environment has some special nuances. +For details refer to <>. \ No newline at end of file diff --git a/docs/en/ingest-management/elastic-agent/install-elastic-agent.asciidoc b/docs/en/ingest-management/elastic-agent/install-elastic-agent.asciidoc index 291818cd2..24c487fd8 100644 --- a/docs/en/ingest-management/elastic-agent/install-elastic-agent.asciidoc +++ b/docs/en/ingest-management/elastic-agent/install-elastic-agent.asciidoc @@ -1,7 +1,7 @@ [[elastic-agent-installation]] = Install {agent}s -[IMPORTANT] +[IMPORTANT] .Restrictions ==== Note the following restrictions when installing {agent} on your system: @@ -12,7 +12,7 @@ run the commands described here. After the {agent} service is installed and runn make sure you run these commands without prepending them with `./` to avoid invoking the wrong binary. * Running {agent} commands using the Windows PowerShell ISE is not supported. -* See also the <> described on this page. +* See also the <> described on this page. ==== You have a few options for installing and managing an {agent}: @@ -57,46 +57,60 @@ Refer to: * {eck-ref}/k8s-elastic-agent.html[Run {agent} on ECK] -- for {eck} users -- -[IMPORTANT] +[IMPORTANT] .Restrictions in {serverless-short} -==== +==== If you are using {agent} with link:{serverless-docs}[{serverless-full}], note these differences from use with {ess} and self-managed {es}: * The number of {agents} that may be connected to an {serverless-full} project is limited to 10 thousand. * The minimum supported version of {agent} supported for use with {serverless-full} is 8.11.0. ==== +[TIP] +.Applying {agent} configurations dynamically +==== +When you set up {agent}, you might not yet have all input configuration details available. To solve this problem, the input configuration accepts variables and conditions that get evaluated at runtime using information from the running environment, allowing you to apply configurations dynamically. To learn more, refer to <>. +==== + [discrete] -[[elastic-agent-installation-minimum-requirements]] -== Minimum requirements +[[elastic-agent-installation-resource-requirements]] +== Resource requirements + +The {agent} resources consumption is influenced by the number of integration and the environment its been running on. + +Using our lab environment as an example, we can observe the following resource consumption: -The following are the minimum system requirements for installing {agent}. // lint ignore mem [discrete] === CPU and RSS memory size -// lint ignore 2vcpu 1gb -Minimum requirements have been determined by running the {agent} on a GCP `e2-micro` instance (2vCPU/1GB). -The {agent} used the default policy, running the system integration and self-monitoring. - +// lint ignore 2 vCPU 8.0 GiB +We tested using an AWS `m7i.large` instance type with 2 vCPUs, 8.0 GB of memory, and up to 12.5 Gbps of bandwidth. The tests ingested a single log file using both the <> with self monitoring enabled. +These tests are representative of use cases that attempt to ingest data as fast as possible. This does not represent the resource overhead when using {integrations-docs}/endpoint[{elastic-defend}]. +[options,header] |=== -| **CPU** | Under 2% total, including all monitoring processes -| **RSS memory size** | 400 MB +| **Resource** | **Throughput** | **Scale** +| **CPU*** | ~67% | ~20% +| **RSS memory size*** | ~280 MB | ~220 MB +| **Write network throughput** | ~3.5 MB/s | 480 KB/s |=== + +^*^ including all monitoring processes + Adding integrations will increase the memory used by the agent and its processes. [discrete] === Size on disk -The disk requirements for {agent} vary by operating system and {stack} version. With version 8.14 we have significantly reduced the size of the {agent} binary. Further reductions are planned to be made in future releases. +The disk requirements for {agent} vary by operating system and {stack} version. [options,header] |=== -|Operating system |8.13 |8.14 +|Operating system |8.13 | 8.14 | 8.15 | 8.18 | 9.0 | -| **Linux** | 1800 MB | 1018 MB -| **macOS** | 1100 MB | 619 MB -| **Windows** | 891 MB | 504 MB +| **Linux** | 1800 MB | 1018 MB | 1060 MB | 1.5 GB | 1.5 GB | +| **macOS** | 1100 MB | 619 MB | 680 MB | 775 MB | 7755 MB | +| **Windows** | 891 MB | 504 MB | 500 MB | 678 MB | 705 MB | |=== -During upgrades, double the disk space is required to store the new {agent} binary. After the upgrade completes, the original {agent} is removed from disk to free up the space. \ No newline at end of file +During upgrades, double the disk space is required to store the new {agent} binary. After the upgrade completes, the original {agent} is removed from disk to free up the space. diff --git a/docs/en/ingest-management/elastic-agent/install-on-kubernetes-using-helm.asciidoc b/docs/en/ingest-management/elastic-agent/install-on-kubernetes-using-helm.asciidoc new file mode 100644 index 000000000..af4a63e94 --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/install-on-kubernetes-using-helm.asciidoc @@ -0,0 +1,24 @@ +[[install-on-kubernetes-using-helm]] += Install {agent} on Kubernetes using Helm + +preview::[] + +Starting with {stack} version 8.16, a Helm chart is available for installing {agent} in a Kubernetes environment. A Helm-based install offers several advantages, including simplified deployment, availability in marketplaces, streamlined ugrades, as well as quick rollbacks whenever they're needed. + +Features of the Helm-based {agent} install include: + +* Support for both standalone and {fleet}-managed {agent}. +* For standalone agents, a built-in Kubernetes policy similar to that available in {fleet} for {fleet}-managed agents. +* Support for custom integrations. +* Support for {es} outputs with authentication through username and password, an API key, or a stored secret. +* Easy switching between privileged (`root`) and unprivileged {agent} deployments. +* Support for {stack} deployments on {eck}. + +For detailed install steps, try one of our walk-through examples: + +* <> +* <> + +NOTE: The {agent} Helm chart is currently available from inside the link:https://github.com/elastic/elastic-agent[elastic/elastic-agent] GitHub repo. It's planned to soon make the chart available from the Elastic Helm repository. + +You can also find details about the Helm chart, including all available YAML settings and descriptions, in the link:https://github.com/elastic/elastic-agent/tree/main/deploy/helm/elastic-agent[{agent} Helm Chart Readme]. Several link:https://github.com/elastic/elastic-agent/tree/main/deploy/helm/elastic-agent/examples[examples] are available if you'd like to explore other use cases. \ No newline at end of file diff --git a/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc b/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc new file mode 100644 index 000000000..581521f42 --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc @@ -0,0 +1,81 @@ +[[otel-agent-transform]] +== Transform an installed {agent} to run as an OTel Collector + +preview::[] + +If you have a currently installed standalone {agent}, it can be configured to run as an <>. This allows you to run {agent} both as a service and in an OTel Collector mode. + +In order to configure an installed standalone {agent} to run as an OTel Collector, it's enough to include a valid <> configuration in the `elastic-agent.yml` file, as shown in the following example. + +=== Example: configure {agent} to ingest host logs and metrics into Elasticsearch using the OTel Collector + +**Prerequisites** + +You'll need the following: + +. A suitable <> for authenticating on Elasticsearch +. An installed standalone {agent} +. A valid OTel Collector configuration. In this example we'll use the OTel sample configuration included in the {agent} repository: `otel_samples/platformlogs_hostmetrics.yml`. +** link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/platformlogs_hostmetrics.yml[Linux version] +** link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/darwin/platformlogs_hostmetrics.yml[MacOS version] + +**Steps** + +To change a running standalone {agent} to run as an OTel Collector: + +. Create a directory where the OTel Collector can save its state. In this example we use `<{agent} install directory>/data/otelcol`. +. Open the `<{agent} install directory>/otel_samples/platformlogs_hostmetrics.yml` file for editing. +. Set environment details to be used by OTel Collector: +* **Option 1:** Define environment variables for the {agent} service: +** `ELASTIC_ENDPOINT`: The URL of the {es} instance where data will be sent +** `ELASTIC_API_KEY`: The API Key to use to authenticate with {es} +** `STORAGE_DIR`: The directory where the OTel Collector can persist its state +* **Option 2:** Replace the environment variable references in the sample configuration with the corresponding values: +** `${env:ELASTIC_ENDPOINT}`:The URL of the {es} instance where data will be sent +** `${env:ELASTIC_API_KEY}`: The API Key to use to authenticate with {es} +** `${env:STORAGE_DIR}`: The directory where the OTel Collector can persist its state +. Save the opened OTel configuration as `elastic-agent.yml`, overwriting the default configuration of the installed agent. +. Run the `elastic-agent status` command to verify that the new configuration has been correctly applied: ++ +[source,shell] +---- +elastic-agent status +---- +The OTel Collector running configuration should appear under `elastic-agent` key (note the `extensions` and `pipeline` keys): ++ +[source,shell] +---- +┌─ fleet +│ └─ status: (STOPPED) Not enrolled into Fleet +└─ elastic-agent + ├─ status: (HEALTHY) Running + ├─ extensions + │ ├─ status: StatusOK + │ └─ extension:file_storage + │ └─ status: StatusOK + ├─ pipeline:logs/platformlogs + │ ├─ status: StatusOK + │ ├─ exporter:elasticsearch/otel + │ │ └─ status: StatusOK + │ ├─ processor:resourcedetection + │ │ └─ status: StatusOK + │ └─ receiver:filelog/platformlogs + │ └─ status: StatusOK + └─ pipeline:metrics/hostmetrics + ├─ status: StatusOK + ├─ exporter:elasticsearch/ecs + │ └─ status: StatusOK + ├─ processor:attributes/dataset + │ └─ status: StatusOK + ├─ processor:elasticinframetrics + │ └─ status: StatusOK + ├─ processor:resource/process + │ └─ status: StatusOK + ├─ processor:resourcedetection + │ └─ status: StatusOK + └─ receiver:hostmetrics/system + └─ status: StatusOK +---- ++ +. Congratulations! Host logs and metrics are now being collected and ingested by the {agent} service running an OTel Collector instance. +For further details about OpenTelemetry collector components supported by {agent}, refer to the link:https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel#components[Elastic Distribution for OpenTelemetry Collector README]. diff --git a/docs/en/ingest-management/elastic-agent/otel-agent.asciidoc b/docs/en/ingest-management/elastic-agent/otel-agent.asciidoc index 4defa6edd..177c73dc1 100644 --- a/docs/en/ingest-management/elastic-agent/otel-agent.asciidoc +++ b/docs/en/ingest-management/elastic-agent/otel-agent.asciidoc @@ -8,3 +8,5 @@ The link:https://opentelemetry.io/docs/collector/[OpenTelemetry Collector] is a When you run {agent} in `otel` mode it supports the standard OTel Collector configuration format that defines a set of receivers, processors, exporters, and connectors. Logs, metrics, and traces can be ingested using OpenTelemetry data formats. For a full overview and steps to configure {agent} in `otel` mode, including a guided onboarding, refer to the link:https://github.com/elastic/opentelemetry/tree/main[Elastic Distributions for OpenTelemetry] repository in GitHub. You can also check the <> in the {fleet} and {agent} Command reference. + +If you have a currently running {agent} you can <>. diff --git a/docs/en/ingest-management/elastic-agent/run-container-common/download-elastic-agent.asciidoc b/docs/en/ingest-management/elastic-agent/run-container-common/download-elastic-agent.asciidoc index f13bcb615..ddc77ef2b 100644 --- a/docs/en/ingest-management/elastic-agent/run-container-common/download-elastic-agent.asciidoc +++ b/docs/en/ingest-management/elastic-agent/run-container-common/download-elastic-agent.asciidoc @@ -1,4 +1,4 @@ -NOTE: You can find {agent} Docker images https://www.docker.elastic.co/r/beats/elastic-agent[here]. +NOTE: You can find {agent} Docker images https://www.docker.elastic.co/r/elastic-agent/elastic-agent[here]. Download the manifest file: diff --git a/docs/en/ingest-management/elastic-agent/running-on-kubernetes-managed-by-fleet.asciidoc b/docs/en/ingest-management/elastic-agent/running-on-kubernetes-managed-by-fleet.asciidoc index 312739395..a578c0438 100644 --- a/docs/en/ingest-management/elastic-agent/running-on-kubernetes-managed-by-fleet.asciidoc +++ b/docs/en/ingest-management/elastic-agent/running-on-kubernetes-managed-by-fleet.asciidoc @@ -67,8 +67,22 @@ To specify different destination/credentials, change the following parameters in <5> The basic authentication username used to connect to {kib} and retrieve a `service_token` to enable {fleet}. <6> The basic authentication password used to connect to {kib} and retrieve a `service_token` to enable {fleet}. -Refer to <> for all available options. +If you need to run {fleet-server} as well, adjust the `docker run` command above by adding these environment variables: + +[source,yaml] +------------------------------------------------ +- name: FLEET_SERVER_ENABLE + value: "true" <1> +- name: FLEET_SERVER_ELASTICSEARCH_HOST + value: "" <2> +- name: FLEET_SERVER_SERVICE_TOKEN + value: "" <3> +------------------------------------------------ +<1> Set to `true` to bootstrap {fleet-server} on this {agent}. This automatically forces {fleet} enrollment as well. +<2> The Elasticsearch host for Fleet Server to communicate with, for example `http://elasticsearch:9200`. +<3> Service token to use for communication with {es} and {kib}. +Refer to <> for all available options. [discrete] == Step 4: Configure tolerations diff --git a/docs/en/ingest-management/elastic-agent/running-on-kubernetes-standalone.asciidoc b/docs/en/ingest-management/elastic-agent/running-on-kubernetes-standalone.asciidoc index 5a2b0f06c..012afbe7d 100644 --- a/docs/en/ingest-management/elastic-agent/running-on-kubernetes-standalone.asciidoc +++ b/docs/en/ingest-management/elastic-agent/running-on-kubernetes-standalone.asciidoc @@ -1,7 +1,7 @@ [[running-on-kubernetes-standalone]] = Run {agent} Standalone on Kubernetes -:manifest: https://raw.githubusercontent.com/elastic/elastic-agent/{branch}/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml +:manifest: https://raw.githubusercontent.com/elastic/elastic-agent/v{bare_version}/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml :show-condition: enabled == What you need diff --git a/docs/en/ingest-management/elastic-agent/uninstall-elastic-agent.asciidoc b/docs/en/ingest-management/elastic-agent/uninstall-elastic-agent.asciidoc index 0925851ad..38510dc16 100644 --- a/docs/en/ingest-management/elastic-agent/uninstall-elastic-agent.asciidoc +++ b/docs/en/ingest-management/elastic-agent/uninstall-elastic-agent.asciidoc @@ -7,7 +7,18 @@ To uninstall {agent}, run the `uninstall` command from the directory where {agent} is running. -IMPORTANT: Be sure to run the `uninstall` command from the directory where {agent} is running, as shown in the example below, and not from the directory where you previously ran the `install` command. Running the command from the wrong directory can leave the agent in an inconsistent state. +[IMPORTANT] +==== +Be sure to run the `uninstall` command from a directory outside of where {agent} is installed. + +For example, on a Windows system the install location is `C:\Program Files\Elastic\Agent`. Run the uninstall command from `C:\Program Files\Elastic` or `\tmp`, or even your default home directory: + +[source,shell] +---- +C:\"Program Files"\Elastic\Agent\elastic-agent.exe uninstall +---- + +==== -- include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/uninstall-widget.asciidoc[] diff --git a/docs/en/ingest-management/elastic-agent/upgrade-standalone-elastic-agent.asciidoc b/docs/en/ingest-management/elastic-agent/upgrade-standalone-elastic-agent.asciidoc index a1788de62..4fe6e2402 100644 --- a/docs/en/ingest-management/elastic-agent/upgrade-standalone-elastic-agent.asciidoc +++ b/docs/en/ingest-management/elastic-agent/upgrade-standalone-elastic-agent.asciidoc @@ -34,6 +34,8 @@ As an alterative, you can do one of the following: * <> for standalone {agent} to access the {artifact-registry}. * <> for standalone {agent} to access binary downloads. +As well, starting from version 8.9.0, during the upgrade process {agent} needs to download a PGP/GPG key. Refer to <> for the steps to configure the key download location in an air-gapped environment. + Refer to <> for more details. [[upgrade-standalone-verify-package]] diff --git a/docs/en/ingest-management/fips-ingest.asciidoc b/docs/en/ingest-management/fips-ingest.asciidoc new file mode 100644 index 000000000..fd8291972 --- /dev/null +++ b/docs/en/ingest-management/fips-ingest.asciidoc @@ -0,0 +1,132 @@ +[role="xpack"] +[[fips-ingest]] += FIPS mode for Ingest tools + +preview::[] + +{agent}, {fleet}, {filebeat}, {metricbeat}, and APM Server binaries are built and are configured to use FIPS 140-2 compliant cryptography. +Generally speaking FIPS 140-2 requirements can be summarized as: + +- linking against a FIPS certified cryptographic library +- using only FIPS approved cryptographic functions +- ensuring that the configuration of the component is FIPS 140-2 compliant. + +[discrete] +[[fips-binaries]] +== FIPS-compatible binaries and configuration + +FIPS compatible binaries for {agent}, {fleet}, {filebeat}, {metricbeat}, and APM Server are available for link:https://www.elastic.co/downloads[download]. +Look for the `Linux 64-bit (FIPS)` or `Linux aarch64 (FIPS)` platform option on the product download pages for {agent} and {fleet}, {filebeat}, and {metricbeat}. +Look for the `Linux x86_64 (FIPS)` or `Linux aarch64 (FIPS)` platform option on the APM Server download page. + +IMPORTANT: The default configurations provided in the binaries are FIPS compatible. Be sure to check and understand the implications of changing default configurations. + +[discrete] +[[ingest-limitations-all]] +== Limitations + +[discrete] +[[ingest-limitations-tls]] +=== TLS + +Only FIPS 140-2 compliant TLS protocols, ciphers, and curve types are allowed to be used: + +* The supported TLS versions are `TLS v1.2` and `TLS v1.3`. +* The supported cipher suites are: + +** `TLS v1.2`: `ECDHE-RSA-AES-128-GCM-SHA256`, `ECDHE-RSA-AES-256-GCM-SHA384`, `ECDHE-ECDSA-AES-128-GCM-SHA256`, `ECDHE-ECDSA-AES-256-GCM-SHA384` +** `TLS v1.3`: `TLS-AES-128-GCM-SHA256`, `TLS-AES-256-GCM-SHA384` + +* The supported curve types are `P-256`, `P-384` and `P-521`. + +Support for encrypted private keys is not available, as the cryptographic modules used for decrypting password protected keys are not FIPS validated. If an output or any other component with an SSL key that is password protected is configured, the components will fail to load the key. When running in FIPS mode, you must provide non-encrypted keys. +Be sure to enforce security in your FIPS environments through other means, such as strict file permissions and access controls on the key file itself, for example. + +These TLS related restrictions apply to all components listed--{agent}, {fleet}, {filebeat}, {metricbeat}, and APM Server. + +[discrete] +[[ingest-inputoutput-limitations]] +=== General output and input limitations (Kerberos protocol) + +The Kerberos protocol is not supported for any output or input, which also impacts the available `sasl.mechanism` for the Kafka output where only `PLAIN` is supported. + +This impacts link:https://www.elastic.co/guide/en/beats/filebeat/8.19/configuration-kerberos.html[Filebeat], link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/configuration-kerberos.html[Metricbeat], and APM Server, as well as output configurations for {agent} with {fleet-server}. + +[discrete] +[[ingest-apm-limitations]] +=== APM Server + +* The link:https://www.elastic.co/guide/en/observability/8.19/apm-keystore.html[Secrets Keystore] is not supported. + +// `{observability-guide}` attribute resolving to 8.x and `404`-ing +// * The link:{observability-guide}/apm-keystore.html[Secrets Keystore] is not supported. + +[discrete] +[[ingest-filebeat-limitations]] +=== Filebeat + +// `{filebeat-ref}` attribute resolving to 8.x and `404`-ing +// * The link:{filebeat-ref}/keystore.html[Secrets Keystore] is not supported. + +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/keystore.html[Secrets Keystore] is not supported. +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/processor-translate-guid.html[Translate GUID processor] is not supported. +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/fingerprint.html[Fingerprint processor] does not support the md5 and sha1 method. +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/community-id.html[Community ID Network Flowhash processor] is not supported. +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-module-azure.html[Azure module] including the link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-input-azure-eventhub.html[Azure eventhub input] and the link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-input-azure-blob-storage.html[Azure Blob Storage Input] are not currently supported. + The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/add-cloud-metadata.html[Add Cloud Metadata processor] does not support the Azure Virtual Machine provider currently. +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-module-o365.html[Office 365 module (Beta)] and the link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-input-o365audit.html[Office 365 input (Deprecated)] are not supported. +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-input-gcp-pubsub.html[GCP Pub/Sub input] and the link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-input-gcs.html[Google Cloud Storage input] are not supported for now. +* The link:https://www.elastic.co/guide/en/beats/filebeat/8.19/filebeat-input-entity-analytics.html[Entity Analytics input] is not supported. + +[discrete] +[[ingest-metricbeat-limitations]] +=== Metricbeat + +// `{metricbeat-ref}` attribute resolving to 8.x and `404`-ing +// * The link:{metricbeat-ref}/keystore.html[Secrets Keystore] is not supported. + +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/keystore.html[Secrets Keystore] is not supported. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/processor-translate-guid.html[Translate GUID processor] is not supported. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/fingerprint.html[Fingerprint processor] does not support the md5 and sha1 method. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/community-id.html[Community ID Network Flowhash processor] is not supported. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-azure.html[Azure module] is currently not supported. + The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/add-cloud-metadata.html[Add Cloud Metadata processor] does not support the Azure Virtual Machine provider currently. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-gcp.html[Google Cloud Platform module] is currently not supported. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-kvm.html[Beta KVM module] is not yet supported. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-mongodb.html[Mongo DB module] is not supported. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-mysql.html[MySQL], link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-postgresql.html[PostgreSQL], link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-mssql.html[MSSQL] and link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-sql.html[SQL] modules are not supported. +* The link:https://www.elastic.co/guide/en/beats/metricbeat/8.19/metricbeat-module-oracle.html[Oracle module] is not supported. + +[discrete] +[[ingest-limitations-agent]] +=== Elastic Agent and Fleet Server + +When you use {agent} and {fleet-server}, these limitations apply: + +* Running {agent} in link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/README.md[OpenTelemetry mode] is not yet supported. + This includes all receivers, such as Filebeat Receiver, Metricbeat Receiver, and link:https://www.elastic.co/docs/reference/integrations/prometheus[Prometheus Receiver]. +* Some Elastic Integrations are not FIPS compatible, as they depend on functionality that is not yet supported for FIPS configuration. + In general, when using {agent} and {fleet-server}, the same restrictions listed previously for {metricbeat} and {filebeat} modules, inputs, and processors apply. + +[discrete] +[[ingest-limitations-integrations]] +=== Elastic Integrations that are not FIPS compatible + +These Elastic Integrations have components that are **not** FIPS compatible, and **cannot** be used in FIPS environments, even if combined with other ingest tools that offer FIPS mode. + +- link:https://www.elastic.co/docs/reference/integrations/azure/events[Azure Logs Integration (v2 preview)] +- link:https://www.elastic.co/docs/reference/integrations/azure/eventhub[Azure Event Hub Input] +- link:https://www.elastic.co/docs/reference/integrations/azure_ai_foundry[Azure AI Foundry Integration] +- link:https://www.elastic.co/docs/reference/integrations/azure_app_service[Azure App Service Integration] +- link:https://www.elastic.co/docs/reference/integrations/azure_application_insights[Azure Application Insights Integration] +- link:https://www.elastic.co/docs/reference/integrations/azure_billing[Azure Billing Metrics Integration] +- link:https://www.elastic.co/docs/reference/integrations/azure_functions[Azure Functions Integration] +- link:https://www.elastic.co/docs/reference/integrations/azure_logs[Custom Azure Logs Integration] +- link:https://www.elastic.co/docs/reference/integrations/azure_metrics[Azure Resource Metrics Integration] +- link:https://www.elastic.co/docs/reference/integrations/azure_openai[Azure OpenAI Integration] +- link:https://www.elastic.co/docs/reference/integrations/sql[SQL Input] +- link:https://www.elastic.co/docs/reference/integrations/postgresql[PostgreSQL Integration] +- link:https://www.elastic.co/docs/reference/integrations/mongodb[MongoDB Integration] +- link:https://www.elastic.co/docs/reference/integrations/mysql[MySQL Integration] +- link:https://www.elastic.co/docs/reference/integrations/microsoft_sqlserver[Microsoft SQL Server Integration] +- link:https://www.elastic.co/docs/reference/integrations/oracle[Oracle Integration] diff --git a/docs/en/ingest-management/fleet-agent-proxy-managed.asciidoc b/docs/en/ingest-management/fleet-agent-proxy-managed.asciidoc index aaee58bf8..e9eac2005 100644 --- a/docs/en/ingest-management/fleet-agent-proxy-managed.asciidoc +++ b/docs/en/ingest-management/fleet-agent-proxy-managed.asciidoc @@ -179,6 +179,4 @@ Equally important is the Certificate Authority that the agents need to use to va image::images/elastic-agent-edit-proxy-secure-settings.png[Screen capture of the Edit Proxy UI, highlighting the Certificate and Certificate key settings] -NOTE: Currently {agents} will not present a certificate for Control Plane traffic to the {fleet-server}. Some proxy servers are setup to mandate that the client setting up a connection presents a certificate to them before allowing that client to connect. This issue will be resolved by link:https://github.com/elastic/elastic-agent/issues/2248[issue #2248]. Our recommendation is to avoid adding a secure proxy as such in a {fleet-server} configuration flyout. - -NOTE: In case {kib} is behind a proxy server or is otherwise unable to access the {package-registry} to download package metadata and content, refer to <>. \ No newline at end of file +NOTE: In case {kib} is behind a proxy server or is otherwise unable to access the {package-registry} to download package metadata and content, refer to <>. diff --git a/docs/en/ingest-management/fleet-agent-proxy-package-registry.asciidoc b/docs/en/ingest-management/fleet-agent-proxy-package-registry.asciidoc index 68b54d360..adc07fd0f 100644 --- a/docs/en/ingest-management/fleet-agent-proxy-package-registry.asciidoc +++ b/docs/en/ingest-management/fleet-agent-proxy-package-registry.asciidoc @@ -17,6 +17,10 @@ example: xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net ---- +If your HTTP proxy requires authentication, you can include the +credentials in the URI, such as `https://username:password@your-nat-gateway.corp.net`, +only when using HTTPS. + == What information is sent to the {package-registry}? In production environments, {kib}, through the {fleet} plugin, is the only service interacting with the {package-registry}. Communication happens when interacting with the Integrations UI, and when upgrading {kib}. The shared information is about discovery of Elastic packages and their available versions. In general, the only deployment-specific data that is shared is the {kib} version. diff --git a/docs/en/ingest-management/fleet/add-fleet-server-cloud.asciidoc b/docs/en/ingest-management/fleet/add-fleet-server-cloud.asciidoc index 3021b057e..05c233ec2 100644 --- a/docs/en/ingest-management/fleet/add-fleet-server-cloud.asciidoc +++ b/docs/en/ingest-management/fleet/add-fleet-server-cloud.asciidoc @@ -30,8 +30,7 @@ image::images/fleet-server-cloud-deployment.png[{fleet-server} Cloud deployment {fleet-server} is compatible with the following Elastic products: * {stack} 7.13 or later. -** For version compatibility: {es} >= {fleet-server} >= {agent} (except for -bugfix releases) +** For version compatibility, {es} must be at the same or a later version than {fleet-server}, and {fleet-server} needs to be at the same or a later version than {agent} (not including patch releases). ** {kib} should be on the same minor version as {es}. * {ece} 2.10 or later diff --git a/docs/en/ingest-management/fleet/add-fleet-server-kubernetes-content.asciidoc b/docs/en/ingest-management/fleet/add-fleet-server-kubernetes-content.asciidoc new file mode 100644 index 000000000..b3358ceac --- /dev/null +++ b/docs/en/ingest-management/fleet/add-fleet-server-kubernetes-content.asciidoc @@ -0,0 +1,209 @@ +// tag::quickstart-secret[] +The following command assumes you have the {es} CA available as a local file. ++ +[source, shell] +------------------------------------------------------------ +kubectl create secret generic fleet-server-ssl \ + --from-file=es-ca.crt= +------------------------------------------------------------ ++ +-- +When running the command, substitute the following values: + +* `` with your local file containing the {es} CA(s). +-- ++ +If you prefer to obtain a *yaml manifest* of the Secret to create, append `--dry-run=client -o=yaml` to the command and save the output to a file. +// end::quickstart-secret[] + +// *************************************************** +// *************************************************** + +// tag::production-secret[] +The following command assumes you have the {es} CA and the {fleet-server} certificate, key and CA available as local files. ++ +[source, shell] +------------------------------------------------------------ +kubectl create secret generic fleet-server-ssl \ + --from-file=es-ca.crt= \ + --from-file=fleet-ca.crt= \ + --from-file=fleet-server.crt= \ + --from-file=fleet-server.key= \ + --from-literal=fleet_url='' +------------------------------------------------------------ ++ +-- +When running the command, substitute the following values: + +* `` with your local file containing the {es} CA(s). +* `` with your local file containing the {fleet-server} CA. +* `` with your local file containing the server TLS certificate for the {fleet-server}. +* `` with your local file containing the server TLS key for the {fleet-server}. +* `` with the URL that points to the {fleet-server}, for example `https://fleet-svc`. This URL will be used by the {fleet-server} during its bootstrap, and its hostname must be included in the server certificate’s x509 Subject Alternative Name (SAN) list. +-- ++ +If you prefer to obtain a *yaml manifest* of the Secret to create, append `--dry-run=client -o=yaml` to the command and save the output to a file. +// end::production-secret[] + +// *************************************************** +// *************************************************** + +// tag::quickstart-deployment[] +["source","yaml",subs="attributes"] +------------------------------------------------------------ +apiVersion: v1 +kind: Service +metadata: + name: fleet-svc +spec: + type: ClusterIP + selector: + app: fleet-server + ports: + - port: 443 + protocol: TCP + targetPort: 8220 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fleet-server +spec: + replicas: 1 + selector: + matchLabels: + app: fleet-server + template: + metadata: + labels: + app: fleet-server + spec: + automountServiceAccountToken: false + containers: + - name: elastic-agent + image: docker.elastic.co/beats/elastic-agent:{version} + env: + - name: FLEET_SERVER_ENABLE + value: "true" + - name: FLEET_SERVER_ELASTICSEARCH_HOST + valueFrom: + secretKeyRef: + name: fleet-server-config + key: elastic_endpoint + - name: FLEET_SERVER_SERVICE_TOKEN + valueFrom: + secretKeyRef: + name: fleet-server-config + key: elastic_service_token + - name: FLEET_SERVER_POLICY_ID + valueFrom: + secretKeyRef: + name: fleet-server-config + key: fleet_policy_id + - name: ELASTICSEARCH_CA + value: /mnt/certs/es-ca.crt + ports: + - containerPort: 8220 + protocol: TCP + resources: {} + volumeMounts: + - name: certs + mountPath: /mnt/certs + readOnly: true + volumes: + - name: certs + secret: + defaultMode: 420 + optional: false + secretName: fleet-server-ssl +------------------------------------------------------------ +// end::quickstart-deployment[] + +// *************************************************** +// *************************************************** + +// tag::production-deployment[] +["source","yaml",subs="attributes"] +------------------------------------------------------------ +apiVersion: v1 +kind: Service +metadata: + name: fleet-svc +spec: + type: ClusterIP + selector: + app: fleet-server + ports: + - port: 443 + protocol: TCP + targetPort: 8220 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fleet-server +spec: + replicas: 1 + selector: + matchLabels: + app: fleet-server + template: + metadata: + labels: + app: fleet-server + spec: + automountServiceAccountToken: false + containers: + - name: elastic-agent + image: docker.elastic.co/beats/elastic-agent:{version} + env: + - name: FLEET_SERVER_ENABLE + value: "true" + - name: FLEET_SERVER_ELASTICSEARCH_HOST + valueFrom: + secretKeyRef: + name: fleet-server-config + key: elastic_endpoint + - name: FLEET_SERVER_SERVICE_TOKEN + valueFrom: + secretKeyRef: + name: fleet-server-config + key: elastic_service_token + - name: FLEET_SERVER_POLICY_ID + valueFrom: + secretKeyRef: + name: fleet-server-config + key: fleet_policy_id + - name: ELASTICSEARCH_CA + value: /mnt/certs/es-ca.crt + - name: FLEET_SERVER_CERT + value: /mnt/certs/fleet-server.crt + - name: FLEET_SERVER_CERT_KEY + value: /mnt/certs/fleet-server.key + - name: FLEET_CA + value: /mnt/certs/fleet-ca.crt + - name: FLEET_URL + valueFrom: + secretKeyRef: + name: fleet-server-ssl + key: fleet_url + - name: FLEET_SERVER_TIMEOUT + value: '60s' + - name: FLEET_SERVER_PORT + value: '8220' + ports: + - containerPort: 8220 + protocol: TCP + resources: {} + volumeMounts: + - name: certs + mountPath: /mnt/certs + readOnly: true + volumes: + - name: certs + secret: + defaultMode: 420 + optional: false + secretName: fleet-server-ssl +------------------------------------------------------------ +// end::production-deployment[] diff --git a/docs/en/ingest-management/fleet/add-fleet-server-kubernetes.asciidoc b/docs/en/ingest-management/fleet/add-fleet-server-kubernetes.asciidoc new file mode 100644 index 000000000..08d095bfc --- /dev/null +++ b/docs/en/ingest-management/fleet/add-fleet-server-kubernetes.asciidoc @@ -0,0 +1,449 @@ +[[add-fleet-server-kubernetes]] += Deploy {fleet-server} on Kubernetes + +[NOTE] +==== +If your {stack} is orchestrated by {eck-ref}[ECK], we recommend to deploy the {fleet-server} through the operator. That simplifies the process, as the operator automatically handles most of the resources configuration and setup steps. + +Refer to {eck-ref}/k8s-elastic-agent-fleet.html[Run Fleet-managed {agent} on ECK] for more information. +==== + +[IMPORTANT] +==== +This guide assumes familiarity with Kubernetes concepts and resources, such as `Deployments`, `Pods`, `Secrets`, or `Services`, as well as configuring applications in Kubernetes environments. +==== + +To use {fleet} for central management, a <> must +be running and accessible to your hosts. + +You can deploy {fleet-server} on Kubernetes and manage it yourself. +In this deployment model, you are responsible for high-availability, +fault-tolerance, and lifecycle management of the {fleet-server}. + +To deploy a {fleet-server} on Kubernetes and register it into {fleet} you will need the following details: + +* The *Policy ID* of a {fleet} policy configured with the {fleet-server} integration. +* A *Service token*, used to authenticate {fleet-server} with Elasticsearch. +* For outgoing traffic: +** The *{es} endpoint URL* where the {fleet-server} should connect to, configured also in the {es} output associated to the policy. +** When a private or intermediate Certificate Authority (CA) is used to sign the {es} certificate, the *{es} CA file* or the *CA fingerprint*, configured also in the {es} output associated to the policy. +* For incoming connections: +** A *TLS/SSL certificate and key* for the {fleet-server} HTTPS endpoint, used to encrypt the traffic from the {agent}s. This certificate has to be valid for the *{fleet-server} Host URL* that {agent}s use when connecting to the {fleet-server}. +* Extra TLS/SSL certificates and configuration parameters in case of requiring <> (not covered in this document). + +This document walks you through the complete setup process, organized into the following sections: + +* <> +* <> +* <> +* <> +* <> +* <> + +[discrete] +[[add-fleet-server-kubernetes-compatibility]] +== Compatibility + +{fleet-server} is compatible with the following Elastic products: + +* {stack} 7.13 or later. +** For version compatibility, {es} must be at the same or a later version than {fleet-server}, and {fleet-server} needs to be at the same or a later version than {agent} (not including patch releases). +** {kib} should be on the same minor version as {es}. + +[discrete] +[[add-fleet-server-kubernetes-prereq]] +== Prerequisites + +Before deploying {fleet-server}, you need to: + +* Prepare the SSL/TLS configuration, server certificate, <>, and needed Certificate Authorities (CAs). +* Ensure components have access to the ports needed for communication. + +[discrete] +[[add-fleet-server-kubernetes-cert-prereq]] +=== {fleet-server} and SSL/TLS certificates considerations + +This section shows the minimum requirements in terms of Transport Layer Security (TLS) certificates for the {fleet-server}, assuming no mutual TLS (mTLS) is needed. Refer to <> and <> for more information about the configuration needs of both approaches. + +There are two main traffic flows for {fleet-server}, each with different TLS requirements: + +[discrete] +[[add-fleet-server-kubernetes-cert-inbound]] +==== [{agent} → {fleet-server}] inbound traffic flow + +In this flow {fleet-server} acts as the server and {agent} acts as the client. Therefore, {fleet-server} requires a TLS certificate and key, and {agent} will need to trust the CA certificate used to sign the {fleet-server} certificate. + +[NOTE] +==== +A {fleet-server} certificate is not required when installing the server using the *Quick start* mode, but should always be used for *production* deployments. In *Quick start* mode, the {fleet-server} uses a self-signed certificate and the {agent}s have to be enrolled with the `--insecure` option. +==== + +If your organization already uses the {stack}, you may have a CA certificate that could be used to generate the new cert for the {fleet-server}. If you do not have a CA certificate, refer to <> for an example to generate a CA and a server certificate using the `elasticsearch-certutil` tool. + +[IMPORTANT] +==== +Before creating the certificate, you need to know and plan in advance the <> that the {agent} clients will use to access the {fleet-server}. This is important because the *hostname* part of the URL needs to be included in the server certificate as an `x.509 Subject Alternative Name (SAN)`. If you plan to make your {fleet-server} accessible through *multiple hostnames* or *FQDNs*, add all of them to the server certificate, and take in mind that the *{fleet-server} also needs to access the {fleet} URL during its bootstrap process*. +==== + +[discrete] +[[add-fleet-server-kubernetes-cert-outbound]] +==== [{fleet-server} → {es} output] outbound traffic flow + +In this flow, {fleet-server} acts as the client and {es} acts as the HTTPS server. For the communication to succeed, {fleet-server} needs to trust the CA certificate used to sign the {es} certificate. If your {es} cluster uses certificates signed by a corporate CA or multiple intermediate CAs you will need to use them during the {fleet-server} setup. + +[NOTE] +==== +If your {es} cluster is on Elastic Cloud or if it uses a certificate signed by a public and known CA, you won't need the {es} CA during the setup. +==== + +In summary, you need: + +* A *server certificate and key*, valid for the {fleet-server} URL. The CA used to sign this certificate will be needed by the {agent} clients and the {fleet-server} itself. + +* The *CA certificate* (or certificates) associated to your {es} cluster, except if you are sure your {es} certificate is fully trusted publicly. + +[discrete] +[[default-port-assignments-kubernetes]] +=== Default port assignments + +When {es} or {fleet-server} are deployed, components communicate over well-defined, pre-allocated ports. +You may need to allow access to these ports. Refer to the following table for default port assignments: + +|=== +| Component communication | Default port +| {agent} → {fleet-server} | 8220 +| {fleet-server} → {es} | 9200 +| {fleet-server} → {kib} (optional, for {fleet} setup) | 5601 +| {agent} → {es} | 9200 +| {agent} → Logstash | 5044 +| {agent} → {kib} (optional, for {fleet} setup) | 5601 +|=== + +In Kubernetes environments, you can adapt these ports without modifying the listening ports of the {fleet-server} or other applications, as traffic is managed by Kubernetes `Services`. This guide includes an example where {agent}s connect to the {fleet-server} through port `443` instead of the default `8220`. + +[discrete] +[[add-fleet-server-kubernetes-add-server]] +== Add {fleet-server} + +A {fleet-server} is an {agent} that is enrolled in a {fleet-server} policy. The policy configures the agent to operate in a special mode to serve as a {fleet-server} in your deployment. + +[discrete] +[[add-fleet-server-kubernetes-preparations]] +=== {fleet} preparations + +[TIP] +==== +If you already have a {fleet} policy with the {fleet-server} integration, you know its ID, and you know how to generate an {ref}/service-tokens-command.html[{es} service token] for the {fleet-server}, skip directly to <>. + +Also note that the `service token` required by the {fleet-server} is different from the `enrollment tokens` used by {agent}s to enroll to {fleet}. +==== + +. In {kib}, open *{fleet} → Settings* and ensure the *Elasticsearch output* that will be used by the {fleet-server} policy is correctly configured, paying special attention that: ++ +** The *hosts* field includes a valid URL that will be reachable by the {fleet-server} Pod(s). +** If your {es} cluster uses certificates signed by private or intermediate CAs not publicly trusted, you have added the trust information in the *Elasticsearch CA trusted fingerprint* field or in the *advanced configuration* section through the `ssl.certificate_authorities` setting. For an example, refer to https://elastic.co/guide/en/fleet/current/secure-connections.html#_encrypt_traffic_between_elastic_agents_fleet_server_and_elasticsearch[Secure Connections] documentation. ++ +[IMPORTANT] +==== +This validation step is critical. The {es} host URL and CA information has to be added *in both the {es} output and the environment variables* provided to the {fleet-server}. It's a common mistake to ignore the output settings believing that the environment variables will prevail, when the environment variables are only used during the bootstrap of the {fleet-server}. + +If the URL that {fleet-server} will use to access {es} is different from the {es} URL used by other clients, you may want to create a dedicated *{es} output* for {fleet-server}. +==== + +. Go to *{fleet} → Agent Policies* and select *Create agent policy* to create a policy for the {fleet-server}: ++ +** Set a *name* for the policy, for example `Fleet Server Policy Kubernetes`. +** Do *not* select the option *Collect system logs and metrics*. This option adds the System integration to the {agent} policy. Because {fleet-server} will run as a Kubernetes Pod without any visibility to the Kubernetes node, there won't be a system to monitor. +** Select the **output** that the {fleet-server} needs to use to contact {es}. This should be the output that you verified in the previous step. +** Optionally, you can set the **inactivity timeout** and **inactive agent unenrollment timeout** parameters to automatically unenroll and invalidate API keys after the {fleet-server} agents become inactive. This is especially useful in Kubernetes environments, where {fleet-server} Pods are ephemeral, and new {agent}s appear in {fleet} UI after Pod recreations. + +. Open the created policy, and from the *Integrations* tab select *Add integration*: ++ +** Search for and select the {fleet-server} integration. +** Select *Add {fleet-server}* to add the integration to the {agent} policy. ++ +At this point you can configure the integration settings per <>. +** When done, select *Save and continue*. Do not add an {agent} at this stage. + +. Open the configured policy, which now includes the {fleet-server} integration, and select *Actions* → *Add {fleet-server}*. In the next dialog: ++ +* Confirm that the *policy for {fleet-server}* is properly selected. +* *Choose a deployment mode for security*: +** If you select *Quick start*, the {fleet-server} generates a self-signed TLS certificate, and subsequent agents should be enrolled using the `--insecure` flag. +** If you select *Production*, you provide a TLS certificate, key and CA to the {fleet-server} during the deployment, and subsequent agents will need to trust the certificate's CA. +* Add your *{fleet-server} Host* information. This is the URL that clients ({agent}s) will use to connect to the {fleet-server}: +** In *Production* mode, the {fleet-server} certificate must include the hostname part of the URL as an `x509 SAN`, and the {fleet-server} itself will need to access that URL during its bootstrap process. +** On Kubernetes environments this could be the name of the `Kubernetes service` or reverse proxy that exposes the {fleet-server} Pods. +** In the provided example we use `https://fleet-svc.` as the URL, which corresponds to the Kubernetes service DNS resolution. +* Select **generate service token** to create a token for the {fleet-server}. +* From *Install {fleet-server} to a centralized host → Linux*, take note of the values of the following settings that will be needed for the {fleet-server} installation: +** Service token(specified by `--fleet-server-service-token` parameter). +** {fleet} policy ID (specified by `--fleet-server-policy` parameter). +** {es} URL (specified by `--fleet-server-es` parameter). + +. Keep the {kib} browser window open and continue with the <>. ++ +When the {fleet-server} installation has succeeded, the *Confirm Connection* UI will show a *Connected* status. + +[discrete] +[[add-fleet-server-kubernetes-install]] +=== {fleet-server} installation + +[discrete] +[[add-fleet-server-kubernetes-install-overview]] +==== Installation overview + +To deploy {fleet-server} on Kubernetes and enroll it into {fleet} you need the following details: + +* *Policy ID* of the {fleet} policy configured with the {fleet-server} integration. +* *Service token*, that you can generate following the <> or manually using the {ref}/service-tokens-command.html[{es}-service-tokens command]. +* *{es} endpoint URL*, configured in both the {es} output associated to the policy and in the Fleet Server as an environment variable. +* *{es} CA certificate file*, configured in both the {es} output associated to the policy and in the Fleet Server. +* {fleet-server} *certificate and key* (for *Production* deployment mode only). +* {fleet-server} *CA certificate file* (for *Production* deployment mode only). +* {fleet-server} URL (for *Production* deployment mode only). + +If you followed the <> and <> you should have everything ready to proceed with the {fleet-server} installation. + +The suggested deployment method for the {fleet-server} consists of: + +* A Kubernetes Deployment manifest that relies on two Secrets for its configuration: +** A Secret named `fleet-server-config` with the main configuration parameters, such as the service token, the {es} URL and the policy ID. +** A Secret named `fleet-server-ssl` with all needed certificate files and the {fleet-server} URL. +* A Kubernetes ClusterIP Service named `fleet-svc` that exposes the {fleet-server} on port 443, making it available at URLs like `https://fleet-svc`, `https://fleet-svc.` and `https://fleet-svc..svc`. + +Adapt and change the suggested manifests and deployment strategy to your needs, ensuring you feed the {fleet-server} with the needed configuration and certificates. For example, you can customize: + +* CPU and memory `requests` and `limits`. Refer to <> for more information about {fleet-server} resources utilization. +* Scheduling configuration, such as `affinity rules` or `tolerations`, if needed in your environment. +* Number of replicas, to scale the Fleet Server horizontally. +* Use an {es} CA fingerprint instead of a CA file. +* Configure other <>. + +[discrete] +[[add-fleet-server-kubernetes-install-steps]] +==== Installation Steps + +. Create the Secret for the {fleet-server} configuration. ++ +[source, shell] +----- +kubectl create secret generic fleet-server-config \ +--from-literal=elastic_endpoint='' \ +--from-literal=elastic_service_token='' \ +--from-literal=fleet_policy_id='' +----- ++ +When running the command, substitute the following values: ++ +-- +* ``: Replace this with the URL of your {es} host, for example `'https://monitoring-es-http.default.svc:9200'`. +* ``: Use the service token provided by {kib} in the {fleet} UI. +* ``: Replace this with the ID of the created policy, for example `'dee949ac-403c-4c83-a489-0122281e4253'`. +-- ++ +If you prefer to obtain a *yaml manifest* of the Secret to create, append `--dry-run=client -o=yaml` to the command and save the output to a file. + +. Create the Secret for the TLS/SSL configuration: ++ +++++ +
+
+ + +
+
+++++ ++ +include::add-fleet-server-kubernetes-content.asciidoc[tag=quickstart-secret] ++ +++++ +
+ +
+++++ ++ +If your {es} cluster runs on Elastic Cloud or if it uses a publicly trusted CA, remove the `es-ca.crt` key from the proposed secret. + +. Save the proposed Deployment manifest locally, for example as `fleet-server-dep.yaml`, and adapt it to your needs: ++ +++++ +
+
+ + +
+
+++++ ++ +include::add-fleet-server-kubernetes-content.asciidoc[tag=quickstart-deployment] ++ +++++ +
+ +
+++++ ++ +Manifest considerations: ++ +* If your {es} cluster runs on Elastic Cloud or if it uses a publicly trusted CA, remove the `ELASTICSEARCH_CA` environment variable from the manifest. +* Check the `image` version to ensure its aligned with the rest of your {stack}. +* Keep `automountServiceAccountToken` set to `false` to disable the <>. +* Consider configuring requests and limits always as a best practice. Refer to <> for more information about resources utilization of the {fleet-server}. +* You can change the listening `port` of the service to any port of your choice, but do not change the `targetPort`, as the {fleet-server} Pods will listen on port 8220. +* If you want to expose the {fleet-server} externally, consider changing the service type to `LoadBalancer`. + +. Deploy the configured manifest to create the {fleet-server} and service: ++ +[source, shell] +------------------------------------------------------------ +kubectl apply -f fleet-server-dep.yaml +------------------------------------------------------------ ++ +[IMPORTANT] +==== +Ensure the `Service`, the `Deployment` and all the referenced `Secrets` are created in the *same Namespace*. +==== + +. Check the {fleet-server} Pod logs for errors and confirm in {kib} that the {fleet-server} agent appears as `Connected` and `Healthy` in *{kib} → {fleet}*. ++ +[source, shell] +------------------------------------------------------------ +kubectl logs fleet-server-69499449c7-blwjg +------------------------------------------------------------ ++ +It can take a couple of minutes for {fleet-server} to fully start. If you left the {kib} browser window open during <> it will show *Connected* when everything has gone well. ++ +[NOTE] +==== +In *Production mode*, during {fleet-server} bootstrap process, the {fleet-server} might be unable to access its own `FLEET_URL`. This is usually a temporary issue caused by the Kubernetes Service not forwarding traffic to the Pod(s). + +If the issue persists consider using `https://localhost:8220` as the `FLEET_URL` for the {fleet-server} configuration, and ensure that `localhost` is included in the certificate's SAN. +==== + +[discrete] +[[add-fleet-server-kubernetes-expose]] +== Expose the {fleet-server} to {agent}s + +This may include the creation of a Kubernetes `service`, an `ingress` resource, and / or DNS registers for FQDNs resolution. There are multiple ways to expose applications in Kubernetes. + +Considerations when exposing {fleet-server}: + +* If your environment requires the {fleet-server} to be reachable through multiple hostnames or URLs, you can create multiple *{fleet-server} Hosts* in *{fleet} → Settings*, and create different policies for different groups of agents. +* Remember that in *Production* mode, the *hostnames* used to access the {fleet-server} must be part of the {fleet-server} certificate as `x.509 Subject Alternative Names`. +* *Align always the service listening port to the URL*. If you configure the service to listen in port 8220 use a URL like `https://service-name:8220`, and if it listens in `443` use a URL like `https://service-name`. + +Below is an end to end example of how to expose the server to external and internal clients using a LoadBalancer service. For this example we assume the following: + +-- +* The {fleet-server} runs in a namespace called `elastic`. +* External clients will access {fleet-server} using a URL like `https://fleet.example.com`, which will be resolved in DNS to the external IP of the Load Balancer. +* Internal clients will access {fleet-server} using the Kubernetes service directly `https://fleet-svc-lb.elastic`. +* The server certificate has both hostnames (`fleet.example.com` and `fleet-svc-lb.elastic`) in its SAN list. +-- + +. Create the `LoadBalancer` Service ++ +[source, shell] +------------------------------------------------------------ +kubectl expose deployment fleet-server --name fleet-svc-lb --type LoadBalancer --port 443 --target-port 8220 +------------------------------------------------------------ ++ +That command creates a service named `fleet-svc-lb`, listening on port `443` and forwarding the traffic to the `fleet-server` deployment's Pods on port `8220`. The listening `--port` (and the consequent URL) of the service can be customized, but the `--target-port` must remain on the default port (`8220`), because it's the port used by the {fleet-server} application. + +. Add `https://fleet-server.example.com` and `https://fleet-svc-lb.elastic` as a new *{fleet-server} Hosts* in *{fleet} → Settings*. Align the port of the URLs if you configured something different from `443` in the Load Balancer. + +. Create a {fleet} policy for external clients using the `https://fleet-server.example.com` {fleet-server} URL. + +. Create a {fleet} policy for internal clients using the `https://fleet-svc-lb.elastic` {fleet-server} URL. + +. You are ready now to enroll external and internal agents to the relevant policies. Refer to <> for more details. + +[discrete] +[[add-fleet-server-kubernetes-troubleshoot]] +== Troubleshoot {fleet-server} + +[discrete] +[[add-fleet-server-kubernetes-troubleshoot-common]] +=== Common Problems + +The following issues may occur when {fleet-server} settings are missing or configured incorrectly: + +* {fleet-server} is trying to access {es} at `localhost:9200` even though the `FLEET_SERVER_ELASTICSEARCH_HOST` environment variable is properly set. ++ +This problem occurs when the `output` of the policy associated to the {fleet-server} is not correctly configured. + +* TLS certificate trust issues occur even when the `ELASTICSEARCH_CA` environment variable is properly set during deployment. ++ +This problem occurs when the `output` of the policy associated to the {fleet-server} is not correctly configured. Add the *CA certificate* or *CA trusted fingerprint* to the {es} output associated to the {fleet-server} policy. + +* In *Production mode*, {fleet-server} enrollment fails due to `FLEET_URL` not being accessible, showing something similar to: ++ +[source, sh] +------------------------------------------------------------ +Starting enrollment to URL: https://fleet-svc/ +1st enrollment attempt failed, retrying enrolling to URL: https://fleet-svc/ with exponential backoff (init 1s, max 10s) +Error: fail to enroll: fail to execute request to fleet-server: dial tcp 34.118.226.212:443: connect: connection refused +Error: enrollment failed: exit status 1 +------------------------------------------------------------ ++ +If the service and URL are correctly configured, this is usually a temporary issue caused by the Kubernetes Service not forwarding traffic to the Pod, and it should be cleared in a couple of restarts. ++ +As a workaround, consider using `https://localhost:8220` as the `FLEET_URL` for the {fleet-server} configuration, and ensure that `localhost` is included in the certificate's SAN. + +[discrete] +[[add-fleet-server-kubernetes-next]] +== Next steps + +Now you're ready to add {agent}s to your host systems. +To learn how, refer to <>, or <> if your {agent}s will also run on Kubernetes. + +When you connect {agent}s to {fleet-server}, remember to use the `--insecure` flag if the *quick start* mode was used, or to provide to the {agent}s the CA certificate associated to the {fleet-server} certificate if *production* mode was used. \ No newline at end of file diff --git a/docs/en/ingest-management/fleet/add-fleet-server-mixed.asciidoc b/docs/en/ingest-management/fleet/add-fleet-server-mixed.asciidoc index 5254c0660..aaacc777f 100644 --- a/docs/en/ingest-management/fleet/add-fleet-server-mixed.asciidoc +++ b/docs/en/ingest-management/fleet/add-fleet-server-mixed.asciidoc @@ -33,8 +33,7 @@ you need to: {fleet-server} is compatible with the following Elastic products: * {stack} 7.13 or later -** For version compatibility: {es} >= {fleet-server} >= {agent} (except for -bugfix releases) +** For version compatibility, {es} must be at the same or a later version than {fleet-server}, and {fleet-server} needs to be at the same or a later version than {agent} (not including patch releases). ** {kib} should be on the same minor version as {es} * {ece} 2.9 or later--allows you to use a hosted {fleet-server} on {ecloud}. diff --git a/docs/en/ingest-management/fleet/add-fleet-server-on-prem.asciidoc b/docs/en/ingest-management/fleet/add-fleet-server-on-prem.asciidoc index a5ebe77e6..07240376a 100644 --- a/docs/en/ingest-management/fleet/add-fleet-server-on-prem.asciidoc +++ b/docs/en/ingest-management/fleet/add-fleet-server-on-prem.asciidoc @@ -39,8 +39,7 @@ containerized {fleet-server}. {fleet-server} is compatible with the following Elastic products: * {stack} 7.13 or later. -** For version compatibility: {es} >= {fleet-server} >= {agent} (except for -bugfix releases) +** For version compatibility, {es} must be at the same or a later version than {fleet-server}, and {fleet-server} needs to be at the same or a later version than {agent} (not including patch releases). ** {kib} should be on the same minor version as {es}. * {ece} 2.9 or later @@ -99,6 +98,9 @@ You may need to allow access to these ports. Refer to the following table for de | {fleet-server} → {es} | 9200 |=== +NOTE: Connectivity to {kib} on port 5601 is optional and not required at all times. {agent} and {fleet-server} may need to connect to {kib} if deployed in a +container environment where an enrollment token can not be provided during deployment. + //[discrete] //[[add-fleet-server-on-prem-hosts]] //= Add {fleet-server} hosts diff --git a/docs/en/ingest-management/fleet/air-gapped.asciidoc b/docs/en/ingest-management/fleet/air-gapped.asciidoc index 696d4929a..62ed27aec 100644 --- a/docs/en/ingest-management/fleet/air-gapped.asciidoc +++ b/docs/en/ingest-management/fleet/air-gapped.asciidoc @@ -36,6 +36,51 @@ Set the following property in {kib} to enable air-gapped mode in {fleet}. This a xpack.fleet.isAirGapped: true ---- +[discrete] +[[air-gapped-pgp-fleet]] +== Configure {agents} to download a PGP/GPG key from {fleet-server} + +Starting from version 8.9.0, when {agent} tries to perform an upgrade, it first verifies the binary signature with the key bundled in the agent. This process has a backup mechanism that will use the key coming from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` instead of the one it already has. + +In an air-gapped environment, an {agent} which doesn't have access to a PGP/GPG key from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` would fail to be upgraded. +For versions 8.9.0 to 8.10.3, you can resolve this problem following the steps described in the associated link:https://www.elastic.co/guide/en/fleet/8.9/release-notes-8.9.0.html#known-issues-8.9.0[known issue] documentation. + +Starting in version 8.10.4, you can resolve this problem by configuring {agents} to download the PGP/GPG key from {fleet-server}. + +Starting in version 8.10.4, {agent} will: + +. Verify the binary signature with the key bundled in the agent. +. If the verification doesn't pass, the agent will download the PGP/GPG key from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` and verify it. +. If that verification doesn't pass, the agent will download the PGP/GPG key from {fleet-server} and verify it. +. If that verification doesn't pass, the upgrade is blocked. + +By default, {fleet-server} serves {agents} with the key located in `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp`. +The key is served through the {fleet-server} endpoint `GET /api/agents/upgrades/{major}.{minor}.{patch}/pgp-public-key`. + +If there isn't a `default.pgp` key in the `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp` directory, {fleet-server} instead will attempt to retrieve a PGP/GPG key from the URL that you can specify with the `server.pgp.upstream_url` setting. + +You can prevent {fleet} from downloading the PGP/GPG key from `server.pgp.upstream_url` by manually downloading it from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` and storing it at `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp`. + +To set a custom URL for {fleet-server} to access a PGP/GPG key and make it available to {agents}: + +. In {kib}, go to *Management > {fleet} > Agent policies*. +. Select a policy for the agents that you want to upgrade. +. On the policy page, in the **Actions** menu for the {fleet-server} integration, select **Edit integration**. +. In the {fleet-server} settings section expand **Change defaults** and **Advanced options**. +. In the **Custom fleet-server configurations** field, add the setting `server.pgp.upstream_url` with the full URL where the PGP/GPG key can be accessed. For example: + +[source,yaml] +---- +server.pgp.upstream_url: +---- + +The setting `server.pgp.upstream_url` must point to a web server hosting the PGP/GPG key, which must be reachable by the host where {fleet-server} is installed. + +Note that: + + * `server.pgp.upstream_url` may be specified as an `http` endpoint (instead of `https`). + * For an `https` endpoint, the CA for {fleet-server} to connect to `server.pgp.upstream_url` must be trusted by {fleet-server} using the `--certificate-authorities` setting that is used globally for {agent}. + [discrete] [[air-gapped-proxy-server]] == Use a proxy server to access the {package-registry} @@ -138,6 +183,9 @@ docker run -it -p 8080:8080 \ --health-cmd "curl -f -L http://127.0.0.1:8080/health" \ docker.elastic.co/package-registry/distribution:{version} ---- ++ +You can use the `/health` or `/health?ready=true` API endpoints to check if EPR is ready to serve requests. +When either endpoint returns a `200` HTTP status code, the service is ready to handle requests. [discrete] [[air-gapped-diy-epr-kibana]] @@ -259,7 +307,7 @@ sub-directories that follow the same convention used by the {artifact-registry}: Where: + * `` is in the format `beats/elastic-agent`, `fleet-server`, `endpoint-dev`, and so on. -* `` is om the format `elastic-agent`, `endpoint-security`, or `fleet-server` and so on. +* `` is in the format `elastic-agent`, `endpoint-security`, or `fleet-server` and so on. * `arch-package-type` is in the format `linux-x86_64`, `linux-arm64`, `windows_x86_64`, `darwin_x86_64`, or darwin_aarch64`. * If you're using the DEB package manager: diff --git a/docs/en/ingest-management/fleet/api-generated/rules/fleet-apis-passthru.asciidoc b/docs/en/ingest-management/fleet/api-generated/rules/fleet-apis-passthru.asciidoc index e0f13e930..12b8e18f8 100644 --- a/docs/en/ingest-management/fleet/api-generated/rules/fleet-apis-passthru.asciidoc +++ b/docs/en/ingest-management/fleet/api-generated/rules/fleet-apis-passthru.asciidoc @@ -7436,6 +7436,7 @@ Any modifications made to this file will be overwritten.
policy_id
String new agent policy id
+
batchSize (optional)
Big Decimal
agents
bulk_reassign_agents_request_agents
diff --git a/docs/en/ingest-management/fleet/enrollment-handling-containerized-agent.asciidoc b/docs/en/ingest-management/fleet/enrollment-handling-containerized-agent.asciidoc new file mode 100644 index 000000000..05dc474b3 --- /dev/null +++ b/docs/en/ingest-management/fleet/enrollment-handling-containerized-agent.asciidoc @@ -0,0 +1,12 @@ +[[containers-agent-enrollment-handling]] += Enrollment handing for containerized {fleet}-managed agents +++++ +Enrollment handing for containerized agents +++++ + +Beginning with version 8.18, for {fleet}-managed {agents} that run in a containerized environment (including Docker, Kubernetes, and others), enrollment handling is managed as follows: + +* **Enrollment Verification:** The {agent} checks the stored enrollment conditions within its container environment and re-enrolls only when necessary. +* **Unenrollment Handling:** If an {agent} is unenrolled via the {fleet} UI but still references a valid enrollment token provided through environment variables, it will automatically re-enroll on the next container restart. + +In versions lower than 8.18, an unenrolled agent remains unenrolled and does not re-enroll, even if a valid enrollment token is still available. diff --git a/docs/en/ingest-management/fleet/filter-agent-list-by-tags.asciidoc b/docs/en/ingest-management/fleet/filter-agent-list-by-tags.asciidoc index 87aafb3e3..c75807661 100644 --- a/docs/en/ingest-management/fleet/filter-agent-list-by-tags.asciidoc +++ b/docs/en/ingest-management/fleet/filter-agent-list-by-tags.asciidoc @@ -99,7 +99,7 @@ docker run \ --env FLEET_URL= \ --env FLEET_ENROLLMENT_TOKEN= \ --env ELASTIC_AGENT_TAGS=docker,dev - --rm docker.elastic.co/beats/elastic-agent:{version} + --rm docker.elastic.co/elastic-agent/elastic-agent:{version} ---- For more information about running on containers, refer to the guides under diff --git a/docs/en/ingest-management/fleet/fleet-api-docs.asciidoc b/docs/en/ingest-management/fleet/fleet-api-docs.asciidoc index 69987b502..9571baf49 100644 --- a/docs/en/ingest-management/fleet/fleet-api-docs.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-api-docs.asciidoc @@ -1,23 +1,8 @@ [[fleet-api-docs]] = {kib} {fleet} APIs -You can find details for all available {fleet} API endpoints in our generated -<>. This documentation is experimental and may be -incomplete or change later. - -The main source of truth for the {fleet} API can be found in the -https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/common/openapi/bundled.json[self-contained spec file] -that you can use to generate docs using Swagger or a similar tool. -For more information, refer to the -https://github.com/elastic/kibana/tree/main/x-pack/plugins/fleet/common/openapi/README.md[{fleet} OpenAPI readme]. - -//However we do provide a self-contained spec file that you can -//https://petstore.swagger.io/?url=https://raw.githubusercontent.com/elastic/kibana/{branch}/x-pack/plugins/fleet/common/openapi/bundled.json[view in Swagger] -//(or a similar tool) to explore our Fleet APIs. - -//For more information, refer to the -//https://github.com/elastic/kibana/blob/{branch}/x-pack/plugins/fleet/common/openapi/README.md[{fleet} OpenAPI file] -//in the {kib} repository. +You can find details for all available {fleet} API endpoints in our generated +{api-kibana}[Kibana API docs]. In this section, we provide examples of some commonly used {fleet} APIs. diff --git a/docs/en/ingest-management/fleet/fleet-deployment-models.asciidoc b/docs/en/ingest-management/fleet/fleet-deployment-models.asciidoc index fca0c334f..3ef712bd7 100644 --- a/docs/en/ingest-management/fleet/fleet-deployment-models.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-deployment-models.asciidoc @@ -18,8 +18,11 @@ If you're running {es} and {kib} hosted on {cloud}/ec-getting-started.html[{ess} {ess} with {fleet-server} on-premise:: When you use a hosted {ess} deployment you may still choose to run {fleet-server} on-premise. For details about this deployment model and set up instructions, refer to <>. +Docker and Kubernetes:: +You can deploy {fleet}-managed {agent} in Docker or on Kubernetes. Refer to <> or <> for all of the configuration instructions. For a Kubernetes install we also have a <> available to simplify the installation. Details for configuring {fleet-server} are included with the {agent} install steps. + {eck}:: -You can deploy {fleet}-managed {agent} in a Kubernetes environment that provides configuration and management capabilities for the full {stack}. For details, refer to {eck-ref}/k8s-elastic-agent-fleet.html[Run {fleet}-managed {agent} on ECK]. +You can deploy {fleet}-managed {agent} in an {ecloud} Kubernetes environment that provides configuration and management capabilities for the full {stack}. For details, refer to {eck-ref}/k8s-elastic-agent-fleet.html[Run {fleet}-managed {agent} on ECK]. Self-managed:: For self-managed deployments, you must install and host {fleet-server} yourself. For details about this deployment model and set up instructions, refer to <>. diff --git a/docs/en/ingest-management/fleet/fleet-server-scaling.asciidoc b/docs/en/ingest-management/fleet/fleet-server-scaling.asciidoc index 8b4798de5..813a4af0c 100644 --- a/docs/en/ingest-management/fleet/fleet-server-scaling.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-server-scaling.asciidoc @@ -232,7 +232,9 @@ of these operations. **{agent} policies** -A single instance of {fleet} supports a maximum of 500 {agent} policies. If more policies are configured, UI performance might be impacted. +A single instance of {fleet} supports a maximum of 1000 {agent} policies. If more policies are configured, UI performance might be impacted. The maximum number of policies is not affected by the number of spaces in which the policies are used. + +If you are using {agent} with link:{serverless-docs}[{serverless-full}], the maximum supported number of {agent} policies is 500. **{agents}** diff --git a/docs/en/ingest-management/fleet/fleet-server-secrets.asciidoc b/docs/en/ingest-management/fleet/fleet-server-secrets.asciidoc index 1ea538db7..25a8fd95f 100644 --- a/docs/en/ingest-management/fleet/fleet-server-secrets.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-server-secrets.asciidoc @@ -80,7 +80,7 @@ inputs: ssl.key_passphrase_path: /path/to/passphrase ---- + -When you are running {fleet-server} under {agent}, you can provide the passphrase path using the `--fleet-server-cert-key-passphrase-path` flag. +When you are running {fleet-server} under {agent}, you can provide the passphrase path using the `--fleet-server-cert-key-passphrase` flag. See <> for more details. + If you are <>, you can use the environment variable `FLEET_SERVER_CERT_KEY_PASSPHRASE` to specify the file path. @@ -210,7 +210,7 @@ docker run \ -v /path/to/creds:/creds:ro \ -e FLEET_SERVER_CERT_KEY_PASSPHRASE=/creds/passphrase \ -e FLEET_SERVER_SERVICE_TOKEN_PATH=/creds/service-token \ - --rm docker.elastic.co/beats/elastic-agent + --rm docker.elastic.co/elastic-agent/elastic-agent ---- === Secrets in Kubernetes @@ -244,7 +244,7 @@ spec: secretName: fleet-server-service-token containers: - name: fleet-server - image: docker.elastic.co/beats/elastic-agent + image: docker.elastic.co/elastic-agent/elastic-agent volumeMounts: - name: key-passphrase mountPath: /var/secrets/passphrase diff --git a/docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc b/docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc index d7f7b1a29..806d7c0cb 100644 --- a/docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc @@ -5,6 +5,8 @@ Specify these settings to send data over a secure connection to Kafka. In the {fleet} <>, make sure that the Kafka output type is selected. +NOTE: If you plan to use {ls} to modify {agent} output data before it's sent to Kafka, please refer to our <> for doing so, further in on this page. + [discrete] == General settings @@ -17,7 +19,7 @@ Specify these settings to send data over a secure connection to Kafka. In the {f | The Kafka protocol version that {agent} will request when connecting. Defaults to `1.0.0`. Currently Kafka versions from `0.8.2.0` to `2.6.0` are supported, however the latest -Kafka version (`3.x.x`) is expected to be compatible when version `2.6.0` is selected. +Kafka version (`3.x.x`) is expected to be compatible when version `2.6.0` is selected. When using Kafka 4.0 and newer, the version must be set to at least `2.1.0`. // ============================================================================= @@ -191,7 +193,18 @@ Use this option to set the Kafka topic for each {agent} event. [id="kafka-output-topics-default"] **Default topic** -| Set a default topic to use for events sent by {agent} to the Kafka output, for example `elastic-agent`. +| Set a default topic to use for events sent by {agent} to the Kafka output. + +You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an {ecs-ref}/ecs-reference.html[Elastic Common Scheme (ECS)] field. Available fields include: + +* `data_stream_type` +* `data_stream.dataset` +* `data_stream.namespace` +* `@timestamp` +* `event-dataset` + +You can also set a custom field. This is useful if you're using the <> as part of your {agent} input. +Otherwise, setting a custom field is not recommended. |=== @@ -531,4 +544,33 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[ |=== +[[kafka-output-settings-ls-warning]] +== Kafka output and using {ls} to index data to {es} + +If you are considering using {ls} to ship the data from `kafka` to {es}, please +be aware the structure of the documents sent from {agent} to `kafka` must not be modified by {ls}. +We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec in order +to make sure the input doesn't edit the fields and their contents. + +The data streams setup by the integrations expect to receive events having the same structure and +field names as they were sent directly from an {agent}. + +The structure of the documents sent from {agent} to `kafka` must not be modified by {ls}. +We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec. + +Refer to the <> documentation for more details. + +[source,yaml] +---- +inputs { + kafka { + ... + ecs_compatibility => "disabled" + codec => json { ecs_compatibility => "disabled" } + ... + } +} +... +---- + :type!: diff --git a/docs/en/ingest-management/fleet/fleet-settings-output-logstash.asciidoc b/docs/en/ingest-management/fleet/fleet-settings-output-logstash.asciidoc index b96778f78..66fdea424 100644 --- a/docs/en/ingest-management/fleet/fleet-settings-output-logstash.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-settings-output-logstash.asciidoc @@ -13,6 +13,44 @@ Before using the {ls} output, you need to make sure that for any integrations th To learn how to generate certificates, refer to <>. +To receive the events in {ls}, you also need to create a {ls} configuration pipeline. +The {ls} configuration pipeline listens for incoming {agent} connections, +processes received events, and then sends the events to {es}. + +The following example configures a {ls} pipeline that listens on port `5044` for +incoming {agent} connections and routes received events to {es}. + +The {ls} pipeline definition below is an example. Please refer to the `Additional Logstash +configuration required` steps when creating the {ls} output in the Fleet outputs page. + +[source,yaml] +---- +input { + elastic_agent { + port => 5044 + enrich => none # don't modify the events' schema at all + ssl => true + ssl_certificate_authorities => [""] + ssl_certificate => "" + ssl_key => "" + ssl_verify_mode => "force_peer" + } +} +output { + elasticsearch { + hosts => ["http://localhost:9200"] <1> + # cloud_id => "..." + data_stream => "true" + api_key => "" <2> + data_stream => true + ssl => true + # cacert => "" + } +} +---- +<1> The {es} server and the port (`9200`) where {es} is running. +<2> The API Key obtained from the {ls} output creation steps in Fleet. + [cols="2*> as your main {es} cluster. -NOTE: Using a remote {es} output with a target cluster that has {cloud}/ec-traffic-filtering-deployment-configuration.html[traffic filters] enabled is not currently supported. +[NOTE] +==== +Note the following restrictions with the remote {es} output: + +* Using a remote {es} output with a target cluster that has {cloud}/ec-traffic-filtering-deployment-configuration.html[traffic filters] enabled is not currently supported. +* Using {elastic-defend} is currently not supported when a remote {es} output is configured for an agent. +==== To configure a remote {es} cluster for your {agent} data: @@ -62,3 +68,5 @@ After the output is created, you can update an {agent} policy to use the new rem The remote {es} cluster is now configured. As a final step before using the remote {es} output, you need to make sure that for any integrations that have been <>, the integration assets have been installed on the remote {es} cluster. Refer to <> for the steps. + +NOTE: When you use a remote {es} output, {fleet-server} performs a test to ensure connectivity to the remote cluster. The result of that connectivity test is used to report the ES Remote output as healthy or unhealthy on the **Fleet** > **Settings** > **Outputs** page, under the **Status** column. In some cases, the remote {es} output used for data from {agent} may be reachable only by those agents and not by {fleet-server}, so the unhealthy state and an associated `Unable to connect` error that appears on the UI can be ignored. \ No newline at end of file diff --git a/docs/en/ingest-management/fleet/fleet-settings.asciidoc b/docs/en/ingest-management/fleet/fleet-settings.asciidoc index 6a47720d7..f68cfe47f 100644 --- a/docs/en/ingest-management/fleet/fleet-settings.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-settings.asciidoc @@ -132,3 +132,16 @@ in the agent policy. You can specify a proxy server to be used in {fleet-server}, {agent} outputs, or for any agent binary download sources. For full details about proxy configuration refer to <>. + +[discrete] +[[delete-unenrolled-agents-setting]] +== Delete unenrolled agents + +After an {agent} has been unenrolled in {fleet}, a number of documents about the agent are retained just in case the agent needs to be recovered at some point. You can choose to have all data related to an unenrolled agent deleted automatically. + +Note that this option can also be enabled by adding the `xpack.fleet.enableDeleteUnenrolledAgents: true` setting to the {kibana-ref}/[{kib} settings file]. + +To enable automatic deletion of unenrolled agents: + +. Go to **{fleet} -> Settings**. +. Under **Advanced Settings**, enable the **Delete unenrolled agents** option. \ No newline at end of file diff --git a/docs/en/ingest-management/fleet/images/collect-agent-diagnostics.png b/docs/en/ingest-management/fleet/images/collect-agent-diagnostics.png deleted file mode 100644 index 4e9647881..000000000 Binary files a/docs/en/ingest-management/fleet/images/collect-agent-diagnostics.png and /dev/null differ diff --git a/docs/en/ingest-management/fleet/images/collect-agent-diagnostics1.png b/docs/en/ingest-management/fleet/images/collect-agent-diagnostics1.png new file mode 100644 index 000000000..b0c39fadc Binary files /dev/null and b/docs/en/ingest-management/fleet/images/collect-agent-diagnostics1.png differ diff --git a/docs/en/ingest-management/fleet/images/collect-agent-diagnostics2.png b/docs/en/ingest-management/fleet/images/collect-agent-diagnostics2.png new file mode 100644 index 000000000..b49d0734a Binary files /dev/null and b/docs/en/ingest-management/fleet/images/collect-agent-diagnostics2.png differ diff --git a/docs/en/ingest-management/fleet/images/fleet-server-agent-policy-page.png b/docs/en/ingest-management/fleet/images/fleet-server-agent-policy-page.png index 1875c445c..fed328d88 100644 Binary files a/docs/en/ingest-management/fleet/images/fleet-server-agent-policy-page.png and b/docs/en/ingest-management/fleet/images/fleet-server-agent-policy-page.png differ diff --git a/docs/en/ingest-management/fleet/images/fleet-server-configuration.png b/docs/en/ingest-management/fleet/images/fleet-server-configuration.png index 1b6861bd1..a730b4c3c 100644 Binary files a/docs/en/ingest-management/fleet/images/fleet-server-configuration.png and b/docs/en/ingest-management/fleet/images/fleet-server-configuration.png differ diff --git a/docs/en/ingest-management/fleet/monitor-elastic-agent.asciidoc b/docs/en/ingest-management/fleet/monitor-elastic-agent.asciidoc index 087133a9e..d4cedef8b 100644 --- a/docs/en/ingest-management/fleet/monitor-elastic-agent.asciidoc +++ b/docs/en/ingest-management/fleet/monitor-elastic-agent.asciidoc @@ -33,6 +33,9 @@ To view the overall status of your {fleet}-managed agents, in {kib}, go to [role="screenshot"] image::images/kibana-fleet-agents.png[Agents tab showing status of each {agent}] +IMPORTANT: The **Agents** tab in **{fleet}** displays a maximum of 10,000 agents, shown on 500 pages with 20 rows per page. +If you have more than 10,000 agents, we recommend using the filtering and sorting options described in this section to narrow the table to fewer than 10,000 rows. + {agent}s can have the following statuses: |=== @@ -67,6 +70,10 @@ For advanced filtering, use the search bar to create structured queries using {kibana-ref}/kuery-query.html[{kib} Query Language]. For example, enter `local_metadata.os.family : "darwin"` to see only agents running on macOS. +You can also sort the list of agents by host, last activity time, or version, by clicking on the table headings for those fields. + +To perform a bulk action on more than 10,000 agents, you can select the **Select everything on all pages** button. + [discrete] [[view-agent-details]] = View details for an agent @@ -174,14 +181,19 @@ To download the diagnostics bundle for local viewing: . In the **Host** column, click the agent's name. -. Click the **Diagnostics** tab. +. Select the **Diagnostics** tab and click the **Request diagnostics .zip** button. ++ +[role="screenshot"] +image::images/collect-agent-diagnostics1.png[Collect agent diagnostics under agent details] -. Click the **Request diagnostics .zip** button. +. In the **Request Diagnostics** pop-up, select **Collect additional CPU metrics** if you'd like detailed CPU data. + [role="screenshot"] -image::images/collect-agent-diagnostics.png[Collect agent diagnostics under agent details] +image::images/collect-agent-diagnostics2.png[Collect agent diagnostics confirmation pop-up] + +. Click the **Request diagnostics** button. -Any in-progress or previously collected bundles for the {agent} will be listed on this page. +When available, the new diagnostic bundle will be listed on this page, as well as any in-progress or previously collected bundles for the {agent}. Note that the bundles are stored in {es} and are removed automatically after 7 days. You can also delete any previously created bundle by clicking the `trash can` icon. @@ -226,6 +238,8 @@ monitoring settings for all agents enrolled in a specific agent policy: . Under **Agent monitoring**, deselect (or select) one or both of these settings: **Collect agent logs** and **Collect agent metrics**. +. Under **Advanced monitoring options** you can configure additional settings including an HTTP monitoring endpoint, diagnostics rate limiting, and diagnostics file upload limits. Refer to <> for details. + . Save your changes. To turn off agent monitoring when creating a new agent policy: @@ -309,7 +323,7 @@ This will generate an alert when one or more agents are reported by the `fleet.a ** SIZE: 100 . Set **Check every** to the frequency at which the rule condition should be evaluated. The default setting is one minute. . Select an action to occur when the rule conditions are met. For example, to set the alert to send an email when an alert occurs, select the Email connector type and specify: -** Email connector: `Elastic-Cloud-SMPT` +** Email connector: `Elastic-Cloud-SMTP` ** Action frequency: `For each alert` and `On check intervals` ** Run when: `Query matched` ** To: diff --git a/docs/en/ingest-management/fleet/unenroll-elastic-agent.asciidoc b/docs/en/ingest-management/fleet/unenroll-elastic-agent.asciidoc index ea36d0871..0230991be 100644 --- a/docs/en/ingest-management/fleet/unenroll-elastic-agent.asciidoc +++ b/docs/en/ingest-management/fleet/unenroll-elastic-agent.asciidoc @@ -2,6 +2,8 @@ = Unenroll {agent}s You can unenroll {agent}s to invalidate the API key used to connect to {es}. +For {agents} installed in a containerized environment, see <> +for details about how enrollment handling is managed. . In {fleet}, select *Agents*. @@ -21,3 +23,6 @@ will show this error instead: `invalid api key to authenticate with fleet`. TIP: If unenrollment hangs, select *Force unenroll* to invalidate all API keys related to the agent and change the status to `inactive` so that the agent no longer appears in {fleet}. + +NOTE: Enrollment handling for {agent} in a containerized environment has some special nuances. +For details refer to <>. \ No newline at end of file diff --git a/docs/en/ingest-management/fleet/upgrade-elastic-agent.asciidoc b/docs/en/ingest-management/fleet/upgrade-elastic-agent.asciidoc index c8e90c42a..a04c4a806 100644 --- a/docs/en/ingest-management/fleet/upgrade-elastic-agent.asciidoc +++ b/docs/en/ingest-management/fleet/upgrade-elastic-agent.asciidoc @@ -30,8 +30,8 @@ By default, {agent}s require internet access to perform binary upgrades from {agent}s to download binaries from it. For more information, refer to <>. -NOTE: The upgrade feature is not supported for upgrading DEB/RPM packages or -Docker images. +NOTE: The upgrade feature is not supported for upgrading DEB/RPM packages or Docker images. +Refer to <> to upgrade a DEB or RPM package manually. For a detailed view of the {agent} upgrade process and the interactions between {fleet}, {agent}, and {es}, refer to the link:https://github.com/elastic/elastic-agent/blob/main/docs/upgrades.md[Communications amongst components] diagram in the `elastic-agent` GitHub repository. @@ -51,7 +51,7 @@ These restrictions apply whether you are upgrading {agents} individually or in b [[upgrade-agent]] == Upgrading {agent} -To upgrade your {agent}s, go to *Management > {fleet} > Agents* in {kib}. You +To upgrade your {agents}, go to *Management* → *{fleet}* → *Agents* in {kib}. You can perform the following upgrade-related actions: [options,header] @@ -78,6 +78,8 @@ can perform the following upgrade-related actions: |=== +With the right {subscriptions}[subscription level], you can also configure an automatic, gradual upgrade of a percentage of the {agents} enrolled in an {agent} policy. For more information, refer to <>. + [discrete] [[upgrade-an-agent]] == Upgrade a single {agent} @@ -262,3 +264,115 @@ To force selected agents to upgrade immediately when the upgrade is triggered, select **Immediately**. Avoid using this setting for batches of more than 10 agents. . Restart the upgrades. + +[discrete] +[[auto-upgrade-agents]] +== Auto-upgrade agents enrolled in a policy + +NOTE: This feature is only available for certain subscription levels. For more information, refer to {subscriptions}[{stack} subscriptions]. + +To configure an automatic rollout of a new minor or patch version to a percentage of the agents enrolled in your {agent} policy. follow these steps: + +. In {kib}, go to **Management** → **{fleet}** → **Agent policies**. +. Select the agent policy for which you want to configure an automatic agent upgrade. +. On the agent policy's details page, find **Auto-upgrade agents**, and select **Manage** next to it. +. In the **Manage auto-upgrade agents** window, click **Add target version**. +. From the **Target agent version** dropdown, select the minor or patch version to which you want to upgrade a percentage of your agents. +. In the **% of agents to upgrade** field, enter the percentage of active agents you want to upgrade to this target version. ++ +Note that: ++ +* Unenrolling, unenrolled, inactive, and uninstalled agents are not included in the count. For example, if you set the target upgrade percentage to 50% for a policy with 10 active agents and 10 inactive agents, the target is met when 5 active agents are upgraded. +* Rounding is applied, and the actual percentage of the upgraded agents may vary slightly. For example, if you set the target upgrade percentage to 30% for a policy with 25 active agents, the target is met when 8 active agents are upgraded (32%). + +. You can then add a different target version, and specify the percentage of agents you want to be upgraded to that version. The total percentage of agents to be upgraded cannot exceed 100%. +. Click **Save**. + +Once the configuration is saved, an asynchronous task runs every 30 minutes, gradually upgrading the agents in the policy to the specified target version. + +In case of any failed upgrades, the upgrades are retried with exponential backoff mechanism until the upgrade is successful, or the maximum number of retries is reached. Note that the maximum number of retries is the number of <>. + +[NOTE] +==== +Only active agents enrolled in the policy are considered for the automatic upgrade. + +If new agents are assigned to the policy, the number of {agents} to be upgraded is adjusted according to the set percentages. +==== + +[discrete] +[[auto-upgrade-settings]] +=== Configure the auto-upgrade settings + +On self-managed and cloud deployments of {stack}, you can configure the default task interval and the retry delays of the automatic upgrade in the {kibana-ref}/fleet-settings-kb.html[{kib} {fleet} settings]. For example: + +[source,yml] +------------ +xpack.fleet.autoUpgrades.taskInterval: 15m <1> +xpack.fleet.autoUpgrades.retryDelays: ['5m', '10m', '20m'] <2> +------------ +<1> The time interval at which the auto-upgrade task should run. Defaults to `30m`. +<2> Array indicating how much time should pass before a failed auto-upgrade is retried. The array's length indicates the maximum number of retries. Defaults to `['30m', '1h', '2h', '4h', '8h', '16h', '24h']`. + +For more information, refer to {kibana-ref}/settings.html[Configure {kib}]. + +[discrete] +[[auto-upgrade-view-status]] +=== View the status of the automatic upgrade + +You can view the status of the automatic upgrade in the following ways: + +* On the agent policy's details page, find **Auto-upgrade agents**, and select **Manage** to open the **Manage auto-upgrade agents** window. ++ +The status of the upgrade is displayed next to the specified target version and percentage, and includes the percentage of agents that have already been upgraded. ++ +To view any failed upgrades, hover over the **Upgrade failed** status, then click **Go to upgrade**. +* On the **{fleet}** → **Agents** page, click **Agent activity** to open a flyout showing logs of the {agent} activity and the progress of the automatic agent upgrade. + +[discrete] +[[upgrade-system-packages]] +== Upgrade RPM and DEB system packages + +If you have installed and enrolled {agent} using either a DEB (for a Debian-based Linux distribution) or RPM (for a RedHat-based Linux distribution) install package, the upgrade cannot be managed by {fleet}. +Instead, you can perform the upgrade using these steps. + +For installation steps refer to <>. + +[discrete] +=== Upgrade a DEB {agent} installation: + +. Download the {agent} Debian install package for the release that you want to upgrade to: ++ +[source,terminal,subs="attributes"] +---- +curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-{version}-amd64.deb +---- + +. Upgrade {agent} to the target release: ++ +[source,terminal,subs="attributes"] +---- +sudo dpkg -i elastic-agent-{version}-amd64.deb +---- + +. Confirm in {fleet} that the agent has been upgraded to the target version. +Note that the **Upgrade agent** option in the **Actions** menu next to the agent will be disabled since [fleet]-managed upgrades are not supported for this package type. + +[discrete] +=== Upgrade an RPM {agent} installation: + +. Download the {agent} RPM install package for the release that you want to upgrade to: ++ +[source,terminal,subs="attributes"] +---- +curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-{version}-x86_64.rpm +---- + +. Upgrade {agent} to the target release: ++ +[source,terminal,subs="attributes"] +---- +sudo rpm -U elastic-agent-{version}-x86_64.rpm +---- + +. Confirm in {fleet} that the agent has been upgraded to the target version. +Note that the **Upgrade agent** option in the **Actions** menu next to the agent will be disabled since [fleet]-managed upgrades are not supported for this package type. diff --git a/docs/en/ingest-management/images/component-templates-list.png b/docs/en/ingest-management/images/component-templates-list.png new file mode 100644 index 000000000..de5c9d26e Binary files /dev/null and b/docs/en/ingest-management/images/component-templates-list.png differ diff --git a/docs/en/ingest-management/images/data-stream-info.png b/docs/en/ingest-management/images/data-stream-info.png index e0f6e3cb7..ff57ec44c 100644 Binary files a/docs/en/ingest-management/images/data-stream-info.png and b/docs/en/ingest-management/images/data-stream-info.png differ diff --git a/docs/en/ingest-management/images/data-stream-overview.png b/docs/en/ingest-management/images/data-stream-overview.png new file mode 100644 index 000000000..503661862 Binary files /dev/null and b/docs/en/ingest-management/images/data-stream-overview.png differ diff --git a/docs/en/ingest-management/images/download-agent-policy.png b/docs/en/ingest-management/images/download-agent-policy.png index db2eb17a1..b7126d395 100644 Binary files a/docs/en/ingest-management/images/download-agent-policy.png and b/docs/en/ingest-management/images/download-agent-policy.png differ diff --git a/docs/en/ingest-management/images/guide-agent-policies.png b/docs/en/ingest-management/images/guide-agent-policies.png index 48d9e668a..651392026 100644 Binary files a/docs/en/ingest-management/images/guide-agent-policies.png and b/docs/en/ingest-management/images/guide-agent-policies.png differ diff --git a/docs/en/ingest-management/images/index-template-system-auth.png b/docs/en/ingest-management/images/index-template-system-auth.png new file mode 100644 index 000000000..c4bedaff0 Binary files /dev/null and b/docs/en/ingest-management/images/index-template-system-auth.png differ diff --git a/docs/en/ingest-management/images/kibana-agent-flyout.png b/docs/en/ingest-management/images/kibana-agent-flyout.png index ab0a72b77..c96b394ca 100644 Binary files a/docs/en/ingest-management/images/kibana-agent-flyout.png and b/docs/en/ingest-management/images/kibana-agent-flyout.png differ diff --git a/docs/en/ingest-management/images/kibana-fleet-privileges-all.png b/docs/en/ingest-management/images/kibana-fleet-privileges-all.png new file mode 100644 index 000000000..128b1862b Binary files /dev/null and b/docs/en/ingest-management/images/kibana-fleet-privileges-all.png differ diff --git a/docs/en/ingest-management/images/kibana-fleet-privileges-read.png b/docs/en/ingest-management/images/kibana-fleet-privileges-read.png new file mode 100644 index 000000000..7288e9974 Binary files /dev/null and b/docs/en/ingest-management/images/kibana-fleet-privileges-read.png differ diff --git a/docs/en/ingest-management/images/migration-add-nginx-integration.png b/docs/en/ingest-management/images/migration-add-nginx-integration.png index cad16d26d..7986548a5 100644 Binary files a/docs/en/ingest-management/images/migration-add-nginx-integration.png and b/docs/en/ingest-management/images/migration-add-nginx-integration.png differ diff --git a/docs/en/ingest-management/images/review-component-template01.png b/docs/en/ingest-management/images/review-component-template01.png new file mode 100644 index 000000000..d057d1a15 Binary files /dev/null and b/docs/en/ingest-management/images/review-component-template01.png differ diff --git a/docs/en/ingest-management/images/review-component-template02.png b/docs/en/ingest-management/images/review-component-template02.png new file mode 100644 index 000000000..5f9309bad Binary files /dev/null and b/docs/en/ingest-management/images/review-component-template02.png differ diff --git a/docs/en/ingest-management/images/tls-overview-mutual-all.jpg b/docs/en/ingest-management/images/tls-overview-mutual-all.jpg new file mode 100644 index 000000000..88ef37a68 Binary files /dev/null and b/docs/en/ingest-management/images/tls-overview-mutual-all.jpg differ diff --git a/docs/en/ingest-management/images/tls-overview-mutual-fs-agent.png b/docs/en/ingest-management/images/tls-overview-mutual-fs-agent.png new file mode 100644 index 000000000..4d05d5c1f Binary files /dev/null and b/docs/en/ingest-management/images/tls-overview-mutual-fs-agent.png differ diff --git a/docs/en/ingest-management/images/tls-overview-mutual-fs-es.png b/docs/en/ingest-management/images/tls-overview-mutual-fs-es.png new file mode 100644 index 000000000..0cd7746bf Binary files /dev/null and b/docs/en/ingest-management/images/tls-overview-mutual-fs-es.png differ diff --git a/docs/en/ingest-management/images/tls-overview-oneway-all.jpg b/docs/en/ingest-management/images/tls-overview-oneway-all.jpg new file mode 100644 index 000000000..e65f24add Binary files /dev/null and b/docs/en/ingest-management/images/tls-overview-oneway-all.jpg differ diff --git a/docs/en/ingest-management/images/tls-overview-oneway-fs-agent.png b/docs/en/ingest-management/images/tls-overview-oneway-fs-agent.png new file mode 100644 index 000000000..7d5326c27 Binary files /dev/null and b/docs/en/ingest-management/images/tls-overview-oneway-fs-agent.png differ diff --git a/docs/en/ingest-management/images/tls-overview-oneway-fs-es.png b/docs/en/ingest-management/images/tls-overview-oneway-fs-es.png new file mode 100644 index 000000000..6b2477384 Binary files /dev/null and b/docs/en/ingest-management/images/tls-overview-oneway-fs-es.png differ diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 7328e2a55..081b2acd7 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -25,6 +25,8 @@ include::fleet/add-fleet-server-on-prem.asciidoc[leveloffset=+2] include::fleet/add-fleet-server-mixed.asciidoc[leveloffset=+2] +include::fleet/add-fleet-server-kubernetes.asciidoc[leveloffset=+2] + include::fleet/fleet-server-scaling.asciidoc[leveloffset=+2] include::fleet/fleet-server-secrets.asciidoc[leveloffset=+2] @@ -45,6 +47,12 @@ include::elastic-agent/elastic-agent-container.asciidoc[leveloffset=+3] include::elastic-agent/running-on-kubernetes-managed-by-fleet.asciidoc[leveloffset=+3] +include::elastic-agent/install-on-kubernetes-using-helm.asciidoc[leveloffset=+3] + +include::elastic-agent/example-kubernetes-standalone-agent-helm.asciidoc[leveloffset=+3] + +include::elastic-agent/example-kubernetes-fleet-managed-agent-helm.asciidoc[leveloffset=+3] + include::elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc[leveloffset=+3] include::elastic-agent/configuring-kubernetes-metadata.asciidoc[leveloffset=+3] @@ -65,6 +73,8 @@ include::elastic-agent/configuration/env/container-envs.asciidoc[leveloffset=+3] include::elastic-agent/otel-agent.asciidoc[leveloffset=+2] +include::elastic-agent/otel-agent-transform.asciidoc[leveloffset=+2] + include::elastic-agent/elastic-agent-unprivileged-mode.asciidoc[leveloffset=+2] include::elastic-agent/install-agent-msi.asciidoc[leveloffset=+2] @@ -91,6 +101,8 @@ include::elastic-agent/start-stop-elastic-agent.asciidoc[leveloffset=+2] include::elastic-agent/elastic-agent-encryption.asciidoc[leveloffset=+2] +include::fips-ingest.asciidoc[leveloffset=+1] + include::security/generate-certificates.asciidoc[leveloffset=+1] include::security/certificates.asciidoc[leveloffset=+2] @@ -99,6 +111,8 @@ include::security/certificates-rotation.asciidoc[leveloffset=+2] include::security/mutual-tls.asciidoc[leveloffset=+2] +include::security/tls-overview.asciidoc[leveloffset=+2] + include::security/logstash-certificates.asciidoc[leveloffset=+2] include::fleet/fleet.asciidoc[leveloffset=+1] @@ -131,6 +145,8 @@ include::fleet/agent-health-status.asciidoc[leveloffset=+3] include::fleet/filter-agent-list-by-tags.asciidoc[leveloffset=+3] +include::fleet/enrollment-handling-containerized-agent.asciidoc[leveloffset=+3] + include::agent-policies.asciidoc[leveloffset=+2] include::create-agent-policies-no-UI.asciidoc[leveloffset=+3] @@ -225,7 +241,7 @@ include::troubleshooting/troubleshooting.asciidoc[leveloffset=+2] include::troubleshooting/faq.asciidoc[leveloffset=+2] -include::release-notes/release-notes-8.15.asciidoc[leveloffset=+1] +include::release-notes/release-notes-8.19.asciidoc[leveloffset=+1] include::elastic-agent/install-fleet-managed-agent.asciidoc[leveloffset=+2] diff --git a/docs/en/ingest-management/integrations/add-integration-to-policy.asciidoc b/docs/en/ingest-management/integrations/add-integration-to-policy.asciidoc index 9033a38e8..6d7e1ad15 100644 --- a/docs/en/ingest-management/integrations/add-integration-to-policy.asciidoc +++ b/docs/en/ingest-management/integrations/add-integration-to-policy.asciidoc @@ -6,6 +6,9 @@ ++++ An <> consists of one or more integrations that are applied to the agents enrolled in that policy. +When you add an integration, the policy created for that integration can be shared with multiple {agent} policies. +This reduces the number of integrations policies that you need to actively manage. + To add a new integration to one or more {agent} policies: . In {kib}, go to the **Integrations** page. @@ -28,6 +31,11 @@ You can update the settings for an installed integration at any time: . In {kib}, go to the **Integrations** page. . On the **Integration policies** tab, for the integration that you like to update open the **Actions** menu and select **Edit integration**. . On the **Edit ** page you can update any configuration settings and also update the list of {agent} polices to which the integration is added. ++ +If you clear the **Agent policies** field, the integration will be removed from any {agent} policies to which it had been added. ++ +To identify any integrations that have been "orphaned", that is, not associated with any {agent} policies, check the **Agent polices** column on the **Integration policies** tab. +Any integrations that are installed but not associated with an {agent} policy are as labeled as `No agent policies`. If you haven't deployed any {agent}s yet or set up agent policies, start with one of our quick start guides: diff --git a/docs/en/ingest-management/integrations/integration-level-outputs.asciidoc b/docs/en/ingest-management/integrations/integration-level-outputs.asciidoc index 009b8a0a6..84541f6fa 100644 --- a/docs/en/ingest-management/integrations/integration-level-outputs.asciidoc +++ b/docs/en/ingest-management/integrations/integration-level-outputs.asciidoc @@ -1,9 +1,9 @@ [[integration-level-outputs]] = Set integration-level outputs -If you have an `Enterprise` link:https://www.elastic.co/subscriptions[{stack} subscription], you can configure {agent} data to be sent to different outputs for different integration policies. -Integration-level outputs are very useful for certain scenarios. -For example: +If you have an `Enterprise` link:https://www.elastic.co/subscriptions[{stack} subscription], you can configure {agent} data to be sent to different outputs for different integration policies. Note that the output clusters that you send data to must also be on the same subscription level. + +Integration-level outputs are very useful for certain scenarios. For example: * You can may want to send security logs monitored by an {agent} to one {ls} output, while informational logs are sent to a another {ls} output. * If you operate multiple {beats} on a system and want to migrate these to {agent}, integration-level outputs enable you to maintain the distinct outputs that are currently used by each Beat. diff --git a/docs/en/ingest-management/integrations/integrations-assets-best-practices.asciidoc b/docs/en/ingest-management/integrations/integrations-assets-best-practices.asciidoc index 2a2ac9027..06e38e457 100644 --- a/docs/en/ingest-management/integrations/integrations-assets-best-practices.asciidoc +++ b/docs/en/ingest-management/integrations/integrations-assets-best-practices.asciidoc @@ -1,5 +1,5 @@ [[integrations-assets-best-practices]] -= Best practices for integrations assets += Best practices for integration assets When you use integrations with {fleet} and {agent} there are some restrictions to be aware of. @@ -7,6 +7,10 @@ When you use integrations with {fleet} and {agent} there are some restrictions t * <> * <> * <> +* <> +* <> +* <> +* <> [discrete] [[assets-restrictions-standalone]] @@ -46,3 +50,41 @@ This way, because the assets are not managed by another integration, there is le Note, however, that creating standalone integration assets based off of {fleet} and {agent} integrations is considered a custom configuration that is not tested nor supported. Whenever possible it's recommended to use standard integrations. +[discrete] +[[assets-restrictions-editing-assets]] +== Editing assets managed by {fleet} + +{fleet}-managed integration assets should not be edited. Examples of these assets include an integration index template, the `@package` component templates, and ingest pipelines that are bundled with integrations. Any changes made to these assets will be overwritten when the integration is upgraded. + +[discrete] +[[assets-restrictions-custom-component-templates]] +== Creating custom component templates + +While creating a `@custom` component template for a package integration is supported, it involves risks which can prevent data from being ingested correctly. This practice can lead to broken indexing, data loss, and breaking of integration package upgrades. + +For example: + + * If the `@package` component template of an integration is changed from a "normal" datastream to `TSDB` or `LogsDB`, some of the custom settings or mappings introduced may not be compatible with these indexing modes. + * If the type of an ECS field is overridden from, for example, `keyword` to `text`, aggregations based on that field may be prevented for built-in dashboards. + +A similar caution against custom index mappings is noted in <>. + +[discrete] +[[assets-restrictions-custom-ingest-pipeline]] +== Creating a custom ingest pipeline + +If you create a custom index pipeline (as documented in the <> tutorial), Elastic is not responsible for ensuring that it indexes and behaves as expected. Creating a custom pipeline involves custom processing of the incoming data, which should be done with caution and tested carefully. + +Refer to <> to learn more. + +[discrete] +[[assets-restrictions-cloning-index-template]] +== Cloning the index template of an integration package + +When you clone the index template of an integration package, this involves risk as any changes made to the original index template when it is upgraded will not be propagated to the cloned version. That is, the structure of the new index template is effectively frozen at the moment that it is cloned. Cloning an index template of an integration package can therefore lead to broken indexing, data loss, and breaking of integration package upgrades. + +Additionally, cloning index templates to add or inject additional component templates cannot be tested by Elastic, so we cannot guarantee that the template will work in future releases. + +If you want to change the ILM Policy, the number of shards, or other settings for the datastreams of one or more integrations, but the changes do not need to be specific to a given namespace, it's highly recommended to use the `package@custom` component templates, as described in <> and <> of the Customize data retention policies tutorial, so as to avoid the problems mentioned above. + +If you want to change these settings for the data streams in one or more integrations and the changes **need to be namespace specific**, then you can do so following the steps in <> of the Customize data retention policies tutorial, but be aware of the restrictions mentioned above. diff --git a/docs/en/ingest-management/processors/processor-add_cloud_metadata.asciidoc b/docs/en/ingest-management/processors/processor-add_cloud_metadata.asciidoc index a05be276c..3f42bec73 100644 --- a/docs/en/ingest-management/processors/processor-add_cloud_metadata.asciidoc +++ b/docs/en/ingest-management/processors/processor-add_cloud_metadata.asciidoc @@ -60,6 +60,8 @@ include::processors.asciidoc[tag=processor-limitations] | a| List of provider names to use. If `providers` is not configured, all providers that do not access a remote endpoint are enabled by default. +The list of providers may alternatively be configured with the environment variable `BEATS_ADD_CLOUD_METADATA_PROVIDERS`, +by setting it to a comma-separated list of provider names. The list of supported provider names includes: diff --git a/docs/en/ingest-management/release-notes/images/upgrade-agent-not-selectable.png b/docs/en/ingest-management/release-notes/images/upgrade-agent-not-selectable.png new file mode 100644 index 000000000..e88399b70 Binary files /dev/null and b/docs/en/ingest-management/release-notes/images/upgrade-agent-not-selectable.png differ diff --git a/docs/en/ingest-management/release-notes/images/upgrade-fleet-server-inconsistent-state.png b/docs/en/ingest-management/release-notes/images/upgrade-fleet-server-inconsistent-state.png new file mode 100644 index 000000000..902eefcc2 Binary files /dev/null and b/docs/en/ingest-management/release-notes/images/upgrade-fleet-server-inconsistent-state.png differ diff --git a/docs/en/ingest-management/release-notes/release-notes-8.13.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.13.asciidoc index c3beb36eb..5060a98c3 100644 --- a/docs/en/ingest-management/release-notes/release-notes-8.13.asciidoc +++ b/docs/en/ingest-management/release-notes/release-notes-8.13.asciidoc @@ -32,6 +32,27 @@ Also see: There are no bug fixes for {fleet} or {agent} in this release. +[discrete] +[[known-issues-8.13.4]] +=== Known issues + +[[known-issue-174855-8.13.4]] +.ECS fields are not included to the `index.query.default_field` in {agent} integrations +[%collapsible] +==== +*Details* + +Due to changes introduced to support the ecs@mappings component template (see link:https://github.com/elastic/kibana/pull/174855[elastic/kibana/pull/174855]), {fleet} no longer includes ECS fields in the integrations' `index.query.default_field`. Not including ECS fields in the `index.query.default_field` setting may affect integrations that rely on fieldless queries (when no field is specified for a query). + +If you run a query without specifying a field, the query will not return results for ECS fields. + +*Impact* + + +In version 8.14.0 and later, {fleet} sets `index.query.default_field` to `*`, so agentless queries will work as expected. We recommend users of {fleet} upgrade to 8.14 when that release becomes available. + +If you are running 8.13.x and unable to upgrade to 8.14.0, you can follow the workarounds described in the link:https://support.elastic.co/knowledge/bbdbeb57. +==== + // end 8.13.4 relnotes // begin 8.13.3 relnotes @@ -48,6 +69,27 @@ Review important information about {fleet-server} and {agent} for the 8.13.3 rel {agent}:: * Update Go version to 1.21.9. {agent-pull}4508[#4508] +[discrete] +[[known-issues-8.13.3]] +=== Known issues + +[[known-issue-174855-8.13.3]] +.ECS fields are not included to the `index.query.default_field` in {agent} integrations +[%collapsible] +==== +*Details* + +Due to changes introduced to support the ecs@mappings component template (see link:https://github.com/elastic/kibana/pull/174855[elastic/kibana/pull/174855]), {fleet} no longer includes ECS fields in the integrations' `index.query.default_field`. Not including ECS fields in the `index.query.default_field` setting may affect integrations that rely on fieldless queries (when no field is specified for a query). + +If you run a query without specifying a field, the query will not return results for ECS fields. + +*Impact* + + +In version 8.14.0 and later, {fleet} sets `index.query.default_field` to `*`, so agentless queries will work as expected. We recommend users of {fleet} upgrade to 8.14 when that release becomes available. + +If you are running 8.13.x and unable to upgrade to 8.14.0, you can follow the workarounds described in the link:https://support.elastic.co/knowledge/bbdbeb57. +==== + [discrete] [[bug-fixes-8.13.3]] === Bug fixes @@ -90,6 +132,23 @@ This issue has been link:https://github.com/elastic/elastic-stack-installers/pul ==== +[[known-issue-174855-8.13.2]] +.ECS fields are not included to the `index.query.default_field` in {agent} integrations +[%collapsible] +==== +*Details* + +Due to changes introduced to support the ecs@mappings component template (see link:https://github.com/elastic/kibana/pull/174855[elastic/kibana/pull/174855]), {fleet} no longer includes ECS fields in the integrations' `index.query.default_field`. Not including ECS fields in the `index.query.default_field` setting may affect integrations that rely on fieldless queries (when no field is specified for a query). + +If you run a query without specifying a field, the query will not return results for ECS fields. + +*Impact* + + +In version 8.14.0 and later, {fleet} sets `index.query.default_field` to `*`, so agentless queries will work as expected. We recommend users of {fleet} upgrade to 8.14 when that release becomes available. + +If you are running 8.13.x and unable to upgrade to 8.14.0, you can follow the workarounds described in the link:https://support.elastic.co/knowledge/bbdbeb57. +==== + [discrete] [[bug-fixes-8.13.2]] === Bug fixes @@ -127,6 +186,23 @@ This issue has been link:https://github.com/elastic/elastic-stack-installers/pul ==== +[[known-issue-174855-8.13.1]] +.ECS fields are not included to the `index.query.default_field` in {agent} integrations +[%collapsible] +==== +*Details* + +Due to changes introduced to support the ecs@mappings component template (see link:https://github.com/elastic/kibana/pull/174855[elastic/kibana/pull/174855]), {fleet} no longer includes ECS fields in the integrations' `index.query.default_field`. Not including ECS fields in the `index.query.default_field` setting may affect integrations that rely on fieldless queries (when no field is specified for a query). + +If you run a query without specifying a field, the query will not return results for ECS fields. + +*Impact* + + +In version 8.14.0 and later, {fleet} sets `index.query.default_field` to `*`, so agentless queries will work as expected. We recommend users of {fleet} upgrade to 8.14 when that release becomes available. + +If you are running 8.13.x and unable to upgrade to 8.14.0, you can follow the workarounds described in the link:https://support.elastic.co/knowledge/bbdbeb57. +==== + [discrete] [[enhancements-8.13.1]] === Enhancements @@ -233,6 +309,23 @@ This issue has been link:https://github.com/elastic/elastic-stack-installers/pul ==== +[[known-issue-174855-8.13.0]] +.ECS fields are not included to the `index.query.default_field` in {agent} integrations +[%collapsible] +==== +*Details* + +Due to changes introduced to support the ecs@mappings component template (see link:https://github.com/elastic/kibana/pull/174855[elastic/kibana/pull/174855]), {fleet} no longer includes ECS fields in the integrations' `index.query.default_field`. Not including ECS fields in the `index.query.default_field` setting may affect integrations that rely on fieldless queries (when no field is specified for a query). + +If you run a query without specifying a field, the query will not return results for ECS fields. + +*Impact* + + +In version 8.14.0 and later, {fleet} sets `index.query.default_field` to `*`, so agentless queries will work as expected. We recommend users of {fleet} upgrade to 8.14 when that release becomes available. + +If you are running 8.13.x and unable to upgrade to 8.14.0, you can follow the workarounds described in the link:https://support.elastic.co/knowledge/bbdbeb57. +==== + [discrete] [[new-features-8.13.0]] === New features diff --git a/docs/en/ingest-management/release-notes/release-notes-8.15.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.15.asciidoc index 05b9a4ecb..dd157a864 100644 --- a/docs/en/ingest-management/release-notes/release-notes-8.15.asciidoc +++ b/docs/en/ingest-management/release-notes/release-notes-8.15.asciidoc @@ -14,6 +14,11 @@ This section summarizes the changes in each release. +* <> +* <> +* <> +* <> +* <> * <> Also see: @@ -21,6 +26,150 @@ Also see: * {kibana-ref}/release-notes.html[{kib} release notes] * {beats-ref}/release-notes.html[{beats} release notes] +// begin 8.15.5 relnotes + +[[release-notes-8.15.5]] +== {fleet} and {agent} 8.15.5 + +[discrete] +[[enhancements-8.15.5]] +=== Enhancements + +{agent}:: +* Emit Pod data only for running Pods in the Kubernetes provider. {agent-pull}6011[#6011] {agent-issue}5835[#5835] {agent-issue}5991[#5991] + +// end 8.15.5 relnotes + +// begin 8.15.4 relnotes + +[[release-notes-8.15.4]] +== {fleet} and {agent} 8.15.4 + +[discrete] +[[bug-fixes-8.15.4]] +=== Bug fixes + +{agent}:: +* Improve upgrade experience by assuming that {agent} is installed, allowing to use proper control socket path to communicate with running {agent}. {agent-pull}5879[#5879] + +// end 8.15.4 relnotes + +// begin 8.15.3 relnotes + +[[release-notes-8.15.3]] +== {fleet} and {agent} 8.15.3 + +Review important information about the {fleet} and {agent} 8.15.3 release. + +[discrete] +[[known-issues-8.15.3]] +=== Known issues + +[[known-issue-issue-41355-8.15.3]] +.The memory usage of {beats} based integrations is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. +[%collapsible] +==== + +*Details* + +In 8.15, events in the memory queue are not freed when they are acknowledged (as intended), but only when they are overwritten by later events in the queue buffer. This means for example if a configuration has a queue size of 5000, but the input data is low-volume and only 100 events are active at once, then the queue will gradually store more events until reaching 5000 in memory at once, then start replacing those with new events. + +See {beats} issue link:https://github.com/elastic/beats/issues/41355[#41355]. + +*Impact* + + +Memory usage may be higher than in previous releases depending on the throughput of {agent}. A fix is planned for 8.15.4. + +- The worst memory increase is for low-throughput configs with large queues. +- For users whose queues were already sized proportionate to their throughput, memory use is increased but only marginally. +- Affected users can mitigate the higher memory usage by lowering their queue size. + +==== + +[discrete] +[[security-updates-8.15.3]] +=== Security updates + +{agent}:: +* Update Go version to 1.22.8. {agent-pull}5718[#5718] + +[discrete] +[[enhancements-8.15.3]] + +=== Enhancements + +{agent}:: +* Adjust the default memory requests and limits for {agent} when it runs in a Kubernetes cluster. {agent-pull}5614[#5614] {agent-issue}5613[#5613] {agent-issue}4729[#4729] +* Use a metadata watcher for ReplicaSets in the K8s provider to collect only the name and OwnerReferences, which are used to connect Pods to Deployments and DaemonSets. {agent-pull}5699[#5699] {agent-issue}5623[#5623] + +[discrete] +[[bug-fixes-8.15.3]] +=== Bug fixes + +{agent}:: +* Add `pprof` endpoints to the monitoring server if they're enabled in the {agent} configuration. {agent-pull}5562[#5562] +* Stop the `elastic-agent inspect` command from printing the output configuration twice. {agent-pull}5692[#5692] {agent-issue}4471[#4471] + +// end 8.15.3 relnotes + +// begin 8.15.2 relnotes + +[[release-notes-8.15.2]] +== {fleet} and {agent} 8.15.2 + +Review important information about the {fleet} and {agent} 8.15.2 release. + +[discrete] +[[known-issues-8.15.2]] +=== Known issues + +[[known-issue-issue-41355-8.15.2]] +.The memory usage of {beats} based integrations is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. +[%collapsible] +==== + +*Details* + +In 8.15, events in the memory queue are not freed when they are acknowledged (as intended), but only when they are overwritten by later events in the queue buffer. This means for example if a configuration has a queue size of 5000, but the input data is low-volume and only 100 events are active at once, then the queue will gradually store more events until reaching 5000 in memory at once, then start replacing those with new events. + +See {beats} issue link:https://github.com/elastic/beats/issues/41355[#41355]. + +*Impact* + + +Memory usage may be higher than in previous releases depending on the throughput of {agent}. A fix is planned for 8.15.4. + +- The worst memory increase is for low-throughput configs with large queues. +- For users whose queues were already sized proportionate to their throughput, memory use is increased but only marginally. +- Affected users can mitigate the higher memory usage by lowering their queue size. + +==== + +[discrete] +[[enhancements-8.15.2]] +=== Enhancements + +Fleet:: +* Bump the maximum supported package spec version to 3.2. ({kibana-pull}193574[#193574]) + +[discrete] +[[bug-fixes-8.15.2]] +=== Bug fixes + +Fleet:: +* Prevent extra `agent_status` call with empty `policyId`, resulting in incorrect agent count on the `Edit Integration` page. ({kibana-pull}192549[#192549]) +* Set the correct title for the `Restart upgrade` agent modal. ({kibana-pull}192536[#192536]) + +{agent}:: +* Add the `health_check` extension to the `otel.yml` file bundled with the {agent} package, in order to prevent installation of the `open-telemetry/opentelemetry-collector` Helm chart from failing. {agent-pull}5369[#5369] {agent-issue}5092[#5092] +* Fix bug that prevented the {beats} {es} output from recovering from an interrupted connection. {beats-pull}40769[#40796] {beats-issue}40705[#40705] +* Prevent the {agent} from crashing when self unenrolling due to too many authentication failures against {fleet-server}. {agent-pull}5438[#5438] {agent-issue}5434[#5434] +* Set the default log level when an {agent} policy is changed and a log level is not set in the policy. {agent-pull}5452[#5452] {agent-issue}5451[#5451] + +{fleet-server}:: +* Enable missing warnings for configuration options that have been deprecated throughout the 8.x lifecycle. {fleet-server-pull}3901[#3901] + +// end 8.15.2 relnotes + // begin 8.15.1 relnotes [[release-notes-8.15.1]] @@ -64,6 +213,27 @@ same package versions. ==== +[[known-issue-issue-41355-8.15.1]] +.The memory usage of {beats} based integrations is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. +[%collapsible] +==== + +*Details* + +In 8.15, events in the memory queue are not freed when they are acknowledged (as intended), but only when they are overwritten by later events in the queue buffer. This means for example if a configuration has a queue size of 5000, but the input data is low-volume and only 100 events are active at once, then the queue will gradually store more events until reaching 5000 in memory at once, then start replacing those with new events. + +See {beats} issue link:https://github.com/elastic/beats/issues/41355[#41355]. + +*Impact* + + +Memory usage may be higher than in previous releases depending on the throughput of {agent}. A fix is planned for 8.15.4. + +- The worst memory increase is for low-throughput configs with large queues. +- For users whose queues were already sized proportionate to their throughput, memory use is increased but only marginally. +- Affected users can mitigate the higher memory usage by lowering their queue size. + +==== + // end 8.15.1 relnotes // begin 8.15.0 relnotes @@ -102,6 +272,27 @@ If you're using {agent} on Windows with any integration which makes use of the A ==== +[[known-issue-issue-41355]] +.The memory usage of {beats} based integrations is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. +[%collapsible] +==== + +*Details* + +In 8.15, events in the memory queue are not freed when they are acknowledged (as intended), but only when they are overwritten by later events in the queue buffer. This means for example if a configuration has a queue size of 5000, but the input data is low-volume and only 100 events are active at once, then the queue will gradually store more events until reaching 5000 in memory at once, then start replacing those with new events. + +See {beats} issue link:https://github.com/elastic/beats/issues/41355[#41355]. + +*Impact* + + +Memory usage may be higher than in previous releases depending on the throughput of {agent}. A fix is planned for 8.15.4. + +- The worst memory increase is for low-throughput configs with large queues. +- For users whose queues were already sized proportionate to their throughput, memory use is increased but only marginally. +- Affected users can mitigate the higher memory usage by lowering their queue size. + +==== + [discrete] [[new-features-8.15.0]] === New features diff --git a/docs/en/ingest-management/release-notes/release-notes-8.16.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.16.asciidoc new file mode 100644 index 000000000..ed5207f10 --- /dev/null +++ b/docs/en/ingest-management/release-notes/release-notes-8.16.asciidoc @@ -0,0 +1,535 @@ +// Use these for links to issue and pulls. +:kibana-issue: https://github.com/elastic/kibana/issues/ +:kibana-pull: https://github.com/elastic/kibana/pull/ +:beats-issue: https://github.com/elastic/beats/issues/ +:beats-pull: https://github.com/elastic/beats/pull/ +:agent-libs-pull: https://github.com/elastic/elastic-agent-libs/pull/ +:agent-issue: https://github.com/elastic/elastic-agent/issue/ +:agent-pull: https://github.com/elastic/elastic-agent/pull/ +:fleet-server-issue: https://github.com/elastic/fleet-server/issues/ +:fleet-server-pull: https://github.com/elastic/fleet-server/pull/ + +[[release-notes]] += Release notes + +This section summarizes the changes in each release. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +Also see: + +* {kibana-ref}/release-notes.html[{kib} release notes] +* {beats-ref}/release-notes.html[{beats} release notes] + +// begin 8.16.6 relnotes + +[[release-notes-8.16.6]] +== {fleet} and {agent} 8.16.6 + +Review important information about the {fleet} and {agent} 8.16.6 release. + +[discrete] +[[enhancements-8.16.6]] +=== Enhancements + +{agent}:: +* Include all metadata that is sent to {fleet} in the `agent-info.yaml` file in diagnostics by default. {agent-pull}7029[#7029] + +[discrete] +[[bug-fixes-8.16.6]] +=== Bug fixes + +{fleet}:: +* Fix an issue with the {agent} binary download field being blank when a policy uses the default download source. {kibana-pull}214360[#214360] + +{agent}:: +* Add conditions to the `copy_fields` processors used with {agent} self-monitoring to prevent spamming the debug logs. {agent-pull}6730[#6730] {agent-issue}5299[#5299] +* Make enroll command backoff more conservative and add backoff when using `--delay-enroll`. {agent-pull}6983[#6983] {agent-issue}6761[#6761] +* Add missing null checks to AST methods. {agent-pull}7009[#7009] {agent-issue}6999[#6999] +* Fix an issue where `fixpermissions` on Windows incorrectly returned an error message due to improper handling of Windows API return values. {agent-pull}7059[#7059] {agent-issue}6917[#6917] +* Support IPv6 hosts in enroll URL. {agent-pull}7036[#7036] +* Support IPv6 hosts in GRPC configuration. {agent-pull}7035[#7035] +* Rotate logger output file when writing to a symbolic link. {agent-pull}6938[#6938] +* Do not fail Windows permission updates on missing files or paths. {agent-pull}7305[#7305] {agent-issue}7301[#7301] +* Make `otelcol` script executable in the Docker image. {agent-pull}7345[#7345] + +// end 8.16.6 relnotes + + + +// begin 8.16.5 relnotes + +[[release-notes-8.16.5]] +== {fleet} and {agent} 8.16.5 + +There are no bug fixes for {fleet} or {agent} in this release. + +// end 8.16.5 relnotes + +[[release-notes-8.16.4]] +== {fleet} and {agent} 8.16.4 + +[discrete] +[[security-updates-8.16.4]] +=== Security updates + +{agent}:: +* Upgrade NodeJS to LTS v18.20.6. {agent-pull}6641[#6641] + +[discrete] +[[bug-fixes-8.16.4]] +=== Bug fixes + +{agent}:: +* Emit vars even if provider data is empty from the start. {agent-pull}6598[#6598] +* Redact secrets within complex nested paths. {agent-pull}6710[#6710] +* Improve the CLI output message when `elastic-agent uninstall` runs after the agent has previously been unenrolled. {agent-pull}6735[#6735] + +// end 8.16.4 relnotes + +// begin 8.16.3 relnotes + +[[release-notes-8.16.3]] +== {fleet} and {agent} 8.16.3 + +Review important information about the {fleet} and {agent} 8.16.3 release. + +[discrete] +[[bug-fixes-8.16.3]] +=== Bug fixes + +{fleet}:: +* Fixed an issue that prevented {agent} tags from being displayed when the agent list is filtered. ({kibana-pull}205163[#205163]) + +// end 8.16.3 relnotes + +// begin 8.16.2 relnotes + +[[release-notes-8.16.2]] +== {fleet} and {agent} 8.16.2 + +Review important information about the {fleet} and {agent} 8.16.2 release. + +[discrete] +[[known-issues-8.16.2]] +=== Known Issues + +[discrete] +[[known-issue-6213-8-16-2]] +.An {agent} with the Defend integration may report an Orphaned status and will not be able to be issued an upgrade action through {fleet}. +[%collapsible] +==== +*Details* + +A known issue in the {agent} may prevent it from being targetted with an upgrade action for a future release. +This may occur if the Defend integration is used and the agent is stopped on a running instance for too long. +An agent may be stopped as part of an upgrade process. + +*Impact* + +A bug fix is present in the 8.16.3 and 8.17.1 releases of {fleet} that will prevent this from occuring. + +If you have agents that are affected, the workaround is as follows: +[source,shell] +---- +# Get a Token to issue an update_by_query request: +curl -XPOST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-unenrolled" + +# Issue an update_by_query request that targets effected agents: +curl -XPOST -H 'Authorization: Bearer ${TOKEN}' -H 'x-elastic-product-origin:fleet' -H 'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "unenrolled_at" } }],"must_not": [{ "term": { "active": "false" } }]}},"script": {"source": "ctx._source.unenrolled_at = null;","lang": "painless"}}' +---- +==== + +[discrete] +[[enhancements-8.16.2]] +=== Enhancements + +In this release we've introduced an image based on the hardened link:https://github.com/wolfi-dev/[Wolfi] image to provide additional security to our self-managed customers, and improve our supply chain security posture. Wolfi-based images require Docker version 20.10.10 or higher. + +{agent}:: +* Perform check for an external package manager only at startup. {agent-pull}6178[#6178] {agent-issue}5835[#5835] {agent-issue}5991[#5991] +* Remove some unnecessary copies when generating component configuration. {agent-pull}6184[#6184] {agent-issue}5835[#5835] {agent-issue}5991[#5991] +* Use xxHash instead of sha256 for hashing AST nodes when generating component configuration. {agent-pull}6192[#6192] {agent-issue}5835[#5835] {agent-issue}5991[#5991] +* Cache conditional sections when applying variables to component configuration. {agent-pull}6229[#6229] {agent-issue}5835[#5835] {agent-issue}5991[#5991] + +// end 8.16.2 relnotes + +// begin 8.16.1 relnotes + +[[release-notes-8.16.1]] +== {fleet} and {agent} 8.16.1 + +Review important information about the {fleet} and {agent} 8.16.1 release. + +[discrete] +[[known-issues-8.16.1]] +=== Known Issues + +[discrete] +[[known-issue-6213-8-16-1]] +.An {agent} with the Defend integration may report an Orphaned status and will not be able to be issued an upgrade action through {fleet}. +[%collapsible] +==== +*Details* + +A known issue in the {agent} may prevent it from being targetted with an upgrade action for a future release. +This may occur if the Defend integration is used and the agent is stopped on a running instance for too long. +An agent may be stopped as part of an upgrade process. + +*Impact* + +A bug fix is present in the 8.16.3 and 8.17.1 releases of the {fleet} that will prevent this from occuring. + +If you have agents that are affected, the workaround is as follows: +[source,shell] +---- +# Get a Token to issue an update_by_query request: +curl -XPOST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-unenrolled" + +# Issue an update_by_query request that targets effected agents: +curl -XPOST -H 'Authorization: Bearer ${TOKEN}' -H 'x-elastic-product-origin:fleet' -H 'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "unenrolled_at" } }],"must_not": [{ "term": { "active": "false" } }]}},"script": {"source": "ctx._source.unenrolled_at = null;","lang": "painless"}}' +---- +==== + +[discrete] +[[bug-fixes-8.16.1]] +=== Bug fixes + +{agent}:: +* During an {agent} upgrade, resolve paths to a proper value assuming that the upgrading agent is installed. {agent-pull}5879[#5879] {agent-issue}5872[#5872] +* Trim spaces in the user input accepted by the cli.confirm function. This allows users to enter spaces around the `yes/no` inputs in CLI confirmation prompts. {agent-pull}5909[#5909] +* Skip calling the `notifyFleetAuditUninstall` function to notify {fleet} on Windows during {agent} uninstall, to significantly reduce likelihood of an exception being thrown. {agent-pull}6065[#6065] {agent-issue}5952[#5952] + +// end 8.16.1 relnotes + +// begin 8.16.0 relnotes + +[[release-notes-8.16.0]] +== {fleet} and {agent} 8.16.0 + +Review important information about the {fleet} and {agent} 8.16.0 release. + +[discrete] +[[security-updates-8.16.0]] +=== Security updates + +{fleet-server}:: +* Update {fleet-server} Go version to 1.23.1. {fleet-server-pull}3924[#3924] + +[discrete] +[[breaking-changes-8.16.0]] +=== Breaking changes + +Breaking changes can prevent your application from optimal operation and +performance. Before you upgrade, review the breaking changes, then mitigate the +impact to your application. + +{agent}:: +* When using the System integration, uppercase characters in the `host.hostname` are being converted to lowercase in {agent} output. This can possibly result in duplicated host entries appearing in {kib}. {beats-issue}39993[#3993] + +[discrete] +[[known-issues-8.16.0]] +=== Known issues + +[[known-issue-191661]] +.{fleet} UI listing shows "No agent found" +[%collapsible] +==== + +*Details* + +In the {fleet} UI in {kib}, the listing {agents} might show "No agent found" with a toast message "Error fetching agents" or "Agent policy ... not found". + +This error can happen if the {agents} being searched and listed in the UI are using an {agent} policy which doesn't exist. + +*Impact* + + +As a workaround for the issue, you can upgrade your {stack} to verion 8.16.1. The issue has been resolved by {kib} link:https://github.com/elastic/kibana/pull/199325[#199325]. + +==== + +[[known-issue-5952]] +.{agent} throws exception when uninstalling on Windows +[%collapsible] +==== + +*Details* + +{fleet}-managed {agent} sometimes throws an exception when uninstalling on Microsoft Windows systems. + +For example: + +[source,shell] +---- +C:\>"C:\Program Files\Elastic\Agent\elastic-agent.exe" uninstall +Elastic Agent will be uninstalled from your system at C:\Program Files\Elastic\Agent. Do you want to continue? [Y/n]:y +[====] Attempting to notify Fleet of uninstall [37s] unexpected fault address 0x18000473ef1 +fatal error: fault +[signal 0xc0000005 code=0x1 addr=0x18000473ef1 pc=0x9f3004] + +goroutine 1 gp=0xc00007c000 m=5 mp=0xc000116008 [running]: +runtime.throw({0x207a4ba?, 0xa2d986?}) + runtime/panic.go:1023 +0x65 fp=0xc000067588 sp=0xc000067558 pc=0xcf8c5 +runtime.sigpanic() + runtime/signal_windows.go:414 +0xd0 fp=0xc0000675d0 sp=0xc000067588 pc=0xe6a10 +(...) + github.com/elastic/elastic-agent/internal/pkg/agent/errors/generators.go:23 +github.com/elastic/elastic-agent/internal/pkg/fleetapi.(*AuditUnenrollCmd).Execute(0xc00073f998, {0x4, 0x23cf148}, 0x0) + github.com/elastic/elastic-agent/internal/pkg/fleetapi/audit_unenroll_cmd.go:74 +0x324 fp=0xc000067738 sp=0xc0000675d0 pc=0x9f3004 +runtime: g 1: unexpected return pc for github.com/elastic/elastic-agent/internal/pkg/fleetapi.(*AuditUnenrollCmd).Execute called from 0xc0006817a0 +stack: frame={sp:0xc0000675d0, fp:0xc000067738} stack=[0xc000064000,0xc000068000) +0x000000c0000674d0: 0x000000c000067508 0x00000000000d14af +0x000000c0000674e0: 0x00000000023c9c90 0x0000000000000001 +0x000000c0000674f0: 0x0000000000000001 0x000000c00006756b +(...) +---- + +For other examples, refer to {agent} link:5952#issuecomment-2475044465[issue #5952]. + +This problem occurs when {agent} notifies {fleet} to audit the uninstall process. + +*Impact* + + +As a workaround, we recommend trying again to uninstall the agent. + +==== + +[discrete] +[[known-issue-6213-8-16-0]] +.An {agent} with the Defend integration may report an Orphaned status and will not be able to be issued an upgrade action through {fleet}. +[%collapsible] +==== +*Details* + +A known issue in the {agent} may prevent it from being targetted with an upgrade action for a future release. +This may occur if the Defend integration is used and the agent is stopped on a running instance for too long. +An agent may be stopped as part of an upgrade process. + +*Impact* + +A bug fix is present in the 8.16.3 and 8.17.1 releases of {fleet} that will prevent this from occuring. + +If you have agents that are affected, the workaround is as follows: +[source,shell] +---- +# Get a Token to issue an update_by_query request: +curl -XPOST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-unenrolled" + +# Issue an update_by_query request that targets effected agents: +curl -XPOST -H 'Authorization: Bearer ${TOKEN}' -H 'x-elastic-product-origin:fleet' -H 'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "unenrolled_at" } }],"must_not": [{ "term": { "active": "false" } }]}},"script": {"source": "ctx._source.unenrolled_at = null;","lang": "painless"}}' +---- +==== + +[discrete] +[[known-issue-206131]] +.Integration output fails when using default output +[%collapsible] +==== +*Details* + +Beginning in version 8.16.0 you can specify an output per integration policy. However, setting the integration output to the default creates an invalid output name. + +*Impact* + +As a workaround, you can create a clone of the default output and then set it as the output for an integration policy. Refer to issue link:https://github.com/elastic/kibana/issues/206131[#206131] for details and status. + +==== + +[discrete] +[[new-features-8.16.0]] +=== New features + +The 8.16.0 release Added the following new and notable features. + +{fleet}:: +* Add support for content-only packages in integrations UI. {kibana-pull}195831[#195831] +* Add advanced agent monitoring options for HTTP endpoint and diagnostics. {kibana-pull}193361[#193361] +* Add support for periodic unenrollment of inactive agents. Once an {agent} transitions to an `inactive` state and after a configurable timeout has expired, the agent will be unenrolled. {kibana-pull}189861[#189861] +* Add support for dynamic topics to the Kafka output. This allows the Kafka output to write to a topic which is dynamically set in an event field. {kibana-pull}192720[#192720] +* Add support for GeoIP processor databases in Ingest Pipelines. {kibana-pull}190830[#190830] +* Add support for reusable/shareable integration policies. This feature allows you to create integrations policies that can be shared with multiple {agent} policies, thereby reducing the number of integrations policies that you need to actively manage. {kibana-pull}187153[#187153] +* Add support for integration-level outputs. This feature enables you to send integration data to a specific output, overwriting the output defined in the {agent} Policy. {kibana-pull}189125[#189125] + + +{fleet-server}:: +* Add `/api/fleet/agents/:id/audit/unenroll` API that an {agent} or Endpoint process may use to report that an agent was uninstalled or unenrolled to {fleet}. {fleet-server-pull}3818[#3818] {agent-issue}484[#484] +* Add a `secret_paths` attribute to the policy data sent to agents. This attribute is a list of keys that {fleet-server} has replaced with a reference to a secret value. {fleet-server-pull}3908[#3908] {fleet-server-issue}3657[#3657] + +{agent}:: +* Uninstalling a {fleet}-managed {agent} instance will now do a best-effort attempt to notify {fleet-server} of the agent removal so the agent status appears correctly in the {fleet} UI (related to {fleet-server-pull}3818[#3818] above). {agent-pull}5302[#5302] {agent-issue}484[#484] +* Introduce a Helm Chart for deploying {agent} in Kubernetes. {agent-pull}5331[#5331] {agent-issue}3847[#3847] +* Remove support for the experimental shippers feature. {agent-pull}5308[#5308] {agent-issue}4547[#4547] +* Add the GCP Asset Inventory input to Cloudbeat. {agent-pull}5422[#5422] +* Add support for passphrase protected mTLS client certificate key during install/enroll. {agent-pull}5494[#5494] {agent-issue}5489[#5489] +* Elastic Defend now accepts a passphrase protected client certificate key for mTLS. {agent-pull}5542[#5542] {agent-issue}5490[#5490] +* Add a Kustomize template to enable hints-based autodiscovery by default when deploying standalone {agent} in a Kubernetes cluster. This also removes `root` privileges from the init container. {agent-pull}5643[#5643] + +[discrete] +[[enhancements-8.16.0]] +=== Enhancements + +{fleet}:: +* Update maximum supported package version. {kibana-pull}196551[#196551] +* Add additional columns to {agent} Logs UI. {kibana-pull}192262[#192262] +* Show `+build` versions for {agent} upgrades. {kibana-pull}192171[#192171] +* Add format parameter to `agent_policies` APIs. {kibana-pull}191811[#191811] +* Add toggles for `agent.monitoring.http.enabled` and `agent.monitoring.http.buffer.enabled` to agent policy advanced settings. {kibana-pull}190984[#190984] +* Support integration policies without agent policy references (aka orphaned integration policies). {kibana-pull}190649[#190649] +* Allow `traces` to be added to the `monitoring_enabled` array in Agent policies. {kibana-pull}189908[#189908] +* Add setup technology selector to the Add Integration page. {kibana-pull}189612[#189612] + +{fleet-server}:: +* Alter the checkin API to remove attributes set by the audit or unenroll API (follow-up to {fleet-server-pull}3818[#3818] above). {fleet-server-pull}3827[#3827] {agent-issue}484[#484] +* Enable warnings for configuration options that have been deprecated throughout the 8.x lifecycle. {fleet-server-pull}3901[#3901] + +{agent}:: +* Re-enable support for Elastic Defend on Windows Server 2012 and 2012 R2. {agent-pull}5429[#5429] +* Include the correct Elastic License 2.0 file in build artifacts and packages. {agent-pull}5464[#5464] +* Add the `pprofextension` to the {agent} OTel collector. {agent-pull}5556[#5556] +* Update the base container image from Ubuntu 20.04 to Ubuntu 24.04. {agent-pull}5644[#5644] {agent-issue}5501[#5501] +* Redact values from the `elastic-agent inspect` command output for any keys in the `secret_paths` array. {agent-pull}5621[#5621] +* Redact secret paths in files written in {agent} diagnostics bundles. {agent-pull}5745[#5745] +* Update the versions of OpenTelemetry Collector components from v0.111.0/v1.17.0 to v0.112.0/v1.18.0. {agent-pull}5838[#5838] + +[discrete] +[[bug-fixes-8.16.0]] +=== Bug fixes + +{fleet}:: +* Revert "Fix client-side validation for agent policy timeout fields". {kibana-pull}194338[#194338] +* Add proxy arguments to install snippets. {kibana-pull}193922[#193922] +* Rollover if dimension mappings changed in dynamic templates. {kibana-pull}192098[#192098] + +{fleet-server}:: +* Fix the error handling when {fleet-server} attempts to authenticate with {es}. {fleet-server-pull}3935[#3935] {fleet-server-issue}3929[#3929] +* Fix an issue that caused {fleet-server} to report a `500` error on {agent} check-in because the agent has upgrade details but the referenced action ID is not found. {fleet-server-pull}3991[#3991] + +{agent}:: +* Fix {agent} crashing when self unenrolling due to too many authentication failures against {fleet-server}. {agent-pull}5438[#5438] {agent-issue}5434[#5434] +* Change the deprecated `maintainer` label in Dockerfile to use the `org.opencontainers.image.authors` label instead. {agent-pull}5527[#5527] + +// end 8.16.0 relnotes + +// --------------------- +//TEMPLATE +//Use the following text as a template. Remember to replace the version info. + +// begin 8.7.x relnotes + +//[[release-notes-8.7.x]] +//== {fleet} and {agent} 8.7.x + +//Review important information about the {fleet} and {agent} 8.7.x release. + +//[discrete] +//[[security-updates-8.7.x]] +//=== Security updates + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[breaking-changes-8.7.x]] +//=== Breaking changes + +//Breaking changes can prevent your application from optimal operation and +//performance. Before you upgrade, review the breaking changes, then mitigate the +//impact to your application. + +//[discrete] +//[[breaking-PR#]] +//.Short description +//[%collapsible] +//==== +//*Details* + +// For more information, refer to {kibana-pull}PR[#PR]. + +//*Impact* + +// For more information, refer to {fleet-guide}/fleet-server.html[Fleet Server]. +//==== + +//[discrete] +//[[notable-changes-8.13.0]] +//=== Notable changes + +//The following are notable, non-breaking updates to be aware of: + +//* Changes to features that are in Technical Preview. +//* Changes to log formats. +//* Changes to non-public APIs. +//* Behaviour changes that repair critical bugs. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[known-issues-8.7.x]] +//=== Known issues + +//[[known-issue-issue#]] +//.Short description +//[%collapsible] +//==== + +//*Details* + +// + +//*Impact* + + +// + +//==== + +//[discrete] +//[[deprecations-8.7.x]] +//=== Deprecations + +//The following functionality is deprecated in 8.7.x, and will be removed in +//8.7.x. Deprecated functionality does not have an immediate impact on your +//application, but we strongly recommend you make the necessary updates after you +//upgrade to 8.7.x. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[new-features-8.7.x]] +//=== New features + +//The 8.7.x release Added the following new and notable features. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[enhancements-8.7.x]] +//=== Enhancements + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[bug-fixes-8.7.x]] +//=== Bug fixes + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +// end 8.7.x relnotes diff --git a/docs/en/ingest-management/release-notes/release-notes-8.17.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.17.asciidoc new file mode 100644 index 000000000..1c3d670c3 --- /dev/null +++ b/docs/en/ingest-management/release-notes/release-notes-8.17.asciidoc @@ -0,0 +1,737 @@ +// Use these for links to issue and pulls. +:kibana-issue: https://github.com/elastic/kibana/issues/ +:kibana-pull: https://github.com/elastic/kibana/pull/ +:beats-issue: https://github.com/elastic/beats/issues/ +:beats-pull: https://github.com/elastic/beats/pull/ +:agent-libs-pull: https://github.com/elastic/elastic-agent-libs/pull/ +:agent-issue: https://github.com/elastic/elastic-agent/issues/ +:agent-pull: https://github.com/elastic/elastic-agent/pull/ +:fleet-server-issue: https://github.com/elastic/fleet-server/issues/ +:fleet-server-pull: https://github.com/elastic/fleet-server/pull/ + +[[release-notes]] += Release notes + +This section summarizes the changes in each release. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +Also see: + +* {kibana-ref}/release-notes.html[{kib} release notes] +* {beats-ref}/release-notes.html[{beats} release notes] + +// begin 8.17.7 relnotes + +[[release-notes-8.17.7]] +== {fleet} and {agent} 8.17.7 + +Review important information about the 8.17.7 release. + +[discrete] +[[security-updates-8.17.7]] +=== Security updates + +* Upgrade Go version to v1.24.3 in {{agent}}. {agent-pull}8109[https://github.com/elastic/elastic-agent/pull/8109] + +[discrete] +[[known-issues-8.17.7]] +=== Known issues + +[[known-issue-1800-8-17-7]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[bug-fixes-8.17.7]] +=== Bug fixes + +* Improve the upgrade process for {{agent}} installed using DEB or RPM packages by copying the run directory from the previous installation into the new version's folder. {agent-pull}7999[https://github.com/elastic/elastic-agent/pull/7999] {agent-issue}3832[https://github.com/elastic/elastic-agent/issues/3832] + +// end 8.17.7 relnotes + +// begin 8.17.6 relnotes + +[[release-notes-8.17.6]] +== {fleet} and {agent} 8.17.6 + +Review important information about the 8.17.6 release. + +[discrete] +[[known-issues-8.17.6]] +=== Known issues + +[[known-issue-1800-8-17-6]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[enhancements-8.17.6]] +=== Enhancements + +{agent}:: +* Use `fullnameOverride` to set the fully qualified application names in the EDOT Kube-Stack Helm chart. {agent-pull}7754[#7754] {agent-issue}7381[#7381] + +// end 8.17.6 relnotes + +// begin 8.17.5 relnotes + +[[release-notes-8.17.5]] +== {fleet} and {agent} 8.17.5 + +Review important information about the 8.17.5 release. + +[discrete] +[[known-issues-8.17.5]] +=== Known issues + +[[known-issue-1800-8-17-5]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[enhancements-8.17.5]] +=== Enhancements + +{agent}:: +* Ensure consistent input order In self-monitoring configuration. {agent-pull}7724[#7724] + +[discrete] +[[bug-fixes-8.17.5]] +=== Bug fixes + +{fleet}:: +* Enable unenroll inactive agent tasks to unenroll some agents if the first set returned is equal to `UNENROLLMENT_BATCH_SIZE`. {kibana-pull}216283[#216283] + +{fleet-server}:: +* Fix host parsing in {es} output diagnostics. {fleet-server-pull}4765[#4765] +* Redact output log lines during bootstrap configuration. {fleet-server-pull}4775[#4775] +* Fix concurrent access to remote bulker configuration. {fleet-server-pull}4776[#4776] + +{agent}:: +* Change how Windows process handles are obtained when assigning sub-processes to job objects. {agent-pull}6825[#6825] +* Rework Windows user password generation to meet security policy constraints. {agent-pull}7507[#7507] {agent-issue}7506[#7506] +* Wait for Windows service to be fully removed before re-installing during agent switch. {agent-pull}7589[#7589] {agent-issue}6970[#6970] +* Fix panic during shutdown in `Fleetgateway`. {agent-pull}7629[#7629] {agent-issue}7309[#7309] + +// end 8.17.5 relnotes + +// begin 8.17.4 relnotes + +[[release-notes-8.17.4]] +== {fleet} and {agent} 8.17.4 + +Review important information about the 8.17.4 release. + +[discrete] +[[known-issues-8.17.4]] +=== Known issues + +[[known-issue-1749-8-17-4]] +.Inaccurate option to upgrade across major releases appears in {fleet} UI +[%collapsible] +==== + +*Details* + +In late 8.x releases, there's a bug in the {fleet} UI that causes `9.0.0` to appear as an option for {agent} upgrades. This option results in an error when selected, as upgrades across major releases are not supported for {agent}. For example, an agent can't be upgraded to version 9.0 while {kib} and {fleet-server} are on version 8.x. + +The {agent} upgrade is not selectable: + +image::images/upgrade-agent-not-selectable.png[{agent} upgrade is not selectable] + +The {fleet-server} upgrade is selectable, with an inconsistent UI state and error on submit: + +image::images/upgrade-fleet-server-inconsistent-state.png[{fleet-server} inconsistent UI state] + +*Impact* + + +In the 9.x releases, the option that appears in the UI for an upgrade across a major release should be ignored. + +==== + +[[known-issue-1800-8-17-4]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[new-features-8.17.4]] +=== New features + +The 8.17.4 release adds the following new and notable features. + +{agent}:: +* Add managed OTLP endpoint sample configurations. {agent-pull}6630[#6630] + +[discrete] +[[enhancements-8.17.4]] +=== Enhancements + +{agent}:: +* Improve `kubernetes_secrets` provider secret logging. {agent-pull}6841[#6841] {agent-issue}6187[#6187] +* Include all metadata that is sent to {fleet} in the `agent-info.yaml` file in diagnostics by default. {agent-pull}7029[#7029] +* Add API key prefix to managed OTLP endpoint host configurations in EDOT collector kube stack Helm chart. {agent-pull}7063[#7063] + +[discrete] +[[bug-fixes-8.17.4]] +=== Bug fixes + +{agent}:: +* Add conditions to the `copy_fields` processors used with {agent} self-monitoring to prevent spamming the debug logs. {agent-pull}6730[#6730] {agent-issue}5299[#5299] +* Improve message when Fleet Server's audit API endpoint returns a 401 Unauthorized response. {agent-pull}6735[#6735] {agent-issue}6711[#6711] +* Fix `secret_paths` redaction along complex paths in diagnostics. {agent-pull}6710[#6710] +* Make enroll command backoff more conservative and add backoff when using `--delay-enroll`. {agent-pull}6983[#6983] {agent-issue}6761[#6761] +* Fix an issue where `fixpermissions` on Windows incorrectly returned an error message due to improper handling of Windows API return values. {agent-pull}7059[#7059] {agent-issue}6917[#6917] +* Support IPv6 hosts in enroll URL. {agent-pull}7036[#7036] +* Support IPv6 hosts in GRPC configuration. {agent-pull}7035[#7035] +* Rotate logger output file when writing to a symbolic link. {agent-pull}6938[#6938] +* Do not fail Windows permission updates on missing files or paths. {agent-pull}7305[#7305] {agent-issue}7301[#7301] +* Make `otelcol` script executable in the Docker image. {agent-pull}7345[#7345] +* Fix {es} exporter configuration in `kube-stack` values. {agent-pull}7380[#7380] + +// end 8.17.4 relnotes + +// begin 8.17.3 relnotes + +[[release-notes-8.17.3]] +== {fleet} and {agent} 8.17.3 + +Review important information about the {fleet} and {agent} 8.17.3 release. + +[discrete] +[[known-issues-8.17.3]] +=== Known issues + +[[known-issue-1749-8-17-3]] +.Inaccurate option to upgrade across major releases appears in {fleet} UI +[%collapsible] +==== + +*Details* + +In late 8.x releases, there's a bug in the {fleet} UI that causes `9.0.0` to appear as an option for {agent} upgrades. This option results in an error when selected, as upgrades across major releases are not supported for {agent}. For example, an agent can't be upgraded to version 9.0 while {kib} and {fleet-server} are on version 8.x. + +The {agent} upgrade is not selectable: + +image::images/upgrade-agent-not-selectable.png[{agent} upgrade is not selectable] + +The {fleet-server} upgrade is selectable, with an inconsistent UI state and error on submit: + +image::images/upgrade-fleet-server-inconsistent-state.png[{fleet-server} inconsistent UI state] + +*Impact* + + +In the 9.x releases, the option that appears in the UI for an upgrade across a major release should be ignored. + +==== + +[[known-issue-1800-8-17-3]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[enhancements-8.17.3]] +=== Enhancements + +{agent}:: +* Add the configuration files for the Elastic Distribution of OTel Collector that will be provided during the {kib} OpenTelemetry Host and Kubernetes onboarding flow for MOTel. {agent-pull}6641[#6630] + +[discrete] +[[bug-fixes-8.17.3]] +=== Bug fixes + +{agent}:: +* Add missing checks for null values in AST collection nodes. {agent-pull}7009[#7009] {agent-issue}6999[#6999] +* Set the gateway Kubernetes `spec.replicas` for the gateway OpenTelemetry Collector to prevent horizontal Pod autoscaler fails. {agent-pull}7011[#7011] + +// end 8.17.3 relnotes + +// begin 8.17.2 relnotes + +[[release-notes-8.17.2]] +== {fleet} and {agent} 8.17.2 + +Review important information about the {fleet} and {agent} 8.17.2 release. + +[discrete] +[[security-updates-8.17.2]] +=== Security updates + +{fleet-server}:: +* Upgrade `golang.org/x/net` to v0.34.0 and `golang.org/x/crypto` to v0.32.0. {fleet-server-pull}4405[#4405] + +[discrete] +[[known-issues-8.17.2]] +=== Known issues + +[[known-issue-1800-8-17-2]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[enhancements-8.17.2]] +=== Enhancements + +{agent}:: +* Upgrade NodeJS for Heartbeat to LTS v18.20.6. {agent-pull}6641[#6641] + +[discrete] +[[bug-fixes-8.17.2]] +=== Bug fixes + +{agent}:: +* Emit variables even if provider data is empty from the start. {agent-pull}6598[#6598] + +// end 8.17.2 relnotes + +// begin 8.17.1 relnotes + +[[release-notes-8.17.1]] +== {fleet} and {agent} 8.17.1 + +Review important information about the {fleet} and {agent} 8.17.1 release. + +[discrete] +[[breaking-changes-8.17.1]] +=== Breaking changes + +Breaking changes can prevent your application from optimal operation and +performance. Before you upgrade, review the breaking changes, then mitigate the +impact to your application. + +{agent}:: +* {agent} Docker images for {ecloud} have been reverted from having been based off of Ubuntu 24.04 to being based off of Ubuntu 20.04. This is to ensure compatibility with {ece}, support for new Wolfi-based images, and for GNU C Library (glibc) compatibility. {agent-pull}6393[#6393] + +[discrete] +[[known-issues-8.17.1]] +=== Known issues + +[[known-issue-1671]] +.{kib} out of memory crashes on 1 GB {ecloud} {kib} instances using {elastic-sec} view +[%collapsible] +==== + +*Details* + +{ecloud} deployments that use the smallest available {kib} instance size of 1 GB may crash due to out of memory errors when the Security UI is loaded. + +*Impact* + + +The root cause is inefficient memory allocation, and this is exacerbated when the prebuilt security rules package is installed on the initial load of the {elastic-sec} UI. + +As a workaround, you can upgrade your deployment to 8.17.1 in which this issue has been resolved by https://github.com/elastic/kibana/pull/208869[#208869] and https://github.com/elastic/kibana/pull/208475[#208475]. + +==== + +[[known-issue-1800-8-17-1]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[new-features-8.17.1]] +=== New features + +The 8.17.1 release added the following new and notable features. + +{agent}:: +* Add the link:https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter[Otel loadbalancing exporter] to {agent}. {agent-pull}6315[#6315] + +[discrete] +[[enhancements-8.17.1]] +=== Enhancements + +{agent}:: + +* Respond with an error message in case the user running the `enroll` command and the user who is the owner of the {agent} program files don't match. {agent-pull}6144[#6144] {agent-issue}4889[#4889] +* Implement the `MarshalJSON` method on the `component.Component` struct, so that when the component model is logged, the output does not contain any secrets that might be part of the component model. {agent-pull}6329[#6329] {agent-issue}5675[#5675] + +[discrete] +[[bug-fixes-8.17.1]] +=== Bug fixes + +{fleet-server}:: +* Do not set the `unenrolled_at` attribute when the audit/unenroll API is called. {fleet-server-pull}4221[#4221] {agent-issue}6213[#6213] +* Remove PGP endpoint auth requirement so that air-gapped {agents} can retrieve a PGP key from {fleet-server}. {fleet-server-pull}4256[#4256] {fleet-server-issue}4255[#4255] + +{agent}:: +* During uninstall, call the audit or unenroll API before components are stopped, if {agent} is running a {fleet-server} instance. {agent-pull}6085[#6085] {agent-issue}5752[#5752] +* Update OTel components to v0.115.0. {agent-pull}6391[#6391] +* Restore the `cloud-defend` binary which was accidentally removed in version 8.17.0. {agent-pull}6470[#6470] {agent-issue}6469[#6469] + +// end 8.17.1 relnotes + +// begin 8.17.0 relnotes + +[[release-notes-8.17.0]] +== {fleet} and {agent} 8.17.0 + +Review important information about the {fleet} and {agent} 8.17.0 release. + +[discrete] +[[breaking-changes-8.17.0]] +=== Breaking changes + +Breaking changes can prevent your application from optimal operation and +performance. Before you upgrade, review the breaking changes, then mitigate the +impact to your application. + +{agent}:: +* {agent} is now compiled using Debian 11 and linked against glibc 2.31 instead of 2.19. Drops support for Debian 10. {agent-pull}5847[#5847] + +[discrete] +[[known-issues-8.17.0]] +=== Known Issues + +[discrete] +[[known-issue-6213-8-17-0]] +.An {agent} with the Defend integration may report an Orphaned status and will not be able to be issued an upgrade action through {fleet}. +[%collapsible] +==== +*Details* + +A known issue in the {agent} may prevent it from being targeted with an upgrade action for a future release. +This may occur if the Defend integration is used and the agent is stopped on a running instance for too long. +An agent may be stopped as part of an upgrade process. + +*Impact* + +A bug fix is present in the 8.17.1 release of {fleet} that will prevent this from occurring. + +If you have agents that are affected, the workaround is as follows: +[source,shell] +---- +# Get a Token to issue an update_by_query request: +curl -XPOST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-unenrolled" + +# Issue an update_by_query request that targets effected agents: +curl -XPOST -H 'Authorization: Bearer ${TOKEN}' -H 'x-elastic-product-origin:fleet' -H 'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "unenrolled_at" } }],"must_not": [{ "term": { "active": "false" } }]}},"script": {"source": "ctx._source.unenrolled_at = null;","lang": "painless"}}' +---- +==== + +[discrete] +[[new-features-8.17.0]] +=== New features + +The 8.17.0 release Added the following new and notable features. + +{fleet}:: +* Expose advanced file logging configuration in the UI. {kibana-pull}200274[#200274] + +{agent}:: +* Add support for running as a pre-existing user when installing in unprivileged mode, with technical preview support for pre-existing Windows Active Directory users. {agent-pull}5988[#5988] {agent-issue}4585[#4585] +* Add a new custom link:https://github.com/elastic/integrations/tree/main/packages/filestream[Filestream logs integration]. This will enable migration from the custom log integration which is based on a log input that is planned for deprecation. https://github.com/elastic/integrations/pull/11332[#11332]. + +[discrete] +[[enhancements-8.17.0]] +=== Enhancements + +{fleet}:: +* Filter the integrations/packages list shown in the UI depending on the `policy_templates_behavior` field. {kibana-pull}200605[#200605] +* Add a `@custom` component template to integrations index template's `composed_of` array. {kibana-pull}192731[#192731] + +{fleet-server}:: +* Update `elastic-agent-libs` to version `0.14.0`. {fleet-server-pull}4042[#4042] + +{agent}:: +* Enable persistence in the configuration provided with our OTel Collector distribution. {agent-pull}5549[#5549] +* Restrict using the CLI to upgrade for {fleet}-managed {agents}. {agent-pull}5864[#5864] {agent-issue}4890[#4890] +* Add `os_family`, `os_platform` and `os_version` to the {agent} host provider, enabling differentiating Linux distributions. This is required to support Debian 12 and other distributions that are moving away from traditional log files in favour of Journald. {agent-pull}5941[#5941] https://github.com/elastic/integrations/issues/10797[10797] https://github.com/elastic/integrations/pull/11618[11618] +* Emit Pod data only for Pods that are running in the Kubernetes provider. {agent-pull}6011[#6011] {agent-issue}5835[#5835] {agent-issue}5991[#5991] +* Remove {endpoint-sec} from Linux container images. {endpoint-sec} cannot run in containers since it has a systemd dependency. {agent-pull}6016[#6016] {agent-issue}5495[#5495] +* Update OTel components to v0.114.0. {agent-pull}6113[#6113] +* Redact common secrets like API keys and passwords in the output from `elastic-agent inspect` command. {agent-pull}6224[#6224] + +[discrete] +[[bug-fixes-8.17.0]] +=== Bug fixes + +{fleet}:: +* Allow creation of an integration policy with no agent policies. {kibana-pull}201051[#201051] + +{fleet-server}:: +* Fix `server.address` field which was appearing empty in HTTP request logs. {fleet-server-pull}4142[#4142] +* Remove a race condition that may occur when remote ES outputs are used. {fleet-server-pull}4171[#4171] {fleet-server-pull}4170[#4170] + +{agent}:: +* Make redaction of common keys in diagnostics case insensitive. {agent-pull}6109[#6109] + + +// end 8.17.0 relnotes + +// --------------------- +//TEMPLATE +//Use the following text as a template. Remember to replace the version info. + +// begin 8.7.x relnotes + +//[[release-notes-8.7.x]] +//== {fleet} and {agent} 8.7.x + +//Review important information about the {fleet} and {agent} 8.7.x release. + +//[discrete] +//[[security-updates-8.7.x]] +//=== Security updates + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[breaking-changes-8.7.x]] +//=== Breaking changes + +//Breaking changes can prevent your application from optimal operation and +//performance. Before you upgrade, review the breaking changes, then mitigate the +//impact to your application. + +//[discrete] +//[[breaking-PR#]] +//.Short description +//[%collapsible] +//==== +//*Details* + +// For more information, refer to {kibana-pull}PR[#PR]. + +//*Impact* + +// For more information, refer to {fleet-guide}/fleet-server.html[Fleet Server]. +//==== + +//[discrete] +//[[notable-changes-8.13.0]] +//=== Notable changes + +//The following are notable, non-breaking updates to be aware of: + +//* Changes to features that are in Technical Preview. +//* Changes to log formats. +//* Changes to non-public APIs. +//* Behaviour changes that repair critical bugs. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[known-issues-8.7.x]] +//=== Known issues + +//[[known-issue-issue#]] +//.Short description +//[%collapsible] +//==== + +//*Details* + +// + +//*Impact* + + +// + +//==== + +//[discrete] +//[[deprecations-8.7.x]] +//=== Deprecations + +//The following functionality is deprecated in 8.7.x, and will be removed in +//8.7.x. Deprecated functionality does not have an immediate impact on your +//application, but we strongly recommend you make the necessary updates after you +//upgrade to 8.7.x. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[new-features-8.7.x]] +//=== New features + +//The 8.7.x release Added the following new and notable features. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[enhancements-8.7.x]] +//=== Enhancements + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[bug-fixes-8.7.x]] +//=== Bug fixes + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +// end 8.7.x relnotes diff --git a/docs/en/ingest-management/release-notes/release-notes-8.18.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.18.asciidoc new file mode 100644 index 000000000..3a9503f53 --- /dev/null +++ b/docs/en/ingest-management/release-notes/release-notes-8.18.asciidoc @@ -0,0 +1,375 @@ +// Use these for links to issue and pulls. +:kibana-issue: https://github.com/elastic/kibana/issues/ +:kibana-pull: https://github.com/elastic/kibana/pull/ +:beats-issue: https://github.com/elastic/beats/issues/ +:beats-pull: https://github.com/elastic/beats/pull/ +:agent-libs-pull: https://github.com/elastic/elastic-agent-libs/pull/ +:agent-issue: https://github.com/elastic/elastic-agent/issues/ +:agent-pull: https://github.com/elastic/elastic-agent/pull/ +:fleet-server-issue: https://github.com/elastic/fleet-server/issues/ +:fleet-server-pull: https://github.com/elastic/fleet-server/pull/ + +[[release-notes]] += Release notes + +This section summarizes the changes in each release. + +* <> +* <> +* <> + +Also see: + +* {kibana-ref}/release-notes.html[{kib} release notes] +* {beats-ref}/release-notes.html[{beats} release notes] + +// begin 8.18.2 relnotes + +[[release-notes-8.18.2]] +== {fleet} and {agent} 8.18.2 + +Review important information about the 8.18.2 release. + +[discrete] +[[security-updates-8.18.2]] +=== Security updates + +{agent}:: +* Update Go version to v1.24.3 {agent-pull}8109[#8109] + +{fleet}:: +* Update Go version to v1.24.3 {fleet-server-pull}4891[#4891] + +[discrete] +[[known-issues-8.18.2]] +=== Known issues + +[[known-issue-1800-8-18-2]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +// end 8.18.2 relnotes + +// begin 8.18.1 relnotes + +[[release-notes-8.18.1]] +== {fleet} and {agent} 8.18.1 + +Review important information about the {fleet} and {agent} 8.18.1 release. + +[discrete] +[[known-issues-8.18.1]] +=== Known issues + +[[known-issue-1800-8-18-1]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[bug-fixes-8.18.1]] +=== Bug fixes + +{fleet-server}:: +* Add `grpcnotrace` build tags and ensure DCE (dead code elimination) is enabled. Reduce binary size by 34%. {fleet-server-pull}4784[#4784] + +// end 8.18.1 relnotes + +// begin 8.18.0 relnotes + +[[release-notes-8.18.0]] +== {fleet} and {agent} 8.18.0 + +Review important information about the {fleet} and {agent} 8.18.0 release. + +[discrete] +[[known-issues-8.18.0]] +=== Known issues + +[[known-issue-1749-8-18]] +.Inaccurate option to upgrade across major releases appears in {fleet} UI +[%collapsible] +==== + +*Details* + +In late 8.x releases, there's a bug in the {fleet} UI that causes `9.0.0` to appear as an option for {agent} upgrades. This option results in an error when selected, as upgrades across major releases are not supported for {agent}. For example, an agent can't be upgraded to version 9.0 while {kib} and {fleet-server} are on version 8.x. + +The {agent} upgrade is not selectable: + +image::images/upgrade-agent-not-selectable.png[{agent} upgrade is not selectable] + +The {fleet-server} upgrade is selectable, with an inconsistent UI state and error on submit: + +image::images/upgrade-fleet-server-inconsistent-state.png[{fleet-server} inconsistent UI state] + +*Impact* + + +In the 9.x releases, the option that appears in the UI for an upgrade across a major release should be ignored. + +==== + +[[known-issue-1800-8-18-0]] +.On Windows, {agent} is unable to re-enroll into {fleet} +[%collapsible] +==== + +*Details* + +There is a known issue where an {agent} installed on Windows and previously enrolled into {fleet} is unable to re-enroll. Attempting to enroll the {agent} fails with the following error: + +[source,shell] +---- +Error: the command is executed as root but the program files are not owned by the root user. +---- + +*Impact* + +The issue affects {agent} installed on Windows. Until a bug fix is available in a later release, you can temporarily resolve the issue by changing the ownership of the {agent} directory: + +[source,shell] +---- +icacls "C:\Program Files\Elastic\Agent" /setowner "NT AUTHORITY\SYSTEM" /t /l +---- + +After the output confirms all files were successfully processed, run the `enroll` command again. + +==== + +[discrete] +[[new-features-8.18.0]] +=== New features + +The 8.18.0 release Added the following new and notable features. + +{fleet}:: +* Add next steps and actions to the agentless integrations flyout. ({kibana-pull}203824[#203824]) +* Add support for selecting columns when exporting agents to CSV. ({kibana-pull}203103[#203103]) +* Add status tracking for agentless integrations. ({kibana-pull}199567[#199567]) + +{agent}:: +* Add ability to run the Elastic Distribution of OTel Collector at the same time as other inputs. This feature is in technical preview. {agent-pull}5767[#5767] {agent-issue}5796[#5796] +* Add a sample configuration to be used when deploying the link:https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack[OpenTelemetry Kube Stack Helm Chart]. {agent-pull}5822[#5822] +* Add the GeoIP OpenTelemetry processor to {agent}. {agent-pull}6134[#6134] +* Add the link:https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/routingconnector[OpenTelemetry routing connector] to the Elastic Distribution of OTel Collector. {agent-pull}6210[#6210] +* Add support for the link:https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter[OTel loadbalancing exporter] to {agent}. {agent-pull}6315[#6315] +* Add a new Kubernetes deployment of the Elastic Distribution of OTel Collector named "gateway", to simplify the daemonset collector configuration and unify managed/self-managed scenarios. {agent-pull}6444[#6444] +* Add the `components` command for {agent} in OTel mode, to list the supported components the the Elastic Distribution of OTel Collector includes. {agent-pull}6539[#6539] +* Add the `receivercreator` and `k8sobserver` components to the Elastic Distribution of OTel Collector to help cover autodiscovery scenarios in Kubernetes. {agent-pull}6561[#6561] +* Add the `kafkaexporter` and `kafkareceiver` the Elastic Distribution of OTel Collector to help prepare support for a Kafka output. {agent-pull}6593[#6593] {agent-issue}6562[#6562] +* Add the nopreceiver to the Elastic Distribution of OTel Collector. {agent-pull}6603[#6603] +* Change the default gRPC port to 0 when {agent} is run in a container. {agent-pull}6585[#6585] + +[discrete] +[[enhancements-8.18.0]] +=== Enhancements + +{fleet}:: +* Enable sub-feature privileges for {fleet}. ({kibana-pull}203182[#203182]) + +{fleet-server}:: +* Validate user pbkdf2 settings for FIPS compliance. {fleet-server-pull}4542[#4542] +* Update {fleet-server} Go version to 1.24.0. {fleet-server-pull}4543[#4543] + +{agent}:: +* Re-enable the OTel subcommand on Windows. {agent-pull}6068[#6068] {agent-issue}4976[#4976] {agent-issue}5710[#5710] +* Update the {agent} to only run composable providers if they are referenced in the agent policy. {agent-pull}6169[#6169] {agent-issue}3609[#3609] {agent-issue}4648[#4648] +* Add a flag to skip {fleet} audit or unenroll when uninstalling {agent}. {agent-pull}6206[#6206] {agent-issue}5757[#5757] +* Embed hints-based inputs in the {agent} Kubernetes container image. {agent-pull}6381[#6381] {agent-issue}5661[#5661] +* Add an error to the Windows Application Event Log if the `install`, `uninstall`, or `enroll` commands fail. {agent-pull}6410[#6410] {agent-issue}6338[#6338] +* Add a logger to print the status and code when an {agent} enrollment call to {fleet} fails. {agent-pull}6477[#6477] {agent-issue}6287[#6287] +* Update {agent} Go version to 1.24.0. {agent-pull}6932[#6932] +* Update OTel components to v0.120.x. {agent-pull}7443[#7443] + +[discrete] +[[bug-fixes-8.18.0]] +=== Bug fixes + +{fleet}:: +* Support `is_default` on integration deployment modes. ({kibana-pull}208284[#208284]) +* Fix a UI error caused when an agent becomes orphaned. ({kibana-pull}207746[#207746]) +* Restrict non-local {es} output types for agentless integrations and policies. ({kibana-pull}207296[#207296]) +* Fix API code to prevent bulk actions from timing out. ({kibana-pull}205735[#205735]) +* Fix generation of dynamic mapping for objects with specific subfields. ({kibana-pull}204104[#204104]) +* Fix logic to ensure that agents are only considered stuck in updating when an upgrade fails. ({kibana-pull}202126[#202126]) + +{fleet-server}:: +* Return a 429 error when the {fleet-server} connection limit is reached instead of silently closing connections. {fleet-server-pull}4402[#4402] + +{agent}:: +* Prevent installation of {elastic-defend} in emulated environment, in which it's not supported. {agent-pull}6095[#6095] {agent-issue}6082[#6082] +* Re-enable notifying {fleet} when [agent] is uninstalled on Windows. {agent-pull}6257[#6257] {agent-issue}5952[#5952] +* Log a warning on same version upgrade attempts and prevent the agent from reporting a failed upgrade state. {agent-pull}6273[#6273] {agent-issue}6186[#6186] +* Add retries for requesting download verifiers when upgrading an agent. {agent-pull}6276[#6276] {agent-issue}5163[#5163] +* Replace `list` with `items` from from `kibanaFetchToken` as `list` is deprecated in the API response and will be removed. {agent-pull}6437[#6437] {agent-issue}6023[#6023] +* Restore `cloud-defend` as an expected binary after it was accidentally removed from containers in 8.17.0 and later versions. {agent-pull}6470[#6470] {agent-issue}6469[#6469] +* Restore the `maintainer` label for container images rather than the default inherited from a base image. {agent-pull}6512[#6512] +* Fix enrollment for containerized {agent} when the enrollment token changes or the agent is unenrolled. {agent-pull}6568[#6568] {agent-issue}3586[#3586] +* Change how Windows process handles are obtained when assigning sub-processes to Job objects. {agent-pull}6825[#6825] + +// end 8.18.0 relnotes + +// --------------------- +//TEMPLATE +//Use the following text as a template. Remember to replace the version info. + +// begin 8.7.x relnotes + +//[[release-notes-8.7.x]] +//== {fleet} and {agent} 8.7.x + +//Review important information about the {fleet} and {agent} 8.7.x release. + +//[discrete] +//[[security-updates-8.7.x]] +//=== Security updates + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[breaking-changes-8.7.x]] +//=== Breaking changes + +//Breaking changes can prevent your application from optimal operation and +//performance. Before you upgrade, review the breaking changes, then mitigate the +//impact to your application. + +//[discrete] +//[[breaking-PR#]] +//.Short description +//[%collapsible] +//==== +//*Details* + +// For more information, refer to {kibana-pull}PR[#PR]. + +//*Impact* + +// For more information, refer to {fleet-guide}/fleet-server.html[Fleet Server]. +//==== + +//[discrete] +//[[notable-changes-8.13.0]] +//=== Notable changes + +//The following are notable, non-breaking updates to be aware of: + +//* Changes to features that are in Technical Preview. +//* Changes to log formats. +//* Changes to non-public APIs. +//* Behaviour changes that repair critical bugs. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[known-issues-8.7.x]] +//=== Known issues + +//[[known-issue-issue#]] +//.Short description +//[%collapsible] +//==== + +//*Details* + + +// + +//*Impact* + + +// + +//==== + +//[discrete] +//[[deprecations-8.7.x]] +//=== Deprecations + +//The following functionality is deprecated in 8.7.x, and will be removed in +//8.7.x. Deprecated functionality does not have an immediate impact on your +//application, but we strongly recommend you make the necessary updates after you +//upgrade to 8.7.x. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[new-features-8.7.x]] +//=== New features + +//The 8.7.x release Added the following new and notable features. + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[enhancements-8.7.x]] +//=== Enhancements + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +//[discrete] +//[[bug-fixes-8.7.x]] +//=== Bug fixes + +//{fleet}:: +//* add info + +//{agent}:: +//* add info + +// end 8.7.x relnotes diff --git a/docs/en/ingest-management/release-notes/release-notes-8.19.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.19.asciidoc new file mode 100644 index 000000000..420ff94d9 --- /dev/null +++ b/docs/en/ingest-management/release-notes/release-notes-8.19.asciidoc @@ -0,0 +1,635 @@ +// Use these for links to issue and pulls. +:kibana-issue: https://github.com/elastic/kibana/issues/ +:kibana-pull: https://github.com/elastic/kibana/pull/ +:beats-issue: https://github.com/elastic/beats/issues/ +:beats-pull: https://github.com/elastic/beats/pull/ +:agent-libs-pull: https://github.com/elastic/elastic-agent-libs/pull/ +:agent-issue: https://github.com/elastic/elastic-agent/issues/ +:agent-pull: https://github.com/elastic/elastic-agent/pull/ +:fleet-server-issue: https://github.com/elastic/fleet-server/issues/ +:fleet-server-pull: https://github.com/elastic/fleet-server/pull/ + +[[release-notes]] += Release notes + +This section summarizes the changes in each release. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +Also check: + +* {kibana-ref}/release-notes.html[{kib} release notes] +* {beats-ref}/release-notes.html[{beats} release notes] + +// begin 8.19.11 relnotes + +[[release-notes-8.19.11]] +== {fleet} and {agent} 8.19.11 + +[discrete] +[[security-updates-8.19.11]] +=== Security updates + +Fleet Server:: + +* Enforce maximum limits on UploadBegin and UploadComplete request body sizes. https://github.com/elastic/fleet-server/pull/6159[#6159] + +[discrete] +[[bug-fixes-8.19.11]] +=== Bug fixes + +Elastic Agent:: + +* OSQuery - Add list capabilities to directories for interactive users. https://github.com/elastic/elastic-agent/pull/12189[#12189] +* Emit the correct error message when the app lock cannot be acquired. https://github.com/elastic/elastic-agent/pull/12225[#12225] +* Fix elasticsearch retry behavior in OTEL runtime for reliable delivery. https://github.com/elastic/elastic-agent/pull/12455[#12455] + +// end 8.19.11 relnotes + +// begin 8.19.10 relnotes + +[[release-notes-8.19.10]] +== {fleet} and {agent} 8.19.10 + +[discrete] +[[features-enhancements-8.19.10]] +=== New features and enhancements + +Elastic Agent:: + +* Update OTel Collector components to v0.141.0. https://github.com/elastic/elastic-agent/pull/11671[#11671] + +[discrete] +[[bug-fixes-8.19.10]] +=== Bug fixes + +Elastic Agent:: + +* Fix signature verification using the upgrade command with the --source-uri flag for fleet-managed agents. https://github.com/elastic/elastic-agent/pull/11826[#11826] https://github.com/elastic/elastic-agent/issues/11152[#11152] +* Fix FLEET_TOKEN_POLICY_NAME environment variable for the container command to handle cases where there are more than 20 agent policies available. https://github.com/elastic/elastic-agent/pull/12073[#12073] https://github.com/elastic/elastic-agent/issues/12069[#12069] + +// end 8.19.10 relnotes + +// begin 8.19.9 relnotes + +[[release-notes-8.19.9]] +== {fleet} and {agent} 8.19.9 + +[discrete] +[[features-enhancements-8.19.9]] +=== New features and enhancements + +Elastic Agent:: + +* Allow setting component runtime per input type. https://github.com/elastic/elastic-agent/pull/11186[#11186] + +[discrete] +[[bug-fixes-8.19.9]] +=== Bug fixes + +Elastic Agent:: + +* Ensure the self-monitoring configuration accounts for the runtime components actually run in. https://github.com/elastic/elastic-agent/pull/11300[#11300] +* Allow host to be a string for OTel configuration translation. https://github.com/elastic/elastic-agent/pull/11394[#11394] https://github.com/elastic/elastic-agent/issues/11352[#11352] +* Add environment.yaml file to diagnostics. https://github.com/elastic/elastic-agent/pull/11484[#11484] https://github.com/elastic/elastic-agent/issues/10966[#10966] +* Merge multiple agent keys when loading config. https://github.com/elastic/elastic-agent/pull/11619[#11619] https://github.com/elastic/elastic-agent/issues/3717[#3717] + +// end 8.19.9 relnotes + +// begin 8.19.8 relnotes + +[[release-notes-8.19.8]] +== {fleet} and {agent} 8.19.8 + +[discrete] +[[security-updates-8.19.8]] +=== Security updates + +Elastic Agent:: + +* Redact secrets in slices. https://github.com/elastic/elastic-agent/pull/11271[#11271] + +[discrete] +[[bug-fixes-8.19.8]] +=== Bug fixes + +Elastic Agent:: + +* Fix filesource provider to work with kubernetes secret mounts. https://github.com/elastic/elastic-agent/pull/11050[#11050] +* Ensure the monitoring input for the OTel collector can only run inside the collector. https://github.com/elastic/elastic-agent/pull/11204[#11204] +* Fix a fatal startup error in Beats Receivers caused by truncation of long UTF-8 hostnames. https://github.com/elastic/elastic-agent/pull/11285[#11285] + +// end 8.19.8 relnotes + +// begin 8.19.7 relnotes + +[[release-notes-8.19.7]] +== {fleet} and {agent} 8.19.7 + +[discrete] +[[bug-fixes-8.19.7]] +=== Bug fixes + +Elastic Agent:: + +* Fix quoting of boolean values in Helm charts. https://github.com/elastic/elastic-agent/pull/10681[#10681] +* Fix issue where switching to OTEL runtime would cause data to be re-ingested. https://github.com/elastic/elastic-agent/pull/10857[#10857] +* Fix signal handling for the EDOT Collector. https://github.com/elastic/elastic-agent/pull/10908[#10908] +* Reload agent binary source settings as configured in Fleet. https://github.com/elastic/elastic-agent/pull/10993[#10993] + +Fleet Server:: + +* Fix issue that prevented `checkin local_metadata` from being updated. https://github.com/elastic/fleet-server/pull/5824[#5824] +* Fix issue where a malformed `components` field prevented agent authentication. https://github.com/elastic/fleet-server/pull/5858[#5858] + +// end 8.19.7 relnotes + +// begin 8.19.6 relnotes + +[[release-notes-8.19.6]] +== {fleet} and {agent} 8.19.6 + +[discrete] +[[features-enhancements-8.19.6]] +=== New features and enhancements + +Elastic Agent:: + +* Add logs_metrics_traces.yml sample in EDOT for Windows. https://github.com/elastic/elastic-agent/pull/10514[#10514] +* Include OTel Collector internal telemetry in Agent monitoring. https://github.com/elastic/elastic-agent/pull/9928[#9928] +* Update OTel Collector components to v0.137.0. https://github.com/elastic/elastic-agent/pull/10391[#10391] + +Fleet Server:: + +* Update Go to v1.25.3. https://github.com/elastic/fleet-server/pull/5699[#5699] + +[discrete] +[[bug-fixes-8.19.6]] +=== Bug fixes + +Elastic Agent:: + +* Improve logging to catch early errors on startup. https://github.com/elastic/elastic-agent/pull/10158[#10158] https://github.com/elastic/elastic-agent/issues/9099[#9099] +* Fix an incorrectly formatted log message when a provider fails. https://github.com/elastic/elastic-agent/pull/10217[#10217] +* Inspect: Handle components with slashes in their name. https://github.com/elastic/elastic-agent/pull/10442[#10442] + +// end 8.19.6 relnotes + +// begin 8.19.5 relnotes + +[[release-notes-8.19.5]] +== {fleet} and {agent} 8.19.5 + +[discrete] +[[features-enhancements-8.19.5]] +=== New features and enhancements + +Elastic Agent:: + +* Agent cleans up downloads directory and the new versioned home if upgrade fails. https://github.com/elastic/elastic-agent/pull/9386[#9386] https://github.com/elastic/elastic-agent/issues/5235[#5235] +* When there is a disk space error during an upgrade, agent responds with clean insufficient disk space error message. https://github.com/elastic/elastic-agent/pull/9392[#9392] https://github.com/elastic/elastic-agent/issues/5235[#5235] +* Add Headers Setter extension to EDOT Collector. https://github.com/elastic/elastic-agent/pull/9903[#9903] https://github.com/elastic/elastic-agent/issues/9889[#9889] +* Update OTel components to v0.132.0. https://github.com/elastic/elastic-agent/pull/9964[#9964] + +[discrete] +[[bug-fixes-8.19.5]] +=== Bug fixes + +Elastic Agent:: + +* Include aggregated agent status in HTTP liveness checks. https://github.com/elastic/elastic-agent/pull/9673[#9673] https://github.com/elastic/elastic-agent/issues/9576[#9576] +* Reduce-default-telemetry-frequency. https://github.com/elastic/elastic-agent/pull/9987[#9987] +* Fix stuck upgrade state by clearing coordinator overridden state after failed upgrade. https://github.com/elastic/elastic-agent/pull/9992[#9992] +* Include components units status in HTTP liveness checks. https://github.com/elastic/elastic-agent/pull/10060[#10060] https://github.com/elastic/elastic-agent/issues/8047[#8047] + +// end 8.19.5 relnotes + +// begin 8.19.4 relnotes + +[[release-notes-8.19.4]] +== {fleet} and {agent} 8.19.4 + +[discrete] +[[features-enhancements-8.19.4]] +=== New features and enhancements + +Elastic Agent:: +* Bump kube-stack Helm Chart to 0.9.1 and enable the cluster collector. link:https://github.com/elastic/elastic-agent/pull/9535[#9535] +* Enhanced loggers for easier debugging of upgrade related issues. link:https://github.com/elastic/elastic-agent/pull/9689[#9689] link:https://github.com/elastic/elastic-agent/issues/9536[#9536] + + +[discrete] +[[bug-fixes-8.19.4]] +=== Bug fixes + +Elastic Agent:: +* Redact secrets from pre-config, computed-config, components-expected, and components-actual files in diagnostics archive. link:https://github.com/elastic/elastic-agent/pull/9560[#9560] +* Retry service start command upon failure with 30-second delay. link:https://github.com/elastic/elastic-agent/pull/9313[#9313] +* Fix reporting of scheduled upgrade details across restarts and cancels. link:https://github.com/elastic/elastic-agent/pull/9562[#9562] link:https://github.com/elastic/elastic-agent/issues/8778[#8778] +* Enable root user to re-enroll unprivileged agent for mac and linux. link:https://github.com/elastic/elastic-agent/pull/9603[#9603] link:https://github.com/elastic/elastic-agent/issues/8544[#8544] +* Fix missing liveness healthcheck during container enrollment. link:https://github.com/elastic/elastic-agent/pull/9612[#9612] link:https://github.com/elastic/elastic-agent/issues/9611[#9611] +* Enable admin user to re-enroll unprivileged agent for windows. link:https://github.com/elastic/elastic-agent/pull/9623[#9623] link:https://github.com/elastic/elastic-agent/issues/8544[#8544] +* Treat exit code 284 from Endpoint binary as non-fatal. link:https://github.com/elastic/elastic-agent/pull/9687[#9687] +* Ensure failed upgrade actions are removed from queue and details are set. link:https://github.com/elastic/elastic-agent/pull/9634[#9634] link:https://github.com/elastic/elastic-agent/issues/9629[#9629] + + +Fleet Server:: +* Restore connection limiter. link:https://github.com/elastic/fleet-server/pull/5372[#5372] ++ +Restore connection level limiter to prevent OOM incidents. This limiter is used in addition to the request-level throttle so that once our in-flight requests reaches max_connections a 429 is returned, but if the total connections the server uses is over max_connections*1.1 the server drops the connection before the TLS handshake. +* Build fleet-server as fully static binary to restore OS matrix compatibility. link:https://github.com/elastic/fleet-server/pull/5392[#5392] link:https://github.com/elastic/fleet-server/issues/5262[#5262] + +// end 8.19.4 relnotes + +// begin 8.19.3 relnotes + +[[release-notes-8.19.3]] +== {fleet} and {agent} 8.19.3 + +Review important information about the 8.19.3 release. + +[discrete] +[[known-issues-8.19.3]] +=== Known issues + +[[known-issue-5515-8.19.3]] +.fleet-agents template is missing mappings +[%collapsible] +==== +include::release-notes-8.19.asciidoc[tag=known-issue-5515-8.19] +==== + +[discrete] +[[security-updates-8.19.3]] +=== Security updates + +Elastic Agent:: + +* Upgrade Go version to 1.24.6. {agent-pull}9287[#9287] + +[discrete] +[[new-features-8.19.3]] +=== New features + +The 8.19.3 release adds the following new and notable features. + +Elastic Agent:: + +* Adjust the timeout for Elastic Defend check command. {agent-pull}9213[#9213] + +[discrete] +[[enhancements-8.19.3]] +=== Enhancements + +Elastic Agent:: + +* Update OTel components to v0.130.0. {agent-pull}9343[#9343] + +[discrete] +[[bug-fixes-8.19.3]] +=== Bug fixes + +Elastic Agent:: + +* On Windows, retry saving the Agent information file to disk. {agent-pull}9224[#9224] {agent-issue}5862[#5862] +* Correct hints annotations parsing to resolve only `${Kubernetes.*}` placeholders instead of resolving all `${...}` patterns. {agent-pull}9307[#9307] +* Treat exit code 28 from endpoint binary as non-fatal. {agent-pull}9320[#9320] +* Fixed jitter backoff strategy reset. {agent-pull}9342[#9342] {agent-issue}8864[#8864] +* Fix Docker container failing to start with `no matching vars: ${Env.elasticsearch_api_key:}` and similar errors by restoring support for `:` to set default values. {agent-pull}9451[#9451] {agent-issue}9328[#9328] +* Fix deb upgrade by stopping `elastic-agent` service before stopping endpoint. {agent-pull}9462[#9462] + +Fleet Server:: + +* Fix 503 handling in enrollment. {fleet-server-pull}5232[#5232] {fleet-server-issue}5197[#5197] +* Remove extra ES search when preparing agent policy. {fleet-server-pull}5283[#5283] +* Reset trace links on bulk items when returning to pool. {fleet-server-pull}5317[#5317] + +// end 8.19.3 relnotes + +// begin 8.19.2 relnotes + +[[release-notes-8.19.2]] +== {fleet} and {agent} 8.19.2 + +Review important information about the 8.19.2 release. + +[discrete] +[[known-issues-8.19.2]] +=== Known issues + +[[known-issue-5515-8.19.2]] +.fleet-agents template is missing mappings +[%collapsible] +==== +include::release-notes-8.19.asciidoc[tag=known-issue-5515-8.19] +==== + +[discrete] +[[bug-fixes-8.19.2]] +=== Bug fixes + +Fleet Server:: + +* Fix upgrade details `download_rate` as string handling. {fleet-server-pull}5062[#5062] {fleet-server-pull}5232[#5232] {fleet-server-pull}5238[#5238] {fleet-server-pull}5176[#5176] {fleet-server-issue}5164[#5164] + +// end 8.19.2 relnotes + +// begin 8.19.1 relnotes + +[[release-notes-8.19.1]] +== {fleet} and {agent} 8.19.1 + +Review important information about the 8.19.1 release. + +[discrete] +[[known-issues-8.19.1]] +=== Known issues + +[[known-issue-5515-8.19.1]] +.fleet-agents template is missing mappings +[%collapsible] +==== +include::release-notes-8.19.asciidoc[tag=known-issue-5515-8.19] +==== + +[discrete] +[[new-features-8.19.1]] +=== New features + +The 8.19.1 release adds the following new and notable features. + +Elastic Agent:: + +* Add K8s leader elector OTel extension. {agent-pull}9065[#9065] + +[discrete] +[[enhancements-8.19.1]] +=== Enhancements + +Elastic Agent:: + +* Include the forwardconnector as an EDOT Collector component. {agent-pull}8753[#8753] +* Update OTel components to v0.129.0. {agent-pull}8819[#8819] +* Update APM config extension to v0.4.0. {agent-pull}8819[#8819] +* Update Elastic trace processor to v0.7.0. {agent-pull}8819[#8819] +* Update Elastic APM connector to v0.4.0. {agent-pull}8819[#8819] +* Update API key auth extension to v0.2.0. {agent-pull}8819[#8819] +* Update Elastic infra metrics processor to v0.16.0. {agent-pull}8819[#8819] +* Rename OTel collector config file in diagnostics from `otel-final.yaml` to `otel-merged.yaml`. {agent-pull}9052[#9052] + +[discrete] +[[bug-fixes-8.19.1]] +=== Bug fixes + +Elastic Agent:: + +* Remove incorrect logging that unprivileged installations are in beta. {agent-pull}8715[#8715] {agent-issue}8689[#8689] +* Ensure standalone Elastic Agent uses log level from configuration instead of persisted state. {agent-pull}8784[#8784] {agent-issue}8137[#8137] +* Resolve deadlocks in runtime checkin communication. {agent-pull}8881[#8881] {agent-issue}7944[#7944] +* Remove init.d support from RPM packages. {agent-pull}8896[#8896] {agent-issue}8840[#8840] + +// end 8.19.1 relnotes + +// begin 8.19.0 relnotes + +[[release-notes-8.19.0]] +== {fleet} and {agent} 8.19.0 + +Review important information about the {fleet} and {agent} 8.19.0 release. + +[discrete] +[[security-updates-8.19.0]] +=== Security updates + +{agent}:: +* Upgrade To Go 1.24.3. {agent-pull}8109[#8109] + +{fleet-server}:: +* Upgrade golang.org/x/net to v0.34.0 and golang.org/x/crypto to v0.32.0. {fleet-server-pull}4405[#4405] + +[discrete] +[[known-issues-8.19.0]] +=== Known issues + +[[known-issue-2444-8.19.0]] +.Setting the log level on individual {agents} is not possible +[%collapsible] +==== + +*Details* + + +There is a known issue where it is not possible to set the log level on individual {agents} as the *Agent logging level* setting is not available on the {agent}'s details page. + +*Impact* + + +No workaround is available at the moment, but a fix is expected to be available in a future patch release. Note that the agent logging level can still be set on a per-policy basis in the agent policy's *Settings* tab. + +==== + +[[known-issue-2398-8.19.0]] +.{agent} does not process Windows security events +[%collapsible] +==== + +*Details* + + +There is a known issue where {agent} does not process Windows security events on hosts running Windows 10, Windows 11, and Windows Server 2022. + +*Impact* + + +No workaround is available at the moment, but a fix is expected to be available in {agent} 8.19.1. + +==== + +[[known-issue-2285-8.19.0]] +.{agents} remain in an "Upgrade scheduled" state +[%collapsible] +==== + +*Details* + + +There is a known issue where {agent} remains in an `Upgrade scheduled` state when a scheduled {agent} upgrade is cancelled. Attempting to restart the upgrade on the UI returns an error: `The selected agent is not upgradeable: agent is already being upgraded.`. + +*Impact* + + +Until this issue is fixed in a later patch version, you can call the https://www.elastic.co/docs/api/doc/kibana/operation/operation-post-fleet-agents-agentid-upgrade[Upgrade an agent] endpoint of the Kibana Fleet API with the `force` parameter set to `true` to force-upgrade the {agent}: + +[source,shell] +---- +curl --request POST \ + --url https:///api/fleet/agents//upgrade \ + --user ":" \ + --header 'Content-Type: application/json' \ + --header 'kbn-xsrf: true' \ + --data '{"version": "","force": true}' +---- + +To force-upgrade multiple {agents}, call the https://www.elastic.co/docs/api/doc/kibana/operation/operation-post-fleet-agents-bulk-upgrade[Bulk upgrade agents] endpoint of the Kibana Fleet API with the `force` parameter set to `true`: + +[source,shell] +---- +curl --request POST \ + --url https:///api/fleet/agents/bulk_upgrade \ + --user ":" \ + --header 'Content-Type: application/json' \ + --header 'kbn-xsrf: true' \ + --data '{"version": "","force": true,"agents":[""]}' +---- + +==== + +[[known-issue-5515-8.19.0]] +.fleet-agents template is missing mappings +[%collapsible] +==== +// tag::known-issue-5515-8.19[] +*Details* + + +On May 2, 2025 a known issue was discovered that the `.fleet-agents` index template was missing a mapping for the `local_metadata.complete` attribute. This may cause agent checkins to be rejected and the agents to appear as offline. + +In this {fleet}'s logs this will appear as: +[source,shell] +---- +elastic fail 400: document_parsing_exception: [1:209] object mapping for [local_metadata] tried to parse field [local_metadata] as object, but found a concrete value +Eat bulk checkin error; Keep on truckin' +---- +And in the {agent} logs it will appear as: +[source,shell] +---- +"log.level":"error","@timestamp":"2025-04-22:12:35:25.295Z","message":"Eat bulk checkin error; Keep on truckin'","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.type":"fleet-server","error.message":"elastic fail 400: document_parsing_exception: [1:209] object mapping for [local_metadata] tried to parse field [local_metadata] as object, but found a concrete value","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0" +---- +This attribute was added to the template in versions: 8.17.11 8.18.3, and 8.19.3. + +Further investigation revealed that the `.fleet-agents` index template was not correctly applied due to an unchanged `_meta.managed_index_mappings_version` number. +This change also affects other attributes as well, such as `upgrade_attempts`, `namespaces`, `unprivileged`, and `unhealthy_reason`. +If there is an error related to any of these attributes, there will be a similar error message in the logs. + +*Impact* + + +Updating to a version with a fixed `_meta.managed_index_mappings_version` will correctly apply the new index template. +The fixed versions are 8.18.8, 8.19.4, 9.0.8, 9.1.4. +// end::known-issue-5515-8.19[] +==== + +[discrete] +[[new-features-8.19.0]] +=== New features + +The 8.19.0 release Added the following new and notable features. + +{agent}:: +* Set replicas for Gateway Collector. {agent-pull}7011[#7011] +* Add nopexporter to EDOT Collector. +* Set collectors fullnameOverride for EDOT kube-stack values. {agent-pull}7754[#7754] {agent-issue}7381[#7381] +* Add cumulativetodeltaprocessor to EDOT Collector. +* Add apmconfig and apikeyauth OTel extensions. +* Add bearertokenauth OTel extension. +* Remove resource/k8s processor and use k8sattributes processor for service attributes. {agent-pull}8599[#8599] ++ +This PR removes the `resource/k8s` processor in honour of the k8sattributes processor that +provides native support for the service attributes: +https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.127.0/processor/k8sattributesprocessor#configuring-recommended-resource-attributes ++ +This change is aligned with the respective semantic conventions' guidance: +https://opentelemetry.io/docs/specs/semconv/non-normative/k8s-attributes/#service-attributes +* Rename OTel collector config file in diagnostics from otel-final.yaml to otel-merged.yaml. + +{fleet-server}:: +* Add ability for enrollment to take an agent id. {fleet-server-pull}4290[#4290] {fleet-server-issue}4226[#4226] + +[discrete] +[[enhancements-8.19.0]] +=== Enhancements + +{agent}:: +* Allow upgrading deb or rpm agents when using Elastic Defend with tamper protection. {agent-pull}6907[#6907] {agent-issue}6394[#6394] +* Include all metadata that is sent to Fleet in the agent-info.yaml file in diagnostics by default. {agent-pull}7029[#7029] +* Add ApiKey prefix to Motel host configurations. {agent-pull}7063[#7063] +* Add elastic.agent.fips to local_metadata. {agent-pull}9159[#9159] {agent-pull}8939[#8939] {agent-pull}9029[#9029] {agent-pull}9095[#9095] {agent-pull}8671[#8671] {agent-pull}8672[#8672] {agent-pull}9143[#9143] {agent-pull}7112[#7112] {agent-issue}7112[#7112] +* Validate pbkdf2 settings when in FIPS mode. {agent-pull}7187[#7187] +* FIPS compliant agent file vault. {agent-pull}7360[#7360] +* With this change FIPS compliant agents will only be able to upgrade to other FIPS compliant agents. This change also restricts non-FIPS to FIPS upgrades as well. {agent-pull}7312[#7312] +* Updated the error messages returned for FIPS upgrades. {agent-pull}7453[#7453] +* Update OTel components to v0.121.0. +* Update OTel components to v0.122.0. {agent-pull}7725[#7725] +* Update OTel components to v0.123.0. {agent-pull}7996[#7996] +* Retry enrollment requests on any error. {agent-pull}8056[#8056] +* Update OTel components to v0.125.0. +* Update OTel components to v0.127.0. +* Remove deprecated OTel Elasticsearch exporter config `*_dynamic_index` from code and samples. {agent-pull}8592[#8592] +* Include the forwardconnector as an EDOT collector component. {agent-pull}8753[#8753] +* Update OTel components to v0.129.0. +* Update apm config extension to v0.4.0. +* Update Elastic trace processor to v0.7.0. +* Update Elastic APM connector to v0.4.0. +* Update API key auth extension to v0.2.0. +* Update Elastic infra metrics processor to v0.16.0. + +{fleet-server}:: +* Bump Go to v1.23.5. {fleet-server-pull}4353[#4353] +* Clear agent.upgrade_attempts when upgrade is complete. {fleet-server-pull}4528[#4528] +* Pbkdf2 settings validation is FIPS compliant. {fleet-server-pull}4542[#4542] +* Update to Go v1.24.0. {fleet-server-pull}4543[#4543] +* Add version metadata to version command output. {fleet-server-pull}4820[#4820] +* Update Go to v1.24.3. {fleet-server-pull}4891[#4891] + +[discrete] +[[upgrades-8.19.0]] +=== Upgrades + +{agent}:: +* Bump apmconfig extension to v0.3.0. + +[discrete] +[[bug-fixes-8.19.0]] +=== Bug fixes + +{agent}:: +* Fix TSDB version_conflict_engine_exception caused by incorrect kube-stack Helm values. {agent-pull}9159[#9159] {agent-pull}8939[#8939] {agent-pull}9029[#9029] {agent-pull}9095[#9095] {agent-pull}8671[#8671] {agent-pull}8672[#8672] {agent-pull}9143[#9143] {agent-pull}6928[#6928] +* Make enroll command backoff more conservative. {agent-pull}9159[#9159] {agent-pull}8939[#8939] {agent-pull}9029[#9029] {agent-pull}9095[#9095] {agent-pull}8671[#8671] {agent-pull}8672[#8672] {agent-pull}9143[#9143] {agent-pull}6983[#6983] {agent-issue}6761[#6761] +* Add missing null checks to AST methods. {agent-pull}9159[#9159] {agent-pull}8939[#8939] {agent-pull}9029[#9029] {agent-pull}9095[#9095] {agent-pull}8671[#8671] {agent-pull}8672[#8672] {agent-pull}9143[#9143] {agent-pull}7009[#7009] {agent-issue}6999[#6999] +* Fixes an issue where `fixpermissions` on Windows incorrectly returned an error message due to improper handling of Windows API return values. {agent-pull}7059[#7059] {agent-issue}6917[#6917] +* Support IPv6 hosts in enroll URL. {agent-pull}7036[#7036] +* Support IPv6 host in gRPC config. {agent-pull}7035[#7035] +* Support IPv6 host in agent monitoring HTTP config. {agent-pull}7073[#7073] +* Rotate logger output file when writing to a symbolic link. elastic-agent-pull}6938[#6938] +* Do not fail Windows permission updates on missing files/paths. {agent-pull}7305[#7305] {agent-issue}7301[#7301] +* Make `otelcol` executable in the Docker image. {agent-pull}9159[#9159] {agent-pull}8939[#8939] {agent-pull}9029[#9029] {agent-pull}9095[#9095] {agent-pull}8671[#8671] {agent-pull}8672[#8672] {agent-pull}9143[#9143] {agent-pull}7345[#7345] +* Fix Elasticsearch exporter configuration in kube-stack values. {agent-pull}9159[#9159] {agent-pull}8939[#8939] {agent-pull}9029[#9029] {agent-pull}9095[#9095] {agent-pull}8671[#8671] {agent-pull}8672[#8672] {agent-pull}9143[#9143] {agent-pull}7380[#7380] +* Ship journalctl in the elastic-agent, elastic-agent-complete, and elastic-agent-ubi Docker images to enable reading journald logs. Journalctl is not present on Wolfi images. {agent-pull}8492[#8492] https://github.com/elastic/beats/issues/44040[#44040] +* Preserve agent run state on DEB and RPM upgrades. {agent-pull}7999[#7999] {agent-issue}3832[#3832] +* Use --header from enrollment when communicating with Fleet Server. {agent-pull}8071[#8071] {agent-issue}6823[#6823] +* Address a race condition that can occur in agent diagnostics if log rotation runs while logs are being zipped. +* Use paths.tempdir for diagnostics actions. {agent-pull}8472[#8472] +* Use Debian 11 to build Linux/ARM to match Linux/AMD64. Upgrades Linux/ARM64's statically linked glibc from 2.28 to 2.31. {agent-pull}8497[#8497] +* Relax file ownership check to allow admin re-enrollment on Windows. {agent-pull}8503[#8503] {agent-issue}7794[#7794] +* Remove incorrect logging that unprivileged installations are in beta. {agent-pull}8715[#8715] {agent-issue}8689[#8689] +* Ensure standalone Elastic Agent uses log level from configuration instead of persisted state. {agent-pull}8784[#8784] {agent-issue}8137[#8137] +* Resolve deadlocks in runtime checkin communication. {agent-pull}8881[#8881] {agent-issue}7944[#7944] +* Removed init.d support from RPM packages. {agent-pull}8896[#8896] {agent-issue}8840[#8840] + +{fleet-server}:: +* Added context deadline around flush bulk queue. {fleet-server-pull}5179[#5179] {fleet-server-pull}5043[#5043] {fleet-server-pull}5062[#5062] {fleet-server-pull}5063[#5063] {fleet-server-pull}3986[#3986] +* Fix server.address field in HTTP logs. {fleet-server-pull}5179[#5179] {fleet-server-pull}5043[#5043] {fleet-server-pull}5062[#5062] {fleet-server-pull}5063[#5063] {fleet-server-pull}4142[#4142] +* Remove race in remote bulker access. {fleet-server-pull}5179[#5179] {fleet-server-pull}5043[#5043] {fleet-server-pull}5062[#5062] {fleet-server-pull}5063[#5063] {fleet-server-pull}4171[#4171] {fleet-server-issue}4170[#4170] +* Audit/unenroll should not set unenrolled_at attribute. {fleet-server-pull}4221[#4221] {fleet-server-issue}6213[#6213] +* Remove auth requirement from PGP key endpoint. {fleet-server-pull}5179[#5179] {fleet-server-pull}5043[#5043] {fleet-server-pull}5062[#5062] {fleet-server-pull}5063[#5063] {fleet-server-pull}4256[#4256] {fleet-server-issue}4255[#4255] +* Return HTTP 429 when connection limit is reached. {fleet-server-pull}5179[#5179] {fleet-server-pull}5062[#5062] {fleet-server-pull}5063[#5063] {fleet-server-pull}4402[#4402] {fleet-server-issue}4200[#4200] +* Fix host parsing in Elasticsearch output diagnostics. {fleet-server-pull}4765[#4765] +* Redact output in bootstrap config logs. {fleet-server-pull}4775[#4775] +* Mutex protection for remote bulker config. {fleet-server-pull}4776[#4776] +* Enable dead code elimination. {fleet-server-pull}4784[#4784] +* Include the base error for JSON decode error responses. {fleet-server-pull}5069[#5069] + +// end 8.19.0 relnotes diff --git a/docs/en/ingest-management/security/certificates.asciidoc b/docs/en/ingest-management/security/certificates.asciidoc index e2d5bb2d6..56dde3b56 100644 --- a/docs/en/ingest-management/security/certificates.asciidoc +++ b/docs/en/ingest-management/security/certificates.asciidoc @@ -7,6 +7,9 @@ in the {stack}. For the install settings specific to mutual TLS, as opposed to one-way TLS, refer to <>. +For a summary of flow by which TLS is established between components using +either one-way or mutual TLS, refer to <>. + TIP: Our {ess-product}[hosted {ess}] on {ecloud} provides secure, encrypted connections out of the box! @@ -34,6 +37,8 @@ openssl pkcs12 -in path.p12 -out private.key -nocerts -nodes Key passwords are not currently supported. ==== +IMPORTANT: When you run {agent} with the {elastic-defend} integration, the link:https://en.wikipedia.org/wiki/X.509[TLS certificates] used to connect to {fleet-server} and {es} need to be generated using link:https://en.wikipedia.org/wiki/RSA_(cryptosystem)[RSA]. For a full list of available algorithms to use when configuring TLS or mTLS, see <>. These settings are available for both standalone and {fleet}-managed {agent}. + [discrete] [[generate-fleet-server-certs]] == Generate a custom certificate and private key for {fleet-server} @@ -226,6 +231,7 @@ sudo ./elastic-agent install \ --fleet-server-port=8220 \ --elastic-agent-cert=/tmp/fleet-server.crt \ --elastic-agent-cert-key=/tmp/fleet-server.key \ + --elastic-agent-cert-key-passphrase=/tmp/fleet-server/passphrase-file \ --fleet-server-es-cert=/tmp/fleet-server.crt \ --fleet-server-es-cert-key=/tmp/fleet-server.key \ --fleet-server-client-auth=required @@ -258,6 +264,10 @@ to the other {agents} The certificate to use as the client certificate for {agent}'s connections to {fleet-server}. `elastic-agent-cert-key`:: The path to the private key to use as for {agent}'s connections to {fleet-server}. +`elastic-agent-cert-key`:: +The path to the file that contains the passphrase for the mutual TLS private key that {agent} will use to connect to {fleet-server}. +The file must only contain the characters of the passphrase, no newline or extra non-printing characters. +This option is only used if the `elastic-agent-cert-key` is encrypted and requires a passphrase to use. `fleet-server-es-cert`:: The path to the client certificate that {fleet-server} will use when connecting to {es}. `fleet-server-es-cert-key`:: diff --git a/docs/en/ingest-management/security/fleet-roles-and-privileges.asciidoc b/docs/en/ingest-management/security/fleet-roles-and-privileges.asciidoc index d2dece98e..dd1b460fb 100644 --- a/docs/en/ingest-management/security/fleet-roles-and-privileges.asciidoc +++ b/docs/en/ingest-management/security/fleet-roles-and-privileges.asciidoc @@ -7,39 +7,57 @@ Assigning the {kib} feature privileges `Fleet` and `Integrations` grants access `all`:: Grants full read-write access. `read`:: Grants read-only access. +`none`:: No access is granted. +Take advantage of these privilege settings by: + +* <> +* <> + +[discrete] +[[fleet-roles-and-privileges-built-in]] +== Built-in roles + +{es} comes with built-in roles that include default privileges. + +`editor`:: The built-in `editor` role grants the following privileges, supporting full read-write access to {fleet} and Integrations: -* {Fleet}: `All` -* Integrations: `All` +* {Fleet}: `all` +* Integrations: `all` +`viewer`:: The built-in `viewer` role grants the following privileges, supporting read-only access to {fleet} and Integrations: -* {Fleet}:: `None` -* Integrations:: `Read` +* {Fleet}: `read` +* Integrations: `read` -You can also create a new role that can be assigned to a user to grant access to {fleet} and Integrations. +You can also create a new role that can be assigned to a user, in order to grant more specific levels of access to {fleet} and Integrations. [discrete] [[fleet-roles-and-privileges-create]] == Create a role for {fleet} -To create a new role with full access to use and manage {fleet} and Integrations: +To create a new role with access to {fleet} and Integrations: . In {kib}, go to **Management -> Stack Management**. . In the **Security** section, select **Roles**. . Select **Create role**. . Specify a name for the role. . Leave the {es} settings at their defaults, or refer to {ref}/security-privileges.html[Security privileges] for descriptions of the available settings. -. In the {kib} section, select **Add Kibana privilege**. -. In the **Spaces** menu, select *** All Spaces**. Since many Integrations assets are shared across spaces, the users needs the {kib} privileges in all spaces. +. In the {kib} section, select **Assign to space**. +. In the **Spaces** menu, select *** All Spaces**. Since many Integrations assets are shared across spaces, the users need the {kib} privileges in all spaces. . Expand the **Management** section. -. Set **Fleet** privileges to **All**. -. Set **Integrations** privileges to **All**. +. Choose the access level that you'd like the role to have with respect to {fleet} and integrations: +.. To grant the role full access to use and manage {fleet} and integrations, set both the **Fleet** and **Integrations** privileges to `All`. ++ [role="screenshot"] -image::images/kibana-fleet-privileges.png[Kibana privileges flyout showing Fleet and Integrations set to All] +image::images/kibana-fleet-privileges-all.png[Kibana privileges flyout showing Fleet and Integrations set to All] -To create a read-only user for Integrations, follow the same steps as above but set the **Fleet** privileges to **None** and the **Integrations** privileges to **Read**. +.. Similarly, to create a read-only user for {fleet} and Integrations, set both the **Fleet** and **Integrations** privileges to `Read`. ++ +[role="screenshot"] +image::images/kibana-fleet-privileges-read.png[Kibana privileges flyout showing Fleet and Integrations set to All] -Read-only access to {fleet} is not currently supported but is planned for development in a later release. +Once you've created a new role you can assign it to any {es} user. You can edit the role at any time by returning to the **Roles** page in {kib}. diff --git a/docs/en/ingest-management/security/logstash-certificates.asciidoc b/docs/en/ingest-management/security/logstash-certificates.asciidoc index 92d59bfbe..e1b3927fe 100644 --- a/docs/en/ingest-management/security/logstash-certificates.asciidoc +++ b/docs/en/ingest-management/security/logstash-certificates.asciidoc @@ -112,12 +112,39 @@ add the following configuration. Replace the path to your file. path.config: "/etc/path/to/elastic-agent-pipeline.conf" ---- -In the `elastic-agent-pipeline.conf` file, add the pipeline configuration. Note -that the configuration needed for {ess} on {ecloud} is different from +In the `elastic-agent-pipeline.conf` file, add the pipeline configuration. +The configuration needed on {ecloud} is different from self-managed {es} clusters. If you copied the configuration shown in {fleet}, adjust it as needed. -{ess} example: +### {es-serverless} example + +[source,text] +---- +input { + elastic_agent { + port => 5044 + ssl_enabled => true + ssl_certificate_authorities => ["/path/to/ca.crt"] + ssl_certificate => "/path/to/logstash.crt" + ssl_key => "/path/to/logstash.pkcs8.key" + ssl_client_authentication => "required" + } +} + +output { + elasticsearch { + hosts => "ELASTICSEARCH_ENDPOINT_URL" <1> + api_key => "xxxx:xxxx" <2> + data_stream => true + } +} +---- +<1> The Elasticsearch endpoint URL for your {es-serverless} project is available in **Application endpoints, cluster and component IDs**. Select **Manage** next to your project to access this information. +<2> In {fleet}, you can generate this API key when you add a {ls} output. + + +### {ech} example [source,text] ---- @@ -137,16 +164,13 @@ output { cloud_id => "xxxx:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx=" <1> api_key => "xxxx:xxxx" <2> data_stream => true - ssl => true <3> } } ---- <1> Use the `cloud_id` shown on your deployment page in {ecloud}. <2> In {fleet}, you can generate this API key when you add a {ls} output. -<3> {ess} uses standard publicly trusted certificates, so there's no need -specify other SSL settings here. -Self-managed {es} cluster example: +=== Self-managed {es} cluster example [source,text] ---- diff --git a/docs/en/ingest-management/security/mutual-tls.asciidoc b/docs/en/ingest-management/security/mutual-tls.asciidoc index f0c26036f..acd276a5a 100644 --- a/docs/en/ingest-management/security/mutual-tls.asciidoc +++ b/docs/en/ingest-management/security/mutual-tls.asciidoc @@ -3,6 +3,8 @@ Mutual Transport Layer Security (mTLS) provides a higher level of security and trust compared to one-way TLS, where only the server presents a certificate. It ensures that not only the server is who it claims to be, but the client is also authenticated. This is particularly valuable in scenarios where both parties need to establish trust and validate each other's identities, such as in secure API communication, web services, or remote authentication. +For a summary of flow by which TLS is established between components using either one-way or mutual TLS, refer to <>. + * <> * <> * <> @@ -46,6 +48,8 @@ When mTLS is required, the secure setup between {agent}, {fleet}, and {fleet-ser .. If the {agent} policy contains mTLS configuration settings, those settings will take precedence over those used during enrollment: This includes both the mTLS settings used for connectivity between {agent} and {fleet-server} (and the {fleet} application in {kib}, for {fleet}-managed {agent}), and the settings used between {agent} and it's specified output. .. If the {agent} policy does not contain any TLS, mTLS, or proxy configuration settings, these settings will remain as they were specified when {agent} enrolled. Note that the initial TLS, mTLS, or proxy configuration settings can not be removed through the {agent} policy; they can only be updated. +IMPORTANT: When you run {agent} with the {elastic-defend} integration, the link:https://en.wikipedia.org/wiki/X.509[TLS certificates] used to connect to {fleet-server} and {es} need to be generated using link:https://en.wikipedia.org/wiki/RSA_(cryptosystem)[RSA]. For a full list of available algorithms to use when configuring TLS or mTLS, see <>. These settings are available for both standalone and {fleet}-managed {agent}. + [discrete] [[mutual-tls-on-premise]] == On-premise deployments @@ -68,6 +72,9 @@ During {agent} installation on premise use the following options: |`--elastic-agent-cert-key` |{agent} certificate key to present to {fleet-server} + +|`--elastic-agent-cert-key-passphrase` +|The path to the file that contains the passphrase for the mutual TLS private key that {agent} will use to connect to {fleet-server} |=== [discrete] @@ -179,6 +186,9 @@ During {agent} installation on premise use the following options: |`--elastic-agent-cert-key` |{agent}'s private certificate key used to decrypt the certificate + +|`--elastic-agent-cert-key-passphrase` +|The path to the file that contains the passphrase for the mutual TLS private key that {agent} will use to connect to {fleet-server} |=== [discrete] @@ -209,6 +219,9 @@ During {agent} installation on premise use the following options, similar to <> +* <> + +[discrete] +[[one-way-tls-connection]] +== Simple one-way TLS connection + +The following `elastic-agent install` command configures a {fleet-server} with the required certificates and certificate authorities to enable one-way TLS connections between the components involved: + +[source,shell] +---- +elastic-agent install --url=https://your-fleet-server.elastic.co:443 \ +--certificate-authorities=/path/to/fleet-ca \ +--fleet-server-es=https://es.elastic.com:443 \ +--fleet-server-es-ca=/path/to/es-ca \ +--fleet-server-cert=/path/to/fleet-cert \ +--fleet-server-cert-key=/path/to/fleet-cert-key \ +--fleet-server-service-token=FLEET-SERVER-SERVICE-TOKEN \ +--fleet-server-policy=FLEET-SERVER-POLICY-ID \ +--fleet-server-port=8220 +---- + +{agent} is configured with `fleet-ca` as the certificate authority that it needs to validate certificates from {fleet-server}. + +During the TLS connection setup, {fleet-server} presents its certificate `fleet-cert` to the agent and the agent (as a client) uses `fleet-ca` to validate the presented certificate. + +image::images/tls-overview-oneway-fs-agent.png[Diagram of one-way TLS connection between Fleet Server and Elastic Agent] + +{fleet-server} also establishes a secure connection to an {es} cluster. In this case, {fleet-server} is configured with the certificate authority from the {es} `es-ca`. {es} presents its certificate, `es-cert`, and {fleet-server} validates the presented certificate using the certificate authority `es-ca`. + +image::images/tls-overview-oneway-fs-es.png[Diagram of one-way TLS connection between Fleet Server and Elasticsearch] + +[discrete] +=== Relationship between components in a one-way TLS connection + +image::images/tls-overview-oneway-all.jpg[Diagram of one-way TLS connection between components] + +[discrete] +[[mutual-tls-connection]] +== Mutual TLS connection + +The following `elastic-agent install` command configures a {fleet-server} with the required certificates and certificate authorities to enable mutual TLS connections between the components involved: + +[source,shell] +---- +elastic-agent install --url=https://your-fleet-server.elastic.co:443 \ +--certificate-authorities=/path/to/fleet-ca,/path/to/agent-ca \ +--elastic-agent-cert=/path/to/agent-cert \ +--elastic-agent-cert-key=/path/to/agent-cert-key \ +--elastic-agent-cert-key=/path/to/agent-cert-key-passphrase \ +--fleet-server-es=https://es.elastic.com:443 \ +--fleet-server-es-ca=/path/to/es-ca \ +--fleet-server-es-cert=/path/to/fleet-es-cert \ +--fleet-server-es-cert-key=/path/to/fleet-es-cert-key \ +--fleet-server-cert=/path/to/fleet-cert \ +--fleet-server-cert-key=/path/to/fleet-cert-key \ +--fleet-server-client-auth=required \ +--fleet-server-service-token=FLEET-SERVER-SERVICE-TOKEN \ +--fleet-server-policy=FLEET-SERVER-POLICY-ID \ +--fleet-server-port=8220 +---- + +As with the <>, {agent} is configured with `fleet-ca` as the certificate authority that it needs to validate certificates from the {fleet-server}. {fleet-server} presents its certificate `fleet-cert` to the agent and the agent (as a client) uses `fleet-ca` to validate the presented certificate. + +To establish a mutual TLS connection, the agent presents its certificate, `agent-cert`, and {fleet-server} validates this certificate using the `agent-ca` that it has stored in memory. + +image::images/tls-overview-mutual-fs-agent.png[Diagram of mutual TLS connection between Fleet Server and Elastic Agent] + +{fleet-server} can also establish a mutual TLS connection to the {es} cluster. In this case, {fleet-server} is configured with the certificate authority from the {es} `es-ca` and uses this to validate the certificate `es-cert` presented to it by {es}. + +image::images/tls-overview-mutual-fs-es.png[Diagram of mutual TLS connection between Fleet Server and Elasticsearch] + +Note that you can also configure mutual TLS for {fleet-server} and {agent} <>. + +[discrete] +=== Relationship between components in a mutual TLS connection + +image::images/tls-overview-mutual-all.jpg[Diagram of mutual TLS connection between components] \ No newline at end of file diff --git a/docs/en/ingest-management/tab-widgets/event-logging-widget.asciidoc b/docs/en/ingest-management/tab-widgets/event-logging-widget.asciidoc new file mode 100644 index 000000000..3655aa465 --- /dev/null +++ b/docs/en/ingest-management/tab-widgets/event-logging-widget.asciidoc @@ -0,0 +1,95 @@ +++++ +
+
+ + + + + +
+
+++++ + +include::event-logging.asciidoc[tag=mac] + +++++ +
+ + + + +
+++++ \ No newline at end of file diff --git a/docs/en/ingest-management/tab-widgets/event-logging.asciidoc b/docs/en/ingest-management/tab-widgets/event-logging.asciidoc new file mode 100644 index 000000000..7ec2ddb09 --- /dev/null +++ b/docs/en/ingest-management/tab-widgets/event-logging.asciidoc @@ -0,0 +1,29 @@ +// tag::mac[] + +`/Library/Elastic/Agent/data/elastic-agent-*/logs/events/elastic-agent-event-log*.ndjson` + +// end::mac[] + +// tag::linux[] + +`/opt/Elastic/Agent/data/elastic-agent-*/logs/events/elastic-agent-event-log*.ndjson` + +// end::linux[] + +// tag::win[] + +`C:\Program Files\Elastic\Agent\data\elastic-agent-*\logs\events\elastic-agent-event-log*.ndjson` + +// end::win[] + +// tag::deb[] + +`/var/lib/elastic-agent/data/elastic-agent-*/logs/events/elastic-agent-event-log*.ndjson` + +// end::deb[] + +// tag::rpm[] + +`/var/lib/elastic-agent/data/elastic-agent-*/logs/events/elastic-agent-event-log*.ndjson` + +// end::rpm[] diff --git a/docs/en/ingest-management/tab-widgets/run-agent-image/content.asciidoc b/docs/en/ingest-management/tab-widgets/run-agent-image/content.asciidoc index 012c7e384..5be787635 100644 --- a/docs/en/ingest-management/tab-widgets/run-agent-image/content.asciidoc +++ b/docs/en/ingest-management/tab-widgets/run-agent-image/content.asciidoc @@ -6,7 +6,7 @@ docker run \ --env FLEET_ENROLL=1 \ <1> --env FLEET_URL= \ <2> --env FLEET_ENROLLMENT_TOKEN= \ <3> - --rm docker.elastic.co/beats/elastic-agent:{version} <4> + --rm docker.elastic.co/elastic-agent/elastic-agent:{version} <4> ---- <1> Set to 1 to enroll the {agent} into {fleet-server}. @@ -29,7 +29,7 @@ docker run \ --env FLEET_SERVER_SERVICE_TOKEN= \ <3> --env FLEET_SERVER_POLICY_ID= \ <4> -p 8220:8220 \ <5> - --rm docker.elastic.co/beats/elastic-agent:{version} <6> + --rm docker.elastic.co/elastic-agent/elastic-agent:{version} <6> ---- <1> Set to 1 to bootstrap Fleet Server on this Elastic Agent. diff --git a/docs/en/ingest-management/troubleshooting/troubleshooting.asciidoc b/docs/en/ingest-management/troubleshooting/troubleshooting.asciidoc index 76c2928a6..99f64a88c 100644 --- a/docs/en/ingest-management/troubleshooting/troubleshooting.asciidoc +++ b/docs/en/ingest-management/troubleshooting/troubleshooting.asciidoc @@ -60,6 +60,8 @@ Find troubleshooting information for {fleet}, {fleet-server}, and {agent} in the * <> * <> * <> +* <> +* <> [discrete] @@ -428,14 +430,19 @@ To download the diagnostics bundle for local viewing: . In the **Host** column, click the agent's name. -. Click the **Diagnostics** tab. +. Select the **Diagnostics** tab and click the **Request diagnostics .zip** button. ++ +[role="screenshot"] +image::images/collect-agent-diagnostics1.png[Collect agent diagnostics under agent details] -. Click the **Request diagnostics .zip** button. +. In the **Request Diagnostics** pop-up, select **Collect additional CPU metrics** if you'd like detailed CPU data. + [role="screenshot"] -image::images/collect-agent-diagnostics.png[Collect agent diagnostics under agent details] +image::images/collect-agent-diagnostics2.png[Collect agent diagnostics confirmation pop-up] + +. Click the **Request diagnostics** button. -Any in-progress or previously collected bundles for the {agent} will be listed on this page. +When available, the new diagnostic bundle will be listed on this page, as well as any in-progress or previously collected bundles for the {agent}. Note that the bundles are stored in {es} and are removed automatically after 7 days. You can also delete any previously created bundle by clicking the `trash can` icon. @@ -830,3 +837,145 @@ Error: error loading agent config: error loading raw config: fail to read config To resolve this, either install {agent} without the `--unprivileged` flag so that it has administrative access, or run the {agent} commands without the `sudo` prefix. +[discrete] +[[agent-kubernetes-kustomize]] +== Troubleshoot {agent} installation on Kubernetes, with Kustomize + +Potential issues during {agent} installation on Kubernetes can be categorized into two main areas: + +. <>. +. <>. + +[discrete] +[[agent-kustomize-manifest]] +=== Problems related to the creation of objects within the manifest + +When troubleshooting installations performed with https://github.com/kubernetes-sigs/kustomize[Kustomize], it's good practice to inspect the output of the rendered manifest. To do this, take the installation command provided by Kibana Onboarding and replace the final part, `| kubectl apply -f-`, with a redirection to a local file. This allows for easier analysis of the rendered output. + +For example, the following command, originally provided by {kib} for an {agent} Standalone installation, has been modified to redirect the output for troubleshooting purposes: + +[source,sh] +---- +kubectl kustomize https://github.com/elastic/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone\?ref\=v8.15.3 | sed -e 's/JUFQSV9LRVkl/ZDAyNnZaSUJ3eWIwSUlCT0duRGs6Q1JfYmJoVFRUQktoN2dXTkd0FNMtdw==/g' -e "s/%ES_HOST%/https:\/\/7a912e8674a34086eacd0e3d615e6048.us-west2.gcp.elastic-cloud.com:443/g" -e "s/%ONBOARDING_ID%/db687358-2c1f-4ec9-86e0-8f1baa4912ed/g" -e "s/\(docker.elastic.co\/beats\/elastic-agent:\).*$/\18.15.3/g" -e "/{CA_TRUSTED}/c\ " > elastic_agent_installation_complete_manifest.yaml +---- + +The previous command generates a local file named `elastic_agent_installation_complete_manifest.yaml`, which you can use for further analysis. It contains the complete set of resources required for the {agent} installation, including: + +* RBAC objects (`ServiceAccounts`, `Roles`, etc.) + +* `ConfigMaps` and `Secrets` for {agent} configuration + +* {agent} Standalone deployed as a `DaemonSet` + +* https://github.com/kubernetes/kube-state-metrics[Kube-state-metrics] deployed as a `Deployment` + +The content of this file is equivalent to what you'd obtain by following the <> steps, with the exception that `kube-state-metrics` is not included in the standalone method. + +**Possible issues** + +* If your user doesn't have *cluster-admin* privileges, the RBAC resources creation might fail. + +* Some Kubernetes security mechanisms (like https://kubernetes.io/docs/concepts/security/pod-security-standards/[Pod Security Standards]) could cause part of the manifest to be rejected, as `hostNetwork` access and `hostPath` volumes are required. + +* If you already have an installation of `kube-state-metrics`, it could cause part of the manifest installation to fail or to update your existing resources without notice. + +[discrete] +[[agent-kustomize-after]] +=== Failures occurring within specific components after installation + +If the installation is correct and all resources are deployed, but data is not flowing as expected (for example, you don't see any data on the *[Metrics Kubernetes] Cluster Overview* dashboard), check the following items: + +. Check resources status and ensure they are all in a `Running` state: ++ +[source,sh] +---- +kubectl get pods -n kube-system | grep elastic +kubectl get pods -n kube-system | grep kube-state-metrics +---- ++ +[NOTE] +==== +The default configuration assumes that both `kube-state-metrics` and the {agent} `DaemonSet` are deployed in the **same namespace** for communication purposes. If you change the namespace of any of the components, the agent configuration will need further policy updates. +==== + +. Describe the Pods if they are in a `Pending` state: ++ +[source,sh] +---- +kubectl describe -n kube-system +---- + +. Check the logs of elastic-agents and kube-state-metrics, and look for errors or warnings: ++ +[source,sh] +---- +kubectl logs -n kube-system +kubectl logs -n kube-system | grep -i error +kubectl logs -n kube-system | grep -i warn +---- ++ +[source,sh] +---- +kubectl logs -n kube-system +---- + +**Possible issues** + +* Connectivity, authorization, or authentication issues when connecting to {es}: ++ +Ensure the API Key and {es} destination endpoint used during the installation is correct and is reachable from within the Pods. ++ +In an already installed system, the API Key is stored in a `Secret` named `elastic-agent-creds-`, and the endpoint is configured in the `ConfigMap` `elastic-agent-configs-`. + +* Missing cluster-level metrics (provided by `kube-state-metrics`): ++ +As described in <>, the {agent} Pod acting as `leader` is responsible for retrieving cluster-level metrics from `kube-state-metrics` and delivering them to {ref}/data-streams.html[data streams] prefixed as `metrics-kubernetes.state_`. In order to troubleshoot a situation where these metrics are not appearing: ++ +. Determine which Pod owns the <> `lease` in the cluster, with: ++ +[source,sh] +---- +kubectl get lease -n kube-system elastic-agent-cluster-leader +---- ++ +. Check the logs of that Pod to see if there are errors when connecting to `kube-state-metrics` and if the `state_*` metrics are being sent to {es}. ++ +One way to check if `state_*` metrics are being delivered to {es} is to inspect log lines with the `"Non-zero metrics in the last 30s"` message and check the values of the `state_*` metrics within the line, with something like: ++ +[source,sh] +---- +kubectl logs -n kube-system elastic-agent-xxxx | grep "Non-zero metrics" | grep "state_" +---- ++ +If the previous command returns `"state_pod":{"events":213,"success":213}` or similar for all `state_*` metrics, it means the metrics are being delivered. ++ +. As a last resort, if you believe none of the Pods is acting as a leader, you can try deleting the `lease` to generate a new one: ++ +[source,sh] +---- +kubectl delete lease -n kube-system elastic-agent-cluster-leader +# wait a few seconds and check for the lease again +kubectl get lease -n kube-system elastic-agent-cluster-leader +---- + +* Performance problems: ++ +Monitor the CPU and Memory usage of the agents Pods and adjust the manifest requests and limits as needed. Refer to <> for more details about the needed resources. + +Extra resources for {agent} on Kubernetes troubleshooting and information: + +* <>. + +* https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-kustomize/default[{agent} Kustomize Templates] documentation and resources. + +* Other examples and manifests to deploy https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes[{agent} on Kubernetes]. + +[discrete] +[[agent-kubernetes-invalid-api-key]] +== Troubleshoot {agent} on Kubernetes seeing `invalid api key to authenticate with fleet` in logs + +If an agent was unenrolled from a Kubernetes cluster, there might be data remaining in `/var/lib/elastic-agent-managed/kube-system/state` on the node(s). Reenrolling an agent later on the same nodes might then result in `invalid api key to authenticate with fleet` error messages. + +To avoid these errors, make sure to delete this state-folder before enrolling a new agent. + +For more information, refer to issue link:https://github.com/elastic/elastic-agent/issues/3586[#3586].