[Stack 9.1.0-SNAPSHOT] Too many packages failing in daily job

[elastic-vault-github-plugin-prod]

• Stack version: 9.1.0-SNAPSHOT
• Packages:
  • 1password
  • akamai
  • atlassian_confluence
  • auditd_manager
  • auth0
  • aws
  • bbot
  • beelzebub
  • bitdefender
  • bitwarden
  • box_events
  • carbonblack_edr
  • cisco_meraki
  • cisco_secure_endpoint
  • cloudflare_logpush
  • f5_bigip
  • forgerock
  • gigamon
  • github
  • google_cloud_storage
  • google_scc
  • hpe_aruba_cx
  • http_endpoint
  • infoblox_bloxone_ddi
  • jamf_compliance_reporter
  • jamf_pro
  • jamf_protect
  • microsoft_defender_endpoint
  • microsoft_exchange_online_message_trace
  • mimecast
  • mysql
  • nats
  • okta
  • opencanary
  • ping_one
  • proofpoint_on_demand
  • proofpoint_tap
  • slack
  • sophos_central
  • sysdig
  • system
  • tanium
  • tenable_sc
  • ti_anomali
  • ti_eset
  • ti_misp
  • ti_rapid7_threat_command
  • trend_micro_vision_one
  • websocket
  • wiz
  • zeronetworks
  • zoom
  • zscaler_zia
• Owners:
  • @elastic/ecosystem

First build failed: https://buildkite.com/elastic/integrations/builds/26453

Latest 5 failed builds:

• https://buildkite.com/elastic/integrations/builds/27791
• https://buildkite.com/elastic/integrations/builds/27798
• https://buildkite.com/elastic/integrations/builds/27814
• https://buildkite.com/elastic/integrations/builds/27879
• https://buildkite.com/elastic/integrations/builds/27946

[mrodm]

Relates also to the errors in #14021 and #14020

Most of the errors in those stack versions (9.1.0-SNAPSHOT and 8.19.0-SNAPSHOT) are related to Elastic Agent logs.
Checking the failures in this build: https://buildkite.com/elastic/integrations/builds/26950

• auth0, bitdefender, gigamon, http_endpoint, jamf_pro, ping_one, sysdig, ti_anomali, wiz... (not a complete list)

test case failed: one or more errors found while examining elastic-agent.logs3251103976: [0] found error "Unit state changed http_endpoint-default-http_endpoint-ping_one-15a31d88-7eb5-4f41-a632-de1672cb4e90 (HEALTHY->DEGRADED): request did not validate: only POST requests are allowed

test case failed: one or more errors found while examining elastic-agent.logs1313329235: [0] found error "Unit state changed http_endpoint-default-http_endpoint-general_settings-e98bc29a-2593-4c89-a953-7bdac7dd05e1 (HEALTHY->DEGRADED): request did not validate: only POST requests are allowed"

• google_cloud_storage:

test case failed: one or more errors found while examining elastic-agent.logs3008678938: [0] found error "Unit state changed gcs-default-gcs-gcs-0481559e-59d5-433b-a411-191e08da8aae (HEALTHY->DEGRADED): found unsupported content-type: text/plain; charset=utf-8

proofpoint_on_demand, websocket (not a complete list):

test case failed: one or more errors found while examining elastic-agent.logs2610086529: [0] found error "Unit state changed streaming-default-websocket-websocket-6f0121bc-7dd5-4e21-823a-8484cf512825 (HEALTHY->DEGRADED): websocket connection encountered an error: websocket: close 1006 (abnormal closure): unexpected EOF"
found error "Unit state changed streaming-default-websocket-websocket-6f0121bc-7dd5-4e21-823a-8484cf512825 (HEALTHY->DEGRADED): websocket connection encountered an error: websocket: close 1006 (abnormal closure): unexpected EOF"

There are other packages, where the Fleet Server container could not start on time and elastic-package stack up -v -d command failed. In that build, these packages failed because of that:

• cisco_nexus
• jamf_protect
• journald
• kafka
• proofpoint_tap
• slack
• windows
• windows_etw

I could not find any log in those builds. Doing some test to try to catch those logs also in the CI here elastic/elastic-package#2642

Any idea what could be causing these errors in the logs in all these packages? Maybe could you help us here @elastic/elastic-agent-control-plane ?

cc @kpollich @jsoriano

[mrodm]

Trying to reproduce the Fleet Server container error locally:

• Most of the times, elastic-package stack up works successfully with no issues.
• When Fleet Server failed, at least from the docker-compose perspective, elasticsearch container is healthy.
• As the Fleet Server container keeps running in background (even with elastic-package stack up fails), after some minutes the Fleet Server container is connected and works.

The following logs are some (I think) that could be related to the failure:

{"log.level":"info","@timestamp":"2025-06-06T13:37:11.393Z","message":"Connecting to backoff(elasticsearch(http://localhost:9200))","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":138,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-06-06T13:37:11.393Z","message":"Error dialing dial tcp 127.0.0.1:9200: connect: connection refused","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func1"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"esclientleg","network.transport":"tcp","server.address":"localhost:9200","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-06T13:37:12.260Z","message":"ack policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"policyRevision":2,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet.policy.id":"fleet-server-policy","@timestamp":"2025-06-06T13:37:12.26Z","http.request.id":"01JX2QBD91R9NZ3G9JNMX3GP6D","server.address":"127.0.0.1:8221","fleet.agent.id":"9043a79c-4192-475b-8f74-0d7bf8dc83df","fleet.access.apikey.id":"max1RZcBrny30iILdF1Q","nEvents":1,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-06T13:37:12.260Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed filestream-monitoring (HEALTHY->CONFIGURING): Configuring","log":{"source":"elastic-agent"},"component":{"id":"filestream-monitoring","state":"HEALTHY"},"unit":{"id":"filestream-monitoring","type":"output","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
...
{"log.level":"warn","@timestamp":"2025-06-06T13:37:11.971Z","message":"Failed to read API Key roles","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"error.message":"[404 Not Found] {\"api_keys\":[]}: api key not found","ecs.version":"1.6.0","service.name":"fleet-server","http.request.id":"01JX2QBD91R9NZ3G9JNMX3GP6D","server.address":"127.0.0.1:8221","fleet.access.apikey.id":"max1RZcBrny30iILdF1Q","fleet.apikey.id":"m6x1RZcBrny30iILs12L","fleet.policy.output.name":"default","service.type":"fleet-server","fleet.agent.id":"9043a79c-4192-475b-8f74-0d7bf8dc83df","nEvents":1,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-06T13:37:12.030Z","message":"Connecting to backoff(elasticsearch(http://localhost:9200))","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":138,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-06-06T13:37:12.030Z","message":"Error dialing dial tcp 127.0.0.1:9200: connect: connection refused","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func1"},"network.transport":"tcp","server.address":"localhost:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
...
{"log.level":"error","@timestamp":"2025-06-06T13:37:13.436Z","message":"elastic-agent-client error: rpc error: code = Canceled desc = context canceled","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":467,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).watchErrChan"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-06T13:37:13.436Z","message":"Stopping 4 runners ...","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"centralmgmt","log.origin":{"file.line":192,"file.name":"cfgfile/list.go","function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-06-06T13:37:13.437Z","message":"elastic-agent-client error: rpc error: code = Canceled desc = context canceled","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":467,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).watchErrChan"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-06T13:37:13.437Z","message":"Stopping filebeat","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":538,"file.name":"beater/filebeat.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Is there any known issue in Fleet Server that could make fail the bootstrap process in this way? @elastic/elastic-agent-control-plane ?

The current configuration for this service in the docker-compose was working successfully in previous versions. These settings are the ones configured as healthcheck for Fleet Server container (3 attempts succeeded with a waiting time of 1 second between each):

  fleet-server:
    healthcheck:
      test: "bash /healthcheck.sh 3 1"
      start_period: 60s
      interval: 5s

And this is the script used: https://github.com/elastic/elastic-package/blob/main/internal/stack/_static/fleet-server-healthcheck.sh

One workaround for the fleet-server container could be tune these settings to be similar as the ones set for the elastic-agent container (timeout, start_period and retries):

  elastic-agent:
    healthcheck:
      test: "elastic-agent status"
      timeout: 2s
      start_period: 360s
      retries: 180
      interval: 5s

[mrodm]

Related to the Fleet Server container errors mentioned in #14022 (comment), I created PR to update health-check settings for the Fleet Server container elastic/elastic-package#2646
This should help to minimize the errors to start a new Elastic stack via the elastic-package stack up command.

[mrodm]

Both PRs to try to minimize the Fleet server errors during the start up of the Elastic stack (docker-compose) have been merged. These changes will be available in the next elastic-package release.

Looking at the latest Buildkite build, there are still errors found in Elastic Agent logs. Example of the errors are shown in #14022 (comment).

In CI, these errors are found for both stack versions 8.19.0-SNAPSHOT and 9.1.0-SNAPSHOT.

Testing just bitdefender package locally (with data stream push_notifications), I got the same errors for 9.0.2 and 9.0.3-SNAPSHOT stack versions.

Testing that same package with Stack versions 9.0.0 and 9.0.1, tests finished successfully.

This is how I tested bitdefender package:

cd packages/bitdefender

# start the Elastic stack with the required version (e.g. 9.0.2)
elastic-package stack up -v -d --version 9.0.2
elastic-package test system -v --data-streams push_notifications
elastic-package stack down -v

Focusing just on 5 different packages failing tests (auth0, bbot, bitdefender, cisco_meraki, jamf_pro), the data streams failing are using the input http_endpoint.

@cmacknz any idea?

cc @elastic/ecosystem

[mrodm]

Tests run in the latest Buildkite builds still fail with the same failures in the Elastic Agent logs mentioned in #14022 (comment)

Changes to solve Fleet Server startup or dump all container logs have been included in elastic-package v0.112.0, and that version is already used in the integrations repository via #14228.

[mrodm]

Same issues found in 9.2.0-SNAPSHOT: #14411

[cmacknz]

elastic-agent-client error: rpc error: code = Canceled desc = context canceled"

@michel-laterman have you ever seen fleet-server bootstrap fail with this error?

[cmacknz]

The various new failures where inputs are marked as degraded are probably a side effect of https://github.com/elastic/security-integrations/issues/289, where the security integrations team is enabling input health reporting in all inputs which means they agent will show as degraded for problems that were invisible before.