Enable partial config reload for the otel collector

[swiatekm]

What does this PR do?

Enabled partial config reload for the otel collector. This includes two patched collector core packages with open-telemetry/opentelemetry-collector#15397 included as a single commit. The intent is to move this to the elastic official fork if it's not merged in time for 9.5.0.

Why is it important?

We depends on this feature to be able to use the otel runtime for dynamic inputs. Filestream in Kubernetes is the primary example.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

Should be invisible to users and quietly improve performance in most cases on config changes.

How to test this PR locally

Related issues

• Closes https://github.com/elastic/ingest-dev/issues/7746

[ycombinator]

Instead of using your personal fork, could we update https://github.com/elastic/opentelemetry-collector and use that instead?

[swiatekm]

The plan is to do that before release if we still need the patch. There's some ownership and admin bureaucracy involved in that fork right now that's being resolved.

[ycombinator]

Since the partial reload feature upstream is marked as alpha, should we provide a feature flag for Elastic Agent users to opt out of it (default = opt in). We don't need to document it necessarily; but it might be a useful escape hatch in case we get SDHs around this feature.

[swiatekm]

That's a good idea, I can add a switch in agent.internal. It'll be possible to shoot yourself in the foot by disabling it without care, but that's true about a lot of internal settings already.

[github-actions]

TL;DR

golangci-lint failed in all three define build-tag jobs because testing/integration/ess/otel_test.go is now being linted as a changed file and has 45 findings. Either revert the one-line change to that file if it is not needed, or fix the full lint set in that file before rerunning the workflow.

Remediation

• If the testing/integration/ess/otel_test.go change is not required for this PR, restore the removed Stack: &define.Stack{}, line so the workflow no longer whole-file lints this unrelated integration test file.
• If the change is required, fix all reported lint findings in testing/integration/ess/otel_test.go: unchecked returns, wrapped-error comparisons, forbidden context.Background() in tests, missing ReadHeaderTimeout, net.Listen without context, ineffectual assignments, staticcheck findings, unnecessary conversions, and unused declarations.
• Rerun golangci-lint, especially the define matrix jobs (integration,requirefips,kubernetes_inner,mage,define).

Investigation details

Root Cause

The failed workflow is golangci-lint for PR #15381. The default-tag jobs passed, while lint (ubuntu-latest, define), lint (macos-latest, define), and lint (windows-latest, define) failed in the golangci-lint step.

The PR only changes testing/integration/ess/otel_test.go by removing one line:

-		Stack: &define.Stack{},

Because this workflow runs golangci-lint with --whole-files on changed files, that one-line touch exposes existing/new lint findings across the whole file under the define build tags. This is broader than the latest prior detective report, which called out only unchecked return values; this run reports 45 findings across 9 lint categories.

Evidence

• Workflow: https://github.com/elastic/elastic-agent/actions/runs/29027292991
• Job/step: lint (ubuntu-latest, define), lint (macos-latest, define), lint (windows-latest, define) / golangci-lint
• Key log excerpt:

testing/integration/ess/otel_test.go:721:22: Error return value of `aTesting.RunProcess` is not checked (errcheck)
testing/integration/ess/otel_test.go:733:28: Error return value of `fixture.RunOtelWithClient` is not checked (errcheck)
testing/integration/ess/otel_test.go:300:32: comparing with == will fail on wrapped errors. Use errors.Is to check for a specific error (errorlint)
testing/integration/ess/otel_test.go:2778:37: use of `context.Background` forbidden because "use t.Context() in tests so the context is cancelled when the test ends" (forbidigo)
testing/integration/ess/otel_test.go:2743:17: G112: Potential Slowloris Attack because ReadHeaderTimeout is not configured in the http.Server (gosec)
testing/integration/ess/otel_test.go:961:2: ineffectual assignment to err (ineffassign)
testing/integration/ess/otel_test.go:2744:29: net.Listen must not be called. use (*net.ListenConfig).Listen (noctx)
testing/integration/ess/otel_test.go:297:3: S1039: unnecessary use of fmt.Sprintf (staticcheck)
testing/integration/ess/otel_test.go:455:53: unnecessary conversion (unconvert)
testing/integration/ess/otel_test.go:857:6: func mapAtLeastOneTrue is unused (unused)
45 issues:
* errcheck: 19
* errorlint: 2
* forbidigo: 5
* gosec: 1
* ineffassign: 5
* noctx: 1
* staticcheck: 5
* unconvert: 5
* unused: 2

Validation

• Not run locally; this was a read-only CI log investigation.

Follow-up

• The lowest-risk path appears to be restoring the removed Stack line unless that integration-test metadata change is intentional.

---

What is this? | From workflow: PR Actions Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[infra-vault-gh-plugin-prod]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 312733c

Failed CI Steps

• Windows:2022:amd64:sudo
• Windows:2022:amd64:sudo
• Windows:2025:amd64:sudo
• linux:amd64:tier1:sudo:fleet
• linux:amd64:tier1:sudo:fleet-endpoint-security
• linux:amd64:tier1:sudo:fleet-endpoint-security

History

• 💔 Build #42761 failed 4148486
• 💔 Build #42716 failed ef1d38a
• 💔 Build #42712 failed bad3fc2
• 💔 Build #42680 failed 7c45cf5

cc @swiatekm

[github-actions]

TL;DR

The Linux endpoint-security failure is a reproducible partial-reload shutdown symptom: after Elastic Defend is removed, filestream-monitoring remains in STOPPING, so TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/protected times out. This PR enables the partial-reload path and should be checked against the known shutdown deadlock tracked in #14049. The Windows failure is a separate Fleet startup-health timeout and should be retried independently.

Remediation

• Investigate the OTel/Beats receiver shutdown path used by service.partialReload,service.partialReloadReceivers; ensure a removed component reaches terminal stopped state and its status is cleared instead of leaving filestream-monitoring in STOPPING. Compare the collector/telemetry shutdown fix referenced from [Flaky Test]: TestInstallWithEndpointSecurityAndRemoveEndpointIntegration – Endpoint component or units are still present #14049 and verify the fork contains it.
• Re-run TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/protected on Linux after the shutdown fix, including the assertion at testing/integration/ess/endpoint_security_test.go:707-713.
• Retry Windows:2022:amd64:sudo separately. If it repeats, inspect the uploaded diagnostics for the component that prevents health convergence; the observed STOPPED -> STARTING startup pattern matches the existing leak-test flake coverage in [Flaky Test]: TestLongRunningAgentForLeaks – agent isn't healthy, current state: DEGRADED #12511.

Investigation details

Root Cause

The Linux test successfully installed and enrolled the agent, then failed only while removing Elastic Defend. The status poll repeatedly reported the filestream-monitoring component as STOPPING even though both its input and output units were reported healthy; the helper rejects any non-healthy component and never observed the endpoint component/units disappear. The PR changes the OTel collector runtime by enabling service.partialReload and service.partialReloadReceivers, which is the relevant path for this reload.

This is the same failure shape documented in flaky-test issue #14049: the component was left in STOPPING during config reload, and the investigation identified a deadlock in the Beats OTel telemetry bridge during receiver shutdown. That issue was closed as fixed by #14070, so the first check should be whether the collector fork/partial-reload changes in this PR bypass or regress that shutdown behavior.

The Windows test failed during initial health convergence: enrollment succeeded, but status polling saw Fleet STOPPED and then STARTING; cleanup began after the health wait and ExecStatus later returned context canceled. No component-specific failure is present in the supplied log, so this does not establish a second PR regression.

Evidence

• Build: https://buildkite.com/elastic/elastic-agent/builds/42781
• Linux job: linux:amd64:tier1:sudo:fleet-endpoint-security
• Windows job: Windows:2022:amd64:sudo
• Linux test: testing/integration/ess/endpoint_security_test.go:708
• Key Linux log excerpt:

endpoint_security_test.go:709: Component is not Healthy
{ID:filestream-monitoring ... State:STOPPING ...}
endpoint_security_test.go:709: Error getting agent state: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Error: Condition never satisfied
Test: TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/protected
Messages: Endpoint component or units are still present.

• Key Windows log excerpt:

tools.go:205: wanted fleet status to be HEALTHY, was STOPPED
tools.go:205: wanted fleet status to be HEALTHY, was STARTING
long_running_test.go:233: agent status returned an error: agent status failed: context canceled

The Linux teardown also reported GitHub authentication failure while destroying the ephemeral ESS cluster. That occurred after the integration test failure and is secondary infrastructure noise; the cluster is reported as ephemeral and will auto-expire.

Verification

Not run locally: these tests require the Buildkite Windows/Linux Fleet and ESS environments, and Docker-in-Docker is unavailable here.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.