[8.19] (backport #15427) Override fleet.ssl.certificate_authorities from env vars in container mode

[mergify]

What does this PR do?

When in container mode, override fleet.ssl.certificate_authorities from env vars FLEET_CA, KIBANA_CA, and ELASTICSEARCH_CA instead of using saved fleet.enc values.

Follows the same first-set-wins fallback chain used in setup_config.go.

Why is it important?

Without this override, an agent running in container mode will crash during startup if fleet.enc still contains a fleet.ssl.certificate_authorities path that no longer exists on disk.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

N/A

How to test this PR locally

Unit test: TestContainerEnvOverridesFleetTLSPaths
Integration test: TestContainerCMDTLSCertOverride
Manually: Using kind, deploy ECK cluster. Enroll an agent with FLEET_SERVER_CLIENT_AUTH=required and FLEET_CA pointing to a mounted CA bundle. Unmount the CA bundle, unset env vars, and verify that agent restarts healthy.

Related issues

Closes #15408
Relates #14408

---

This is an automatic backport of pull request #15427 done by Mergify.

[infra-vault-gh-plugin-prod]

💔 Build Failed

• Buildkite Build
• Commit: a4ba132

Failed CI Steps

• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - fips140=only Ubuntu 22.04

History

cc @lorienhu

[github-actions]

TL;DR

The FIPS-only unit-test job fails deterministically in the new TestContainerEnvOverridesFleetCA test, not in the container CA override implementation. The test creates and writes an encrypted fleet.enc; vault encryption uses an arbitrary GCM IV, which Go rejects under GODEBUG=fips140=only.

Remediation

• Add fipsutils.SkipIfFIPSOnly(t, "vault does not use NewGCMWithRandomNonce") at the start of TestContainerEnvOverridesFleetCA, following the existing pattern in internal/pkg/crypto/io_aesgcm_replace_test.go:303, or refactor the fixture to avoid the non-FIPS vault/storage writer.
• Keep the non-FIPS unit/integration coverage for the override behavior and rerun GODEBUG="fips140=only,tlsmlkem=0" .buildkite/scripts/steps/unit-tests.sh.

Investigation details

Root Cause

internal/pkg/agent/cmd/container_test.go:1390 calls secret.CreateAgentSecret, and :1392-1393 creates an encrypted disk store used by the test cases. The first case then calls store.Save, which reaches vault encryption. The failing test is TestContainerEnvOverridesFleetCA at container_test.go:1315 in the Buildkite checkout. The production changes in internal/pkg/agent/cmd/container.go are not on the failing stack; the failure occurs during test setup.

The repository’s internal/pkg/testutils/fipsutils.SkipIfFIPSOnly explicitly skips tests that require legacy NewGCM behavior, and the existing crypto compatibility test uses this guard.

Evidence

• Build: https://buildkite.com/elastic/elastic-agent/builds/42813
• Job/step: Unit tests - fips140=only Ubuntu 22.04
• Key log excerpt: vault Set: could not encrypt key: crypto/cipher: use of GCM with arbitrary IVs is not allowed in FIPS 140-only mode, use NewGCMWithRandomNonce
• Result: DONE 3306 tests, 67 skipped, 1 failure

Verification

• The failure is reproduced in the provided Buildkite log at internal/pkg/agent/cmd/container_test.go:1315.
• No matching flaky-test issue was found, and the PR has no prior detective comment for this root cause.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.