Propagate all feature flags to Beat receivers

[belimawr]

What does this PR do?

Preserves the original agent.features subtree in component feature metadata and copies it into generated Beat receiver configs. Removes the separate global FQDN-only propagation path so feature source metadata is authoritative and future Beat-supported flags propagate without translator changes.

Adds coverage for FQDN, log_input_run_as_filestream, aws_s3_v2, unknown future flags, explicit false values, and missing feature metadata.

Why is it important?

The OTel translator previously propagated only FQDN. Other Beat-supported feature flags were dropped, causing Beat receivers to behave differently from the Agent policy.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Documentation, default configuration, and new integration coverage are not required; this fixes internal propagation and existing receiver integration coverage exercises these flags.

Disruptive User Impact

None. Configured feature flags now consistently reach OTel Beat receivers.

How to test this PR locally

Run the tests

go test ./pkg/features
go test ./internal/pkg/otel/translate
go vet ./pkg/features ./internal/pkg/otel/translate

Manual test

1. Build and unpack a development Agent package

From the repository root:

mage dev:package
cd build/distributions
tar -xf elastic-agent-9.6.0-SNAPSHOT-linux-x86_64.tar.gz 
cd elastic-agent-9.6.0-SNAPSHOT-linux-x86_64

2. Create a test log and elastic-agent.yml

The policy deliberately configures a legacy log input, forces it to run as a Beat receiver, and enables agent.features.log_input_run_as_filestream.

printf 'manual feature propagation test line %04d\n' {1..200} > /tmp/elastic-agent-feature-test.log

elastic-agent.yml

agent:
  logging:
    level: debug
    to_stderr: true
  features:
    log_input_run_as_filestream:
      enabled: true
  internal:
    runtime:
      default: process
      filebeat:
        log: otel

outputs:
  default:
    type: elasticsearch
    hosts: [http://127.0.0.1:9200]
    username: elastic
    password: changeme

inputs:
  - id: manual-log-receiver
    type: log
    use_output: default
    data_stream:
      namespace: default
    streams:
      - id: manual-log-receiver.generic
        data_stream:
          dataset: generic
        paths:
          - /tmp/elastic-agent-feature-test.log

agent.grpc:
  port: 5050

3. Start Elastic Agent in standalone mode

./elastic-agent run -e 2>&1 | tee /tmp/elastic-agent-feature-test-output.log

Connection errors for 127.0.0.1:9200 are expected when Elasticsearch is not running and do not affect this test.

4. Verify that the log input started as Filestream

In another terminal:

grep "running as Filestream" /tmp/elastic-agent-feature-test-output.log | jq '[.log.source, .message]'
grep "Input 'filestream' starting" /tmp/elastic-agent-feature-test-output.log| jq '[.log.source, .message]'

The output should contain a Filebeat receiver log similar to:

[
  "log-default",
  "Log input (deprecated) running as Filestream input"
]
[
  "filestream-monitoring",
  "Input 'filestream' starting"
]
[
  "log-default",
  "Input 'filestream' starting"
]

The standalone policy still declares type: log; seeing input.filestream and Input 'filestream' starting confirms that agent.features.log_input_run_as_filestream.enabled: true reached the generated Beat receiver and was applied. Stop the Agent with Ctrl-C after the assertion succeeds.

Related issues

• Closes Pass all agent.features.* flags to Beat receivers #15437

[mergify]

This pull request does not have a backport label. Could you fix it @belimawr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[Copilot]

Pull request overview

This PR makes Beat receiver configuration generation in the OTel translator propagate the full agent.features subtree (from component feature metadata) instead of special-casing only fqdn. This aligns Beat receivers with the Agent policy’s feature flags and enables forward-compatible propagation of future Beat-supported flags without translator changes.

Changes:

• Preserve the raw agent.features subtree in proto.Features.Source during feature parsing (instead of deriving Source from typed fields only).
• Update the OTel translator to populate Beat receiver features.* from the component’s feature Source metadata and remove the separate FQDN-only injection path.
• Add/adjust tests to cover multiple propagated flags, explicit false, unknown future flags, and absence of feature metadata.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pkg/features/features.go	Reworks feature Source generation to preserve the raw agent.features subtree for downstream consumers.
pkg/features/features_test.go	Adds coverage to ensure Parse preserves the agent.features subtree (including unknown flags).
internal/pkg/otel/translate/otelconfig.go	Removes FQDN-only propagation and derives Beat receiver features from component feature Source metadata.
internal/pkg/otel/translate/otelconfig_test.go	Updates expectations and adds tests ensuring multiple flags propagate and missing feature metadata omits features.
changelog/fragments/1783638976-propagate-all-feature-flags-to-beat-receivers.yaml	Adds a changelog fragment documenting the bug fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

TL;DR

This build has two independent, deterministic failures. check-ci detects the beats submodule changing from 768b58b5d81c to af686c255e; all three package jobs fail compiling Beats code because its Quark API usage does not match the resolved Quark dependency. Neither failure is in the five application files changed by this PR.

Remediation

• Align the committed beats submodule pointer with the Beats module/dependency metadata: either run mage update and commit the generated pointer/module changes, or restore/pin the submodule to the intended 768b58b5d81c revision if the update is not intended.
• Ensure the selected Beats revision and its resolved Quark dependency are compatible. The package build must no longer report missing quark.QQ_NO_SNAPSHOT, quark.Queue.GetEvents, quark.Process.Filename, or quark.QUARK_EV_SNAPSHOT, nor pass two arguments to quark.OpenQueue when the API accepts one.
• Re-run make check-ci and the affected amd64/arm64 non-FIPS and FIPS package builds after the alignment.

Investigation details

Root Cause

check-ci

The check compares the worktree after the update/check steps and reports:

• committed submodule: 768b58b5d81c0d45295f3b871396b117390c4ddb
• generated submodule: af686c255e1f319b47524835f63b06d31365d47b

The failure is the repository hygiene check (Modified: [beats]), not the translator or feature-source changes.

Package builds

The arm64 non-FIPS, amd64 FIPS, and arm64 FIPS jobs all fail in the same Beats packages:

• beats/x-pack/auditbeat/processors/sessionmd/provider/kerneltracingprovider/kerneltracingprovider_linux.go
• beats/x-pack/auditbeat/module/system/process/quark_provider_linux.go

The compiler reports undefined: quark.QQ_NO_SNAPSHOT, too many arguments in call to quark.OpenQueue (have quark.QueueAttr, number; want quark.QueueAttr), missing Queue.GetEvents, missing Process.Filename, and undefined quark.QUARK_EV_SNAPSHOT. This is a Beats/Quark source-versus-dependency mismatch and is repeated across architectures and FIPS modes.

The PR diff only changes internal/pkg/otel/translate/otelconfig.go, internal/pkg/otel/translate/otelconfig_test.go, pkg/features/features.go, pkg/features/features_test.go, and a changelog fragment; it does not change either failing Beats package.

Evidence

• Build: https://buildkite.com/elastic/elastic-agent/builds/42789
• Jobs: check-ci, :package: arm64: Containers, :package: amd64: FIPS Containers, :package: arm64: FIPS Containers
• Key check-ci excerpt: diff --git a/beats b/beats followed by 768b58... to af686c... and Modified: [beats]
• Key package excerpt: kerneltracingprovider_linux.go:91:69: undefined: quark.QQ_NO_SNAPSHOT and quark_provider_linux.go:79:40: too many arguments in call to quark.OpenQueue

Verification

• Reviewed the complete pre-fetched logs for all four failed jobs.
• Reviewed PR Propagate all feature flags to Beat receivers #15458 diff and changed-file list.
• Searched open flaky-test issues for quark and kerneltracingprovider; no matches found.
• No additional CI commands were run locally because the failure is in the cross-build/submodule dependency setup.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 11d615c

History

• 💔 Build #42789 failed 51c7e3c

cc @belimawr