Beat receiver feature flags are process-global in standalone OTel configs

[belimawr]

Scope

This issue affects only standalone / hand-crafted OTel receiver configurations where multiple Beat receivers are defined in the same Elastic Agent process and those receiver definitions set conflicting features.* values.

Fleet-managed policies and Agent-generated receiver configs should normally provide one consistent agent.features.* set for the process. The problem described here is about raw standalone OTel configs such as manually authored filebeatreceiver definitions.

Problem

Beat receiver feature flags are parsed from each receiver's raw config, but the parsed values are stored in package-global libbeat/features state. If two Beat receiver definitions in the same Elastic Agent process configure different values for the same feature flag, each receiver overwrites the same global flag.

This creates a "data race" at the configuration semantics level: the final behavior depends on receiver construction, startup, or reload ordering. It is not necessarily a Go race-detector issue because the current values are stored with atomics. The bug is that receiver-local configuration mutates process-global behavior.

Example standalone OTel config:

receivers:
  filebeatreceiver/log_as_log:
    filebeat:
      inputs:
        - type: log
          id: logs-a
          paths: [/var/log/a.log]
    features:
      log_input_run_as_filestream:
        enabled: false

  filebeatreceiver/log_as_filestream:
    filebeat:
      inputs:
        - type: log
          id: logs-b
          paths: [/var/log/b.log]
    features:
      log_input_run_as_filestream:
        enabled: true

In this configuration, each receiver appears to define an independent feature flag value. In practice, both receivers share the same process-global features.log_input_run_as_filestream.enabled value.

Current behavior

Each Beat receiver creates a Beat instance through NewBeatForReceiver. During initialization, the receiver's raw config is passed to features.UpdateFromConfig:

https://github.com/elastic/beats/blob/af686c255e1f319b47524835f63b06d31365d47b/x-pack/libbeat/cmd/instance/beat.go#L152-L173

b.RawConfig = cfg

// ...

if err := features.UpdateFromConfig(b.RawConfig); err != nil {
	return nil, fmt.Errorf("could not parse features: %w", err)
}
b.RegisterHostname(features.FQDN())

features.UpdateFromConfig parses features.fqdn, features.log_input_run_as_filestream, and features.aws_s3_v2, then stores their values in package-global state:

https://github.com/elastic/beats/blob/af686c255e1f319b47524835f63b06d31365d47b/libbeat/features/features.go#L34-L67

type fflags struct {
	callbackMut sync.RWMutex

	fqdnEnabled   atomic.Bool
	fqdnCallbacks map[string]boolValueOnChangeCallback

	logRunAsFilestream atomic.Bool
	awsS3V2            atomic.Bool
}

// ...

flags.SetFQDNEnabled(parsedFlags.Features.FQDN.Enabled())
flags.SetLogInputRunFilestream(parsedFlags.Features.LogRunAsFilestream.Enabled())
flags.SetAwsS3V2(parsedFlags.Features.AwsS3V2.Enabled())

Those global values are later read by Beat code paths such as the Filebeat log input redirect:

https://github.com/elastic/beats/blob/af686c255e1f319b47524835f63b06d31365d47b/filebeat/input/logv2/input.go#L58-L63

// Global feature flag that forces all Log input instances
// to run as Filestream, even when not running under
// Elastic Agent
if features.LogInputRunFilestream() {
	return true, nil
}

and the aws-s3 input implementation switch:

https://github.com/elastic/beats/blob/af686c255e1f319b47524835f63b06d31365d47b/x-pack/filebeat/input/awss3/input.go#L43-L51

func (im *s3InputManager) Create(cfg *conf.C) (v2.Input, error) {
	config := defaultConfig()
	if err := cfg.Unpack(&config); err != nil {
		return nil, err
	}

	if features.AwsS3V2() {
		return newInputV2(config, im.store, im.path, im.logger)
	}

Why this is unsafe

The receiver config gives the impression that features.* is scoped to the receiver instance. The implementation scopes it to the process.

As a result:

• The receiver initialized last can overwrite feature values used by previously initialized receivers.
• Reloads can change behavior for unrelated receivers because every reload can re-apply a receiver's local config into shared global state.

Expected behavior

Receiver-local configuration should not mutate behavior for other receiver instances in the same Elastic Agent process.

Either:

• Beat receiver feature flags should be represented as receiver-local / Beat-instance-local state and passed to the code paths that need them, or
• standalone OTel validation should reject multiple Beat receiver definitions with conflicting process-global feature flag values, making the unsupported configuration explicit.

Acceptance criteria

• Document or fix the current process-global behavior for Beat receiver features.*.
• Add test coverage with two filebeatreceiver definitions in one collector process using conflicting features.log_input_run_as_filestream.enabled values.
• Ensure one receiver's feature flag config cannot silently change another receiver's input behavior.
• If receiver-local feature flags are not feasible short term, reject conflicting feature flag values during standalone OTel config validation with a clear error.

Related issues

• Pass all agent.features.* flags to Beat receivers #15437

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[github-actions]

The safest short-term fix is to reject conflicting process-global Beat feature flags during standalone OTel config resolution; receiver-local isolation requires a coordinated Beats change and is not implementable solely in Elastic Agent.

Recommendation

Add a config-resolver validation step that examines the fully resolved receivers map before collector construction. For every Beat receiver, collect the effective values of process-global features.* flags and return a clear error when two receivers specify different values. Apply this to both normal standalone startup and otel validate through the shared NewSettings path. Keep the receiver-local redesign as a follow-up in Beats.

Findings

• beats/x-pack/filebeat/fbreceiver/factory.go:createReceiver passes each receiver config to xpInstance.NewBeatForReceiver.
• beats/x-pack/libbeat/cmd/instance/beat.go:152-173 assigns b.RawConfig and calls features.UpdateFromConfig for every receiver.
• beats/libbeat/features/features.go:34-67 stores fqdn, log_input_run_as_filestream, and aws_s3_v2 in package-global atomic state; beats/filebeat/input/logv2/input.go:58-63 reads that global state. Thus the last receiver created or reloaded controls behavior for all relevant inputs.
• Standalone OTel startup and validation share the collector settings path: internal/edot/otelcol/run.go:73-112 builds NewSettings, while internal/edot/otelcol/validate.go:15-21 creates the collector and runs DryRun. The current settings contain no resolver converter or conflict check.
• Existing standalone coverage is single-receiver only: testing/integration/ess/testdata/filebeat_receiver_log_as_filestream.yml:1-19.
• The generated Agent-managed path is separate (internal/pkg/otel/manager/manager.go:436-506) and normally emits one consistent process-level feature set; do not broaden this issue into the unrelated propagation gap in Pass all agent.features.* flags to Beat receivers #15437.
• The translator also hard-codes generated receiver features.fqdn.enabled at internal/pkg/otel/translate/otelconfig.go:449-454, but that is not the raw standalone configuration path described here.

Verification

I ran the focused OTel tests from internal/edot:

$ go test ./otelcol
FAIL: github.com/elastic/beats/v7 ... replaced by ../../beats: reading ../../beats/go.mod: no such file or directory

The checkout lacks the initialized beats submodule, so the package test could not compile. git status --short remained clean.

Detailed Action Plan

1. Add a pure validator in internal/edot/otelcol that receives the resolved confmap.Conf, locates receivers, and considers Beat receiver IDs (at minimum filebeatreceiver/*; cover other Beat receiver types registered by this distribution as applicable).
2. Normalize both supported configuration spellings (features: { ... } and collector-compatible dotted keys such as features.log_input_run_as_filestream.enabled) and compare every process-global Beat flag present across receivers. Preserve absent-vs-default semantics so a receiver that omits a flag is not incorrectly treated as an explicit conflicting value unless Beats applies that same default.
3. Return an error naming the feature and both receiver IDs/values, for example: conflicting global Beat feature "log_input_run_as_filestream.enabled" configured by receivers ....
4. Register the validator as a resolver converter in the shared settings construction (internal/edot/otelcol/run.go:73-112) so Run and Validate fail before any Beat receiver construction. Avoid putting the check only in validate.go, which would leave normal startup vulnerable.
5. Add focused validator tests for two receivers with conflicting values, matching values, omitted values, both dotted and nested forms, and malformed feature values. Extend internal/edot/cmd/validate_test.go with a standalone two-receiver conflict fixture and assert the error identifies the receivers and feature.
6. Add a runtime/integration fixture based on testing/integration/ess/testdata/filebeat_receiver_log_as_filestream.yml only if the existing standalone integration harness can exercise validation without external services. Document that process-global Beat feature flags must be consistent across receivers until receiver-local support lands upstream.

Related Items

Type	Reference	Relevance
Issue	#15437	Separate issue about propagating all agent.features.* flags into generated Beat receiver configs
Upstream issue	elastic/beats#49877	Tracks the lack of instance-local Beat feature state
File	internal/edot/otelcol/run.go:73-112	Shared standalone collector settings path
File	internal/edot/otelcol/validate.go:15-21	Validation currently only performs collector DryRun
File	testing/integration/ess/testdata/filebeat_receiver_log_as_filestream.yml:1-19	Existing single-receiver feature coverage

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.