[9.5.0] OSQuery live queries do not work in the OTel runtime

[cmacknz]

• Relates Failing test: should properly show static values in form and results - EcsMapping should properly show static values in form and results kibana#276569
• Relates Switch osquerybeat to run as a receiver by default. #15319
• Relates Revert "Run osquerybeat as a receiver by default (#15319)" #15411

Our integration test for osquery does not include live queries, only scheduled queries. Scheduled queries flow via configuration while live queries listen for actions using the no longer wired up control protocol RegisterAction method.

https://github.com/elastic/beats/blob/a82b855159ca5d52bf3f7dd950845435548fff81/x-pack/otel/otelmanager/manager.go#L66-L67

func (n *OtelManager) RegisterAction(action management.Action)   {}
func (n *OtelManager) UnregisterAction(action management.Action) {}

We need a functioning implementation of the action handlers in the otel manager in Beats for live queries to work.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Fixing this looks like we are going to need something like the diagnostics extension where we can use a unix socket+HTTP instead of the control protocol.

An OpAMP custom action would also work but not sure we are able to make that change over in time.

[cmacknz]

Reverting osquerybeat to use the process runtime by default until we can fix this #15411.

When we fix this, we'll need to revert the revert.

[swiatekm]

The real question is how much work is it to add a beats actions extension in agent and appropriate wiring in beats receivers. Opamp custom actions are definitely the right move architecturally, but I'm reluctant to add another big change to 9.5.0 so late in the cycle. The extension would be a more narrowly scoped addition that wouldn't affect much else, as opposed to redoing health checks in the otel manager completely.

[cmacknz]

Yeah I think adding a temporary actions extension and then removing it later once Opamp is wired up is likely the right move given the timnig.

[cmacknz]

We could augment the diagnostics extension to handle this as well to avoid another unix socket, if we believe it is temporary.

[swiatekm]

It'd be hack, but that would also give us more of an incentive to fix it in 9.6.0. Reasons to adopt opamp internally are piling up anyway. Reusing the diagnostics extension would be nice in that all the wiring is already set up.

[cmacknz]

Yeah, let's not introduce a new extension that fundamentally just adds a new HTTP API for actions and reuse the diagnostics extension. Then we can replace it with opamp once that's ready.