[9.5] (backport #15422) Add stack provisioning to mage integration:local

[mergify]

What does this PR do?

The mage integration:local target can now run tests that need a real Elastic Stack. It provisions or reuses it automatically, the same way mage integration:single does but runs the test directly on your machine instead of provisioning a VM.

Why is it important?

Allow running integration tests on your own machine.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None.

How to test this PR locally

Make sure your test has Stack: &define.Stack{} and Local: true:

define.Require(t, define.Requirements{
    Group: integration.Default,
    Stack: &define.Stack{},
    Local: true,
...
})

Then run:

TEST_PACKAGES=tar.gz mage -v integration:local <your-test-name>

You should see it provision (or reuse) a stack and then run the test directly on your machine, with no VM created:

>>> (linux-amd64-ubuntu-2404-default) Waiting for stack to be ready...
>>> Creating cloud stack 9.5.0-SNAPSHOT [stack_id: 950-SNAPSHOT]
>>> Created cloud stack 9.5.0-SNAPSHOT [stack_id: 950-SNAPSHOT, deployment_id: 1234...]
>>> Waiting for cloud stack 9.5.0-SNAPSHOT to be ready [stack_id: 950-SNAPSHOT, deployment_id: 1234...]
>>> (linux-amd64-ubuntu-2404-default) Using Stack with Kibana host https://<redacted>.us-west2.gcp.elastic-cloud.com, credentials available under .integration-cache
...
--- PASS: <your-test-name> (22.66s)
PASS
ok  	github.com/elastic/elastic-agent/testing/integration/ess	22.763s

---

This is an automatic backport of pull request #15422 done by [Mergify](https://mergify.com).

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[infra-vault-gh-plugin-prod]

💔 Build Failed

• Buildkite Build
• Commit: 62237d4

Failed CI Steps

• Trigger Serverless Beats Tests

History

cc @samuelvl

[github-actions]

TL;DR

The Serverless Beats Tests failure is a deterministic regression in this PR, not a transient Buildkite issue. VM provisioning now occurs before the SSH key is generated, so GCE provisioning cannot read .integration-cache/id_rsa.pub.

Remediation

• Restore SSH-key preparation before r.ip.Provision for VM provisioners, or otherwise generate the key before calling the GCE provisioner. Keep local provisioning on the no-SSH path.
• Re-run the Serverless Beats Tests job after that ordering fix.

Investigation details

Root Cause

pkg/testing/runner/runner.go:140-167 calls ensureCacheDir and then provisions instances. The PR moved prepare (which calls createSSHKey) into the later VM-only branch at pkg/testing/runner/runner.go:179-190. The GCE provisioner requires that key during provisioning at pkg/testing/gcloud/provisioner.go:82-85, where it reads filepath.Join(cfg.StateDir, "id_rsa.pub"). ensureCacheDir only creates the directory; it does not create the key.

The PR’s integration:local path correctly selects the local provisioner, but the failing Serverless Beats job uses the GCE VM provisioner, so it hits this ordering regression.

Evidence

• Build: https://buildkite.com/elastic/elastic-agent/builds/42756
• Job/step: Serverless Beats Tests (beats-agent-serverless-tests, build 5202)
• Key log excerpt: Error: error running test: failed to read SSH public key at .integration-cache/id_rsa.pub: open .integration-cache/id_rsa.pub: no such file or directory

Verification

• No code changes or tests were run; this workflow is diagnosing the failed build.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.