Add retries to network calls in Buildkite CI scripts

[swiatekm]

What does this PR do?

Transient network failures (timeouts, connection resets, HTTP 429/5xx) in the CI scripts fail whole jobs outright because many network operations run without any retry protection. Add retries to the unprotected call sites:

• Extract the retry() helper from common.sh into retry.sh so scripts can use it without common.sh's side effects (Docker restart, package manager tweaks).
• curl: use --retry (plus -f where an HTTP error page could previously be consumed as valid output, e.g. as the kind/kubectl binary or the DRA manifest version).
• version_qualifier.sh: fetch the qualifier once with retries instead of the unretried existence-check + fetch pair; a 404 still fast-fails into the empty qualifier.
• Wrap docker pull/push/imagetools, vault reads, asdf install, and Go tool installs (bash and PowerShell) in retries.

Why is it important?

Failing a whole long workflow because a request to the artifacts API at the very end ran into a proxy timeout sucks.

Related issues

• Relates Harden the CI by ensuring all network calls are retried reasonably #15448

[github-actions]

TL;DR

The pre-commit job failed because .buildkite/scripts/retry.sh has a shebang but is not marked executable. Set the executable bit and push a new commit.

Remediation

• Run chmod +x .buildkite/scripts/retry.sh and stage the mode change with git add --chmod=+x .buildkite/scripts/retry.sh.
• Push the updated branch and rerun the pre-commit check.

Investigation details

Root Cause

The check-shebang-scripts-are-executable pre-commit hook detected the newly added retry.sh file as non-executable.

Evidence

• Workflow: https://github.com/elastic/elastic-agent/actions/runs/29102343645
• Job/step: pre-commit / Run elastic/oblt-actions/pre-commit@v1
• Key log excerpt: check that scripts with shebangs are executable...Failed followed by .buildkite/scripts/retry.sh: has a shebang but is not marked executable! and exit code 1.

Validation

• The failing workflow log was inspected; no local tests were run because this is a Git file-mode issue.

Follow-up

• Confirm the mode is 100755 in the next commit and rerun the check.

---

What is this? | From workflow: PR Actions Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[infra-vault-gh-plugin-prod]

💔 Build Failed

• Buildkite Build
• Commit: f49a295

Failed CI Steps

• Trigger Serverless Beats Tests

History

• 💔 Build #42758 failed ee00a0f

cc @swiatekm

[github-actions]

TL;DR

The Serverless Beats Tests job fails because the integration runner provisions a GCP test VM before creating the SSH key that the GCP provisioner requires. This is a pre-existing runner/pipeline ordering bug, not a transient network failure addressed by this PR, so retrying the Buildkite step will reproduce the same error on a clean workspace.

Remediation

• Create the SSH key before calling the VM provisioner, or have the provisioner create/ensure the key itself; keep the existing serverless stack setup independent of VM provisioning.
• Validate the dedicated beats_tests.sh path from a clean workspace, confirming .integration-cache/id_rsa.pub exists before GCP provisioning and that the serverless Beats test proceeds.

Investigation details

Root Cause

Runner.Run creates only the cache directory at pkg/testing/runner/runner.go:140-144, starts the serverless stack at pkg/testing/runner/runner.go:146-150, and then invokes r.ip.Provision at pkg/testing/runner/runner.go:165-168. The SSH key is created later in pkg/testing/runner/runner.go:181-186 via prepare/createSSHKey. However, the GCP provisioner immediately reads .integration-cache/id_rsa.pub in pkg/testing/gcloud/provisioner.go:82-85.

The failing pipeline runs .buildkite/scripts/steps/beats_tests.sh, which packages Filebeat and invokes only mage integration:testBeatServerless filebeat (.buildkite/scripts/steps/beats_tests.sh:66-68) in a fresh checkout. The log shows serverless stack creation begins successfully, then fails with failed to read SSH public key at .integration-cache/id_rsa.pub: open .integration-cache/id_rsa.pub: no such file or directory.

Evidence

• Build: https://buildkite.com/elastic/elastic-agent/builds/42760
• Job/step: Serverless Beats Tests / .buildkite/scripts/steps/beats_tests.sh
• Key log excerpt: Creating serverless stack 9.6.0-SNAPSHOT followed by failed to read SSH public key at .integration-cache/id_rsa.pub: open .integration-cache/id_rsa.pub: no such file or directory
• PR scope: the changes extract retry() and add network retries; none changes the runner's key-generation order.

Verification

• Inspected the supplied Buildkite log and PR diff.
• Traced the runner call order to the GCP provisioner and the dedicated Beats test script.
• Searched for an existing matching flaky-test issue; none was found.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[samuelvl]

Awesome!