fix(prow-job-executor): retry Key Vault prow-token lookup on transient failures (AROSLSRE-1228)

[raelga]

Problem

The prowjobexecutor execute command used in EV2 gating looks up the Prow token in Key Vault before it ever talks to Prow, and that lookup had no retry logic. When the managed-identity token acquisition hits a transient failure — most commonly an EOF / connection reset reading the instance metadata endpoint (169.254.169.254) on the EV2 runner — the whole EV2 gating step fails immediately and the entire deployment job must be restarted from scratch.

This is a separate failure point from #251 / AROSLSRE-1168, which added retries to the Gangway SubmitJob() 429 path. This failure happens earlier, during Key Vault credential acquisition.

Observed in PROD (branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel, eastus2euap):

failed to lookup prow token in Key Vault: failed to get secret "prow-token" from
Key Vault "https://arohcpprod-global.vault.azure.net": DefaultAzureCredential:
failed to acquire a token. ... ManagedIdentityCredential: Get
"http://169.254.169.254/metadata/identity/oauth2/token...": EOF

az login --identity succeeded seconds earlier in the same step, so this is a momentary IMDS/networking blip, not a misconfiguration.

What changes

• Wrap lookupProwTokenInKeyVault in an exponential-backoff retry (retryProwTokenLookup), mirroring the existing SubmitJob pattern in prowjob/client.go.
• isRetryableKeyVaultError retries transient failures — credential/IMDS and network errors, plus Key Vault HTTP 429/5xx — and fails fast on permanent Key Vault 4xx (401/403/404/400/409).
• The single-shot lookup is factored into lookupProwTokenInKeyVaultOnce; the fetch is injectable so retry behavior is unit-tested without a live Key Vault.
• Retries are logged and bounded by the parent context. Backoff: 30s, 1, 2, 4, 8, 16 min over up to 6 attempts (~31 min worst-case).

Testing

• go test -race ./... — pass (new prowtoken_test.go: classifier 429/5xx/4xx/IMDS-EOF, retry-then-succeed, fail-fast on 403/404, exhaust on persistent transient).
• go tool golangci-lint run ./... — 0 issues.
• go vet ./... — clean. go mod tidy + go work sync clean (promotes azcore to a direct require).

Follow-ups

• After merge + tag, bump the dependency in Azure/ARO-HCP (tooling/templatize, test) so the EV2-shipped prowjobexecutor binary picks up the retry.

Fixes AROSLSRE-1228.

[Copilot]

Pull request overview

This PR improves the robustness of prowjobexecutor execute in EV2 gating by adding exponential-backoff retries around the Key Vault prow-token lookup, addressing transient managed-identity/IMDS and Key Vault service failures before the tool contacts Prow.

Changes:

• Add a retry wrapper (retryProwTokenLookup) around the Key Vault secret fetch, with a retryability classifier (isRetryableKeyVaultError).
• Factor the single-attempt Key Vault lookup into lookupProwTokenInKeyVaultOnce to enable unit testing of retry behavior.
• Add unit tests for retry classification and retry loop behavior, and promote azcore to a direct module dependency.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
tools/prow-job-executor/prowtoken.go	Adds retry/backoff logic and retryability classification for Key Vault prow-token lookup.
tools/prow-job-executor/prowtoken_test.go	Adds unit tests covering retry classification and retry/exhaust/fail-fast scenarios.
tools/prow-job-executor/go.mod	Promotes github.com/Azure/azure-sdk-for-go/sdk/azcore to a direct requirement.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

tools/prow-job-executor/prowtoken.go:175

• cmdutils.GetAzureTokenCredentials() failures are local/deterministic (e.g., Azure CLI not available, invalid credential chain options) and are unlikely to be fixed by waiting. Right now they’ll be treated as retryable (non-HTTP, non-sentinel) and can burn the full ~31.5 minute backoff before failing. Consider marking this error as permanent (wrap with errPermanentKeyVaultLookup) so misconfiguration fails fast while still retrying genuine IMDS/token-acquisition blips during GetSecret().

	// Get Azure credentials using ARO-Tools
	cred, err := cmdutils.GetAzureTokenCredentials()
	if err != nil {
		return "", fmt.Errorf("failed to get Azure credentials: %w", err)
	}