ci: keyless Harbor robot + K8s creds via GitHub OIDC->Vault (retire admin HARBOR_* + KUBECONFIG_PROD)

[mattmattox]

Keyless CI: GitHub OIDC -> Vault (retire org-admin secrets)

Converts .github/workflows/pipeline.yml from long-lived org-admin GitHub secrets
to keyless, short-lived credentials minted at deploy time from HashiCorp Vault
(vault.support.tools) via the GitHub OIDC token. Auth mechanism only — build,
push, helm-upgrade, and rollout/verify logic are unchanged.

Removed (org-admin secrets, no longer referenced anywhere)

• secrets.HARBOR_USERNAME
• secrets.HARBOR_PASSWORD
• secrets.KUBECONFIG_PROD

How auth works now

Vault jwt mount github-actions, role gha-kubetty (audience
https://github.com/SupportTools):

• Harbor (build job, and the disabled security-scan job): hashicorp/vault-action@v3
  exchanges the OIDC token for the repo-scoped Harbor robot at
  secret/data/harbor/kubetty-ci-robot (username/password -> HARBOR_ROBOT_USER/
  HARBOR_ROBOT_TOKEN); docker/login-action now uses those env vars. Robot is scoped
  to kubetty push / cache push / dockerhub pull.
• Kubernetes (deploy-production job): vault-action logs in with exportToken: true,
  then a mint step does PUT kubernetes-onprem/creds/kubetty
  (kubernetes_namespace: arc-runners-supporttools) to obtain a 1-hour SA token for
  kubetty-ci-deployer (CRUD in kubetty-gateway-prd only). The token is guarded
  (non-empty / not null), ::add-mask::ed, and written into a kubeconfig at the same
  path the deploy steps already use ($HOME/.kube/config), pointing at the in-cluster
  apiserver (https://kubernetes.default.svc:443) with the in-cluster CA. The existing
  helm upgrade --install kubetty-gateway ... -n kubetty-gateway-prd, rollout, and
  kubectl get pods/svc steps are untouched.

Permissions

• build: added permissions: { contents: read, id-token: write }.
• security-scan (disabled, if: false): added id-token: write to its existing block;
  converted to keyless so no secrets.HARBOR_* remain anywhere.
• deploy-production: added permissions: { contents: read, id-token: write }.

runs-on change (please note)

deploy-production moves ubuntu-latest -> self-hosted-linux. The keyless K8s path
targets the in-cluster apiserver via the pod's ServiceAccount CA
(/var/run/secrets/kubernetes.io/serviceaccount/ca.crt), which only exists on the
in-cluster ARC runner (arc-runners-supporttools). A GitHub-hosted ubuntu-latest runner
cannot reach kubernetes.default.svc or read that CA. The build/security-scan Harbor
jobs stay on ubuntu-latest.

Triggers unchanged

on: is untouched (push to main + tags v*, pull_request to main,
workflow_dispatch). deploy-production keeps its
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') guard, so
this PR branch push does not deploy. The Harbor/Vault steps in build are guarded by
if: github.event_name != 'pull_request', so the PR's pull_request run does not attempt
a keyless login either.

Verification

• YAML parses (yaml.safe_load), all 7 jobs preserved.
• grep -E 'secrets\.(HARBOR|KUBECONFIG)' pipeline.yml returns nothing.
• No other secrets.* references were touched (there were none besides the three removed).

Draft — backend (Vault jwt role, Harbor robot, kubernetes-onprem role) is already
provisioned and verified; open as draft for review before merge.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d5c4773539

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Generate the deploy token in the target namespace

The deploy job mints Vault Kubernetes credentials for arc-runners-supporttools, but every subsequent Helm/kubectl operation targets kubetty-gateway-prd. HashiCorp documents kubernetes_namespace as the namespace where the credentials are generated (and the response reports that service account namespace), so with the namespace-scoped token described here this job will authenticate as a service account scoped to the runner namespace and fail to deploy into kubetty-gateway-prd; request credentials for kubetty-gateway-prd instead, or explicitly use a cluster-wide binding if that is intended. See https://developer.hashicorp.com/vault/api-docs/secret/kubernetes#generate-credentials.

Useful? React with 👍 / 👎.