App composition: pods admitted before the PodIdentityAssociation propagates never get AWS credentials

[Smana]

What

The App composition renders the Deployment and the EKSPodIdentity (→ PodIdentityAssociation) concurrently. Nothing orders them. If a pod is admitted before the association has propagated to the EKS Pod Identity webhook, that pod gets no AWS credentials — and because injection happens at admission, no amount of restarting fixes it. The pod crashloops forever against a correctly-configured IAM role.

Hit on xplane-image-gallery on the freshly rebuilt cluster (2026-08-02): 36 restarts, CrashLoopBackOff, dying on

{"level":"fatal","error":"Access Denied.","message":"Failed to connect to storage"}

Why it is easy to misdiagnose

Every piece of the IAM story looks healthy, so the investigation goes to AWS and finds nothing:

• EKSPodIdentity/xplane-image-gallery-s3-pod-identity — Synced=True Ready=True
• PodIdentityAssociation — Ready=True, bound to exactly ns=apps sa=xplane-image-gallery
• IAM policy — correctly scoped to the right bucket:

  s3:ListBucket           arn:aws:s3:::eu-west-3-ogenki-xplane-image-gallery
  s3:GetObject/PutObject/DeleteObject   .../*
  

• Bucket MR — Ready=True, external name matches STORAGE_BUCKET exactly
• Region matches, endpoint matches

The tell is in the pod spec, not in AWS:

$ kubectl get pod -n apps <pod> -o json | jq '.spec.containers[0].env[] | select(.name|test("AWS_CONTAINER"))'
# (empty — nothing injected)
$ kubectl get pod -n apps <pod> -o json | jq '.spec.volumes[] | select(.name=="eks-pod-identity-token")'
# (empty)

A working pod-identity workload (runlore-0) has all three:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
volume: eks-pod-identity-token

The timing

Event	Timestamp
PodIdentityAssociation object created	07:35:51Z
Pod …-xbmpv admitted	07:35:49Z — 2s before the PIA object existed
PIA Ready=True	07:35:59Z
Pod …-67lg9 admitted	07:36:00Z — 1s after Ready, still inside webhook propagation lag

Both replicas lost the race. Neither ever recovered, across 36 restarts and ~3 hours.

Ruled out

automountServiceAccountToken: false (set by the claim, and a reasonable suspect since the token is a projected SA volume) is not the cause. Controlled admission test with the same ServiceAccount in the same namespace:

Pod	automountServiceAccountToken	Injected?
epi-test-automount-false	false	✅ env + volume
epi-test-automount-default	unset	✅ env + volume

The webhook injects regardless. The variable that mattered was when the pod was admitted.

Workaround

Delete the pods once the association is live — they are re-admitted and the webhook injects correctly:

kubectl delete pod -n apps -l app.kubernetes.io/instance=xplane-image-gallery

Confirmed: both replicas came back 1/1 Running, the AWS env vars are present, the app logs Storage client initialized instead of the fatal, and the XR is Synced=True Ready=True.

Proposed fix

Latch Deployment rendering on PodIdentityAssociation readiness for any App that declares an EPI — the same option("params").ocds pattern the composition already uses to gate AIGatewayRoute rendering on Deployment readiness (SPEC-002 FR-002). No Deployment until the association reports Ready=True, so pods can never be admitted into the gap.

Worth considering alongside it: a guard so this fails loudly rather than silently. An App that declares an EPI but whose pods have no AWS_CONTAINER_CREDENTIALS_FULL_URI is always a bug — right now the only signal is an application-level Access Denied that points at IAM.

Impact

Any App claim combining a workload with an EKSPodIdentity, on first creation — so it reproduces on every cluster rebuild, and is invisible on steady-state clusters where the association long predates any pod. xplane-image-gallery is the instance that surfaced it; xplane-harbor's S3 path uses a static IAM user and is unaffected, and app-wizard does not need AWS credentials at runtime.

[Smana]

Reopening: the composition fix (#1703) was reverted in #1718. The race is real but cannot be solved where I put it.

Why the gate was withdrawn

1. PodIdentityAssociation.Ready does not mean the association is usable.

The race lives in the pod-identity webhook's cache — aws/amazon-eks-pod-identity-webhook#264 — and there is no API to ask whether that cache is warm: aws/containers-roadmap#2507 is open requesting exactly that.

The original incident data already showed the gate missing half the problem:

Replica	Admitted	Gate would have helped?
…-xbmpv	2s before the association existed	✅
…-67lg9	1s after Ready=True	❌

A gate that fires on Ready cannot catch the second case, and the second case is the one with no upstream signal to wait on.

2. Suppressing a resource from desired state also suppresses it from crossplane render.

Every App Wizard preview of an s3Bucket app rendered without a workload — the wizard told developers their claim creates no Deployment. #1713 tried to special-case render by detecting a missing metadata.uid; that was wrong (crossplane render populates one), and I only caught it because I finally tested through the wizard instead of a hand-built fixture.

This is not fixable by choosing a better field. Any ordering mechanism has it — function-sequencer, the upstream option, omits from desired state the same way and tells you to pass --observed-resources when rendering, which the wizard does not do.

What to do instead — the runbook

Upstream's remedy, and what actually recovered image-gallery: recreate the pods once the association exists.

# Symptom: crashloop with an application-level "Access Denied" while the EPI,
# the PodIdentityAssociation, the IAM policy and the bucket all report healthy.
# The tell is in the POD SPEC, not in AWS:
kubectl get pod -n <ns> <pod> -o json | jq '.spec.containers[0].env[]|select(.name|test("AWS_CONTAINER"))'
#   empty  => the webhook injected nothing; the pod predates a usable association

kubectl delete pod -n <ns> -l app.kubernetes.io/instance=<app>

This is the same fix AWS documents for retrofitting Pod Identity onto Load Balancer Controller, Cluster Autoscaler and Karpenter.

Remaining work

• KB entry so the next person does not re-derive the diagnosis — the failure presents as an IAM problem with every AWS-side signal green, which is what made it cost ~3h
• Consider whether the App composition should surface it: an App that declares an EPI whose pods have no AWS_CONTAINER_CREDENTIALS_FULL_URI is always a bug, and today nothing says so

Kept from this work

The duplicate-composed-resource fix (#1715) is unrelated and stays — the s3Bucket set was emitting two Buckets and two EPIs whenever versioning was off, which broke renders outright. Also #1720, a hotfix for an indentation regression the revert introduced.

[Smana]

KB entry written and merged: eks-pod-identity-credentials-never-injected.md (Smana/runlore-kb#108).

Written as a Playbook rather than an Incident — it applies to any EPI-backed workload, not just image-gallery, and reproduces on every cluster rebuild.

It records the three things that made this expensive, so the next person does not re-derive them:

1. The tell is in the pod spec, not in AWS. No AWS_CONTAINER_* env and no eks-pod-identity-token volume. Everything on the AWS side reports healthy, which is precisely why the investigation goes the wrong way.
2. Ordering cannot fix it. Ready=True ≠ the webhook can serve the association, and there is no API to ask (containers-roadmap#2507). The gate attempted here (fix(app): gate the workload on EKS Pod Identity readiness #1703) and reverted (revert(app): drop the EKS Pod Identity admission gate #1718) is documented with its reasons so it is not re-attempted.
3. automountServiceAccountToken: false is ruled out by a controlled admission test — recorded so nobody repeats that experiment.

Remaining on this issue is the second checkbox only: whether the composition should surface the condition (an App declaring an EPI whose pods have no AWS_CONTAINER_CREDENTIALS_FULL_URI is always a bug, and today nothing says so). That is a detection question, not an ordering one, so it does not run into the problem above.