Single-replica deployments that own a ReadWriteOnce volume use the default RollingUpdate strategy. With one replica, maxSurge: 25% rounds up to 1, so Kubernetes creates the replacement pod before the old one releases the volume - and the replacement cannot start until it does. Neither side can move.
Seen twice during the node pool migration (#146):
auth-proxy - the new pod sat in Multi-Attach error while the old node held the volume. About three minutes of HTTP 503 on the front door, because auth-proxy is the only way in.grafana - deadlocked outright and did not recover on its own. Both ReplicaSets held one pod in ContainerCreating until the older ReplicaSet was scaled to zero by hand.
admin already carries MaxSurge = 0 / MaxUnavailable = 1 for exactly this reason, with the rationale written next to it. The others never got the same treatment: grafana, loki, transcription, source-code-nfs.
This is not specific to the migration - any node change reschedules the pod and reproduces it, so it fires on ordinary deploys too. It is simply guaranteed when the pod has to change node.
Fix
Give every single-replica deployment holding an RWO volume MaxSurge = 0 / MaxUnavailable = 1 (or Recreate). The old pod then terminates and releases the volume before the replacement is scheduled, trading a short planned gap for a deadlock.
Separately
auth-proxy being a single replica makes it a front-door single point of failure. Running two would need the data-protection keys somewhere shared rather than on an RWO volume - worth its own change.
Single-replica deployments that own a ReadWriteOnce volume use the default
RollingUpdatestrategy. With one replica,maxSurge: 25%rounds up to 1, so Kubernetes creates the replacement pod before the old one releases the volume - and the replacement cannot start until it does. Neither side can move.Seen twice during the node pool migration (#146):
auth-proxy- the new pod sat inMulti-Attach errorwhile the old node held the volume. About three minutes of HTTP 503 on the front door, because auth-proxy is the only way in.grafana- deadlocked outright and did not recover on its own. Both ReplicaSets held one pod inContainerCreatinguntil the older ReplicaSet was scaled to zero by hand.adminalready carriesMaxSurge = 0/MaxUnavailable = 1for exactly this reason, with the rationale written next to it. The others never got the same treatment:grafana,loki,transcription,source-code-nfs.This is not specific to the migration - any node change reschedules the pod and reproduces it, so it fires on ordinary deploys too. It is simply guaranteed when the pod has to change node.
Fix
Give every single-replica deployment holding an RWO volume
MaxSurge = 0/MaxUnavailable = 1(orRecreate). The old pod then terminates and releases the volume before the replacement is scheduled, trading a short planned gap for a deadlock.Separately
auth-proxybeing a single replica makes it a front-door single point of failure. Running two would need the data-protection keys somewhere shared rather than on an RWO volume - worth its own change.