[BUG]: Rolling update can create a zero-routable-worker gap after a replacement Pod becomes Ready

[tmonty12]

Describe the Bug

With two worker replicas and a rolling-update strategy of maxUnavailable: 1 and maxSurge: 0, Kubernetes readiness can advance the rollout before the Dynamo frontend has registered the replacement worker.

The first new worker Pod becomes Kubernetes Ready based on worker-local health. The rollout controller then deletes the last old Pod, but the frontend is still downloading, validating, and committing metadata for the new worker runtime namespace. During this gap, there are no routable workers and requests return 404 or 503.

This was found using a Grove PCS-backed DynamoGraphDeployment, whose default rolling-update strategy is maxUnavailable: 1 and maxSurge: 0.

For comparison, a DynamoComponentDeployment uses standard Deployment defaults of maxUnavailable: 25% and maxSurge: 25%. With two replicas, these round to maxUnavailable: 0 and maxSurge: 1, which preserves an old routable worker while the first replacement is admitted. A DCD configured with maxUnavailable: 1 and maxSurge: 0 has the same failure mode.

Steps to Reproduce

1. Deploy a DynamoGraphDeployment with Grove enabled, two worker replicas, and one served model.
2. Send continuous requests through the Dynamo frontend.
3. Apply a worker update that creates a new runtime namespace/WorkerSet, such as an update that changes the rendered worker Pod template.
4. Observe the rollout sequence:
   1. One old worker is deleted.
   2. The first new worker Pod starts and becomes Kubernetes Ready.
   3. The rollout controller deletes the remaining old worker Pod.
   4. The frontend is still fetching, validating, and registering the first new worker.
5. Observe 404 or 503 responses until frontend registration completes.

Expected Behavior

Kubernetes readiness must not allow deletion of the last routable old worker until the replacement worker is routable through the frontend. Rolling updates should preserve at least one usable worker throughout the transition.

Actual Behavior

The replacement Pod's Ready condition is driven by worker-local health rather than frontend admission. Once that first new Pod becomes ready, the controller is permitted to delete the last old Pod. The frontend is still registering the new worker, producing a temporary zero-routable-worker interval and 404/503 responses.

Environment

• Worker replicas: 2
• Rolling-update strategy: maxUnavailable: 1, maxSurge: 0
• Affected rollout: Grove PCS-backed DynamoGraphDeployment
• Dynamo version and dynamo env output: to be added

Additional Context

DYN_SELF_HOST_METADATA=true can increase the length of the gap because the frontend fetches metadata over HTTP from the replacement worker. It is not the root cause: the race exists whenever frontend registration completes after Kubernetes marks the new Pod ready.