Every Rolling Deploy Causes 502s. Add Ingress Retries?

Each rolling deploy causes a burst of 502s at the load balancer. A teammate wants ingress retries and night-only deploys. Defend the real fix.

intermediate12 min5 roundsKubernetesAWS

Answer key reviewed by Branko Petric

Read first: Kubernetes Networking: Services, Ingress, and DNS

Your decision

Every rolling deploy of your API causes a short burst of 502s at the load balancer, then it settles. A teammate wants to turn on retries at the ingress and move deploys to the night window. Do you agree, and what do you do instead?

The situation

Your orders API runs as a Deployment of 6 replicas on EKS, exposed through an AWS Application Load Balancer created by the AWS Load Balancer Controller in IP target mode, so the ALB sends traffic straight to pod IPs. Every rolling deploy produces a burst of 502 responses in the ALB access logs, a few seconds each time an old pod is replaced, and then traffic is clean again until the next deploy. Customer support has started linking failed checkouts to deploy times.

Context

The Deployment uses the default rolling update strategy (maxUnavailable 25 percent, maxSurge 25 percent). There is no preStop hook, terminationGracePeriodSeconds is left at the default of 30, and there is no readiness probe. The app is a web server that exits as soon as it receives SIGTERM. The ALB target group keeps its default deregistration delay. In the ALB logs, the failing requests all point at IPs of pods that kubectl get pods shows as Terminating at that moment. Most of the API's write endpoints are POSTs that create orders. A teammate proposes enabling retries at the ingress layer and moving all deploys to the 02:00 window.

More: Kubernetes scenario interview questions