Every Rolling Deploy Causes 502s. Add Ingress Retries?
Each rolling deploy causes a burst of 502s at the load balancer. A teammate wants ingress retries and night-only deploys. Defend the real fix.
Answer key reviewed by Branko Petric
Read first: Kubernetes Networking: Services, Ingress, and DNS
Your decision
Every rolling deploy of your API causes a short burst of 502s at the load balancer, then it settles. A teammate wants to turn on retries at the ingress and move deploys to the night window. Do you agree, and what do you do instead?
The situation
Your orders API runs as a Deployment of 6 replicas on EKS, exposed through an AWS Application Load Balancer created by the AWS Load Balancer Controller in IP target mode, so the ALB sends traffic straight to pod IPs. Every rolling deploy produces a burst of 502 responses in the ALB access logs, a few seconds each time an old pod is replaced, and then traffic is clean again until the next deploy. Customer support has started linking failed checkouts to deploy times.
Context
The Deployment uses the default rolling update strategy (maxUnavailable 25 percent, maxSurge 25 percent). There is no preStop hook, terminationGracePeriodSeconds is left at the default of 30, and there is no readiness probe. The app is a web server that exits as soon as it receives SIGTERM. The ALB target group keeps its default deregistration delay. In the ALB logs, the failing requests all point at IPs of pods that kubectl get pods shows as Terminating at that moment. Most of the API's write endpoints are POSTs that create orders. A teammate proposes enabling retries at the ingress layer and moving all deploys to the 02:00 window.
Related challenges
- An EKS Pod Needs AWS Access. Attach It to the Node Role?
A pod needs to read S3, and a teammate wants to add the permission to the EKS node role to unblock it. Defend the least-privilege approach.
- Pods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
- A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A spike took the service down even though autoscaling was on. A teammate wants to crank the max replicas. Defend the real fix.
- Service A Intermittently Cannot Reach Service B
One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.