A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A spike took the service down even though autoscaling was on. A teammate wants to crank the max replicas. Defend the real fix.
Answer key reviewed by Branko Petric
Read first: Kubernetes Autoscaling: HPA, VPA, and Cluster Autoscaler
Your decision
A traffic spike took your service down even though horizontal autoscaling was enabled - it scaled too slowly to keep up. A teammate says just raise the max replicas way up. What is the real fix?
The situation
A marketing push sent a sudden burst of traffic at your service. Latency spiked and requests started failing. Horizontal autoscaling was enabled, but by the time extra pods came up the spike had already caused an outage.
Context
The service scales on CPU at 70 percent. Pods take close to a minute to pull their image and warm up before serving, and the cluster was running near capacity. A teammate's only suggestion is to raise the maximum replica count significantly.
Related challenges
- Pods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
- Under Load, Healthy Pods Keep Restarting. Add Replicas?
During traffic peaks, Kubernetes keeps killing pods that are not actually broken. A teammate wants to scale up replicas. Defend the real fix.
- Service A Intermittently Cannot Reach Service B
One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.
- p99 Spikes With CPU at 40 Percent of Limit. Add Replicas?
A latency-sensitive service shows p99 spikes while average CPU sits at 40 percent of its limit. A teammate blames the database. Defend the real cause.