Service A Intermittently Cannot Reach Service B
One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.
Answer key reviewed by Branko Petric
Read first: Kubernetes Networking: Services, Ingress, and DNS
Your decision
Service A intermittently fails to reach Service B inside the cluster - roughly one call in twenty times out, the rest are fine. A teammate says just add retries and move on. How do you actually diagnose this?
The situation
Service A calls Service B many times a second over the cluster network. Most calls succeed, but roughly one in twenty times out. There is no obvious pattern in the application logs.
Context
Both services run as Deployments with several replicas behind ClusterIP Services. The cluster uses CoreDNS and has NetworkPolicies in place. A teammate, seeing that the failure rate is low, wants to wrap the call in a retry and consider it handled.
Related challenges
- Pods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
- A Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A spike took the service down even though autoscaling was on. A teammate wants to crank the max replicas. Defend the real fix.
- Under Load, Healthy Pods Keep Restarting. Add Replicas?
During traffic peaks, Kubernetes keeps killing pods that are not actually broken. A teammate wants to scale up replicas. Defend the real fix.
- p99 Spikes With CPU at 40 Percent of Limit. Add Replicas?
A latency-sensitive service shows p99 spikes while average CPU sits at 40 percent of its limit. A teammate blames the database. Defend the real cause.