Service A Intermittently Cannot Reach Service B

One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.

advanced15 min5 roundsKubernetesObservability

Answer key reviewed by Branko Petric

Read first: Kubernetes Networking: Services, Ingress, and DNS

Your decision

Service A intermittently fails to reach Service B inside the cluster - roughly one call in twenty times out, the rest are fine. A teammate says just add retries and move on. How do you actually diagnose this?

The situation

Service A calls Service B many times a second over the cluster network. Most calls succeed, but roughly one in twenty times out. There is no obvious pattern in the application logs.

Context

Both services run as Deployments with several replicas behind ClusterIP Services. The cluster uses CoreDNS and has NetworkPolicies in place. A teammate, seeing that the failure rate is low, wants to wrap the call in a retry and consider it handled.

More: Kubernetes scenario interview questions