Kubernetes Observability: Logs, Metrics, and Monitoring

You cannot fix what you cannot see, and Kubernetes goes out of its way to make things invisible: pods are disposable, their IPs change, and the logs of a crashed container vanish with it unless you catch them in time. Observability is the discipline of getting enough signal out of a moving system to answer two questions - is it healthy, and if not, why? This guide covers the two pillars you will live in day to day (logs and metrics), the tooling the industry has standardized on, and the exact commands to reach for when something is on fire.

The three pillars, briefly

Observability is usually split into three kinds of signal, and it helps to know where each fits before diving in:

• Logs - discrete, timestamped records of events ("request failed", "config loaded", "connection refused"). They tell you what happened and are your first stop for a specific error.
• Metrics - numeric measurements sampled over time (CPU usage, request rate, error count, queue depth). They are cheap to store and aggregate, so they tell you how much and how the trend is moving - the right tool for dashboards and alerts.
• Traces - the path of a single request as it hops across services, with timing at each hop. They tell you where the time went in a distributed call, which is invaluable in a microservice mesh but heavier to set up (OpenTelemetry is the standard instrumentation layer, with Jaeger or Tempo as the backend).

Traces matter most once you have many services calling each other. For most Kubernetes operations work, the day-to-day is logs and metrics, so that is where this guide spends its time. Get those two solid first.

Logs: kubectl logs and its flags

The most direct way to see what a container is saying is kubectl logs. Kubernetes captures whatever a container writes to stdout and stderr - which is exactly why the twelve-factor rule "log to stdout, not to a file" matters: it is what makes your logs reachable this way at all.

kubectl logs web                       # a container's stdout/stderr
kubectl logs web -f                     # follow (stream live, like tail -f)
kubectl logs web --tail=100             # just the last 100 lines
kubectl logs web -f --tail=100          # follow, starting from the last 100
kubectl logs web --previous             # the PREVIOUS, crashed container
kubectl logs web -c sidecar             # a specific container in a multi-container pod
kubectl logs web --since=15m            # only the last 15 minutes

Two flags earn their place in every incident. --previous (or -p) shows the logs of the container instance that ran before the current one - essential when a pod is in CrashLoopBackOff, because by the time you look, the failing container has already been replaced and its dying words are gone from the live stream. And -l selects by label instead of by name, so you can read across every pod behind a Deployment at once instead of guessing pod names:

kubectl logs -l app=web --tail=50       # logs from ALL pods labelled app=web
kubectl logs -l app=web --prefix        # prefix each line with its pod name

That label form is the closest kubectl gets to "show me what this whole service is doing." It is a lifesaver when three replicas are up and only one is misbehaving.

Why pod logs are ephemeral

Here is the trap. kubectl logs does not read from some durable log store - it reads the container runtime's log files on the node where the pod is running. Those files live and die with the container:

• When a pod is deleted or rescheduled to another node, its logs are gone.
• When a container crashes and restarts, you get one previous instance back via --previous, and that is it - the one before that is lost.
• Node log rotation truncates old lines even for a running container.
• If the node itself dies, every log on it goes with it.

So kubectl logs is perfect for "what is this pod doing right now" and useless for "what did the pod that failed at 3am on Tuesday say." In a system where pods are cattle and get replaced constantly, relying on it as your only log access means you are always one restart away from losing the evidence. You need to ship logs off the node into something durable.

The node-level log aggregation pattern

The standard solution is a DaemonSet - one collector pod on every node - that tails the container log files the node already writes, enriches each line with Kubernetes metadata (pod name, namespace, labels), and ships it to a central store you can search later. Because it runs the collection at the node level rather than inside each app, your applications do nothing special: they keep logging to stdout, and the collector picks it up.

This is a natural fit for a DaemonSet precisely because it guarantees one instance per node, so no node's logs are missed, and new nodes get a collector automatically.

The two common stacks:

• Loki + Promtail (or the newer Grafana Alloy) - Grafana Loki indexes only labels, not full text, which makes it cheap to run and pairs naturally with Grafana for viewing. Promtail (or Alloy) is the DaemonSet doing the shipping.
• ELK / EFK - Elasticsearch for storage and full-text search, Kibana for the UI, and Fluentd or Fluent Bit as the DaemonSet collector (the "F" in EFK). More powerful search, heavier to operate.

The shape of the collector is always the same: a DaemonSet that mounts the node's log directory read-only and forwards from there.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: logging
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
        - name: collector
          image: grafana/promtail:latest
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log        # the node's log directory

Once this is in place, "what did that crashed pod say last night" becomes a query in Grafana or Kibana instead of a lost cause. That is the whole point: decouple the durability of your logs from the lifespan of your pods.

Metrics: metrics-server, kubectl top, and the HPA

Logs tell you about individual events; metrics tell you about magnitude and trend. The lightest piece of the metrics story is metrics-server, a small cluster add-on that scrapes basic CPU and memory usage from every kubelet and keeps it in memory. It is not installed by default on a from-scratch cluster (most managed clusters include it), and it powers two things:

kubectl top nodes               # CPU / memory usage per node
kubectl top pods                # per-pod usage in the current namespace
kubectl top pods -A             # across all namespaces
kubectl top pods --containers   # break it down per container

kubectl top is your quick "what is hot right now" read - which pod is eating memory, which node is saturated. The same metrics-server data also feeds the HorizontalPodAutoscaler, which reads current CPU or memory against the pods' requests and scales replica count up or down to keep utilization near a target. In other words, metrics-server is the sensor the autoscaler steers by. (For how the HPA and its cousins use this, see the autoscaling guide.)

Be clear about what metrics-server is not: it holds only recent, in-memory samples for top and autoscaling. It has no history, no dashboards, no alerting, and no application-level metrics. For real monitoring you need a proper metrics system.

Prometheus and Grafana: the standard stack

The de facto standard for Kubernetes monitoring is Prometheus for collecting and storing metrics, and Grafana for visualizing them. Understanding the model matters more than the config.

Prometheus works by scraping: on an interval, it makes an HTTP request to a /metrics endpoint on each target and reads back the current values in a simple text format. Your application (via a client library) or a sidecar exporter exposes that endpoint; Prometheus pulls from it. This pull model is why Prometheus discovers targets dynamically - it queries the Kubernetes API to find pods and services to scrape, so as pods churn, the scrape list follows them automatically, no manual registration.

In a typical setup you install the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, and the exporters for cluster-level metrics. Once running, you tell Prometheus what to scrape declaratively with a ServiceMonitor (a custom resource): it selects Services by label and describes which port and path to scrape, and the Prometheus Operator turns that into scrape config for you.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  labels:
    release: prometheus     # so the Operator's Prometheus picks it up
spec:
  selector:
    matchLabels:
      app: web               # scrape Services labelled app=web
  endpoints:
    - port: metrics          # the named port to hit
      path: /metrics
      interval: 15s

You do not hand-edit Prometheus config; you drop in a ServiceMonitor and the Operator reconciles it - the same declarative desired-state pattern the rest of Kubernetes runs on. Grafana then queries Prometheus with PromQL and renders dashboards; the community-maintained Kubernetes dashboards give you cluster and pod views out of the box, and Alertmanager fires alerts (to Slack, PagerDuty, email) when a PromQL rule crosses a threshold.

The golden signals: what to actually measure

A monitoring stack is only as useful as the questions you point it at, and it is easy to drown in hundreds of metrics that never tell you anything. Google's SRE practice cuts through this with the four golden signals - if you monitor only these per service, you will catch most user-facing problems:

• Latency - how long requests take. Watch the tail (p95, p99), not just the average; the average hides the slow requests that actually hurt users. And separate the latency of failed requests from successful ones, or a flood of fast errors will make latency look great.
• Traffic - how much demand the service is getting: requests per second, or connections, or messages consumed. This is your baseline for "is this normal for right now?"
• Errors - the rate of failed requests: HTTP 5xxs, timeouts, wrong answers. A rising error rate is usually your earliest and clearest sign that something broke.
• Saturation - how full the system is: CPU, memory, disk, connection pool, queue depth relative to capacity. Saturation is the leading indicator - it climbs before latency and errors do, giving you warning while there is still time to act.

Latency, traffic, and errors are what your users feel; saturation is what warns you it is coming. Build dashboards and alerts around these four first, then add service-specific metrics only where you have a real question they answer. More metrics is not more insight.

Events: the cluster's own narration

Separate from application logs, Kubernetes emits Events - short records of what the control plane is doing to your objects: scheduling decisions, image pulls, probe failures, OOM kills, volume mounts. They are the cluster narrating its own actions, and they explain a huge share of "why is my pod not starting" questions.

kubectl get events                          # events in the current namespace
kubectl get events -A                        # across all namespaces
kubectl get events --sort-by=.lastTimestamp  # newest last (the default order is not chronological)
kubectl get events --field-selector involvedObject.name=web   # just this object

The trap: Events are retained for only about an hour by default and then discarded - they are as ephemeral as pod logs. So they are excellent for a live incident and useless for post-mortems unless you are also shipping them somewhere durable (many log-aggregation setups collect events too). Note also that kubectl describe pod <name> prints the recent events for that one object at the bottom of its output, which is usually the fastest way to see why a specific pod is misbehaving.

Health: probes feed the monitoring picture

The readiness and liveness probes you set on containers are not just self-healing machinery - they are also a signal source your monitoring should watch.

• livenessProbe - "is the process wedged?" On failure the kubelet restarts the container. A container stuck restarting shows up as CrashLoopBackOff and as liveness-failure Events, and a rising restart count is a metric worth alerting on.
• readinessProbe - "can this pod take traffic right now?" On failure the pod is pulled from Service load-balancing but not restarted. A pod that is Running but 0/1 ready is failing readiness - it is silently getting no traffic.

These states surface in three places at once: kubectl get pods (the READY column and restart count), Events (the probe-failure messages), and metrics (kube_pod_status_ready, kube_pod_container_status_restarts_total, exposed by kube-state-metrics into Prometheus). A good alert on "pods not ready" or "restart count climbing" catches a whole class of problems before a human notices the service degrading. Keep liveness shallow (is this process hung?) so a slow dependency does not trigger a restart storm that your monitoring then has to untangle.

Where to look first in an incident

When something breaks, the signals above form a fast triage order. The goal is to narrow from "the cluster is unhappy" to a specific pod and a specific reason in a couple of minutes.

1. Metrics / dashboards first, for scope. Is this one service or the whole cluster? Are errors up, latency up, or a node saturated? The golden-signals dashboard tells you where to point the rest of your attention. Saturation climbing across nodes is a very different incident from one service's error rate spiking.
2. kubectl get pods for state. Which pods are Pending, CrashLoopBackOff, ImagePullBackOff, or Running but not ready? The STATUS column usually names the failure class immediately.
3. kubectl describe pod <name> and events for the why. The Events at the bottom explain scheduling failures, image-pull errors, OOM kills, and probe failures in plain language - "0/3 nodes available: insufficient memory", "Back-off pulling image", "Liveness probe failed".
4. Logs for the application's side of the story. kubectl logs <name> for a live pod, kubectl logs <name> --previous for a crashed one, or a query in Loki/Kibana for anything already gone. This is where the actual exception or bad-config message lives.

That order - metrics for scope, pod status for the failure class, events for the cluster's explanation, logs for the app's - resolves the large majority of incidents quickly. This is deliberately the same STATUS -> describe -> logs spine as the debugging guide, with metrics added on the front to tell you where to aim it. The mistake under pressure is jumping straight to reading logs on a random pod before you know whether the problem is even in that service; let the metrics point you first.

The shape of it

Kubernetes hides things by design - pods are disposable, logs and events expire, IPs move - so observability is the work of getting durable, aggregated signal out of a system that keeps erasing its own tracks. Logs tell you what happened: kubectl logs (with --previous and -l) for live triage, and a DaemonSet shipping to Loki or ELK so the evidence outlives the pod. Metrics tell you how much and which way the trend is going: metrics-server for kubectl top and the HPA, and Prometheus plus Grafana as the real stack, scraping targets it discovers automatically and watched through the four golden signals. Events narrate the control plane's own actions, and probes double as health signals your alerts should watch. When it breaks, read metrics for scope, pod status for the failure class, events for the why, and logs for the app's story - in that order.