Observability for DevOps: What to Learn First

Why observability comes after Terraform

You now have a real system in the cloud, and nobody is watching it. If it failed tonight, you would hear about it from a user or not at all. Until now, there was little worth measuring. Now there is traffic, a database and a server that can run out of memory.

It also comes before Kubernetes on purpose. A single app on one server has few moving parts, so a graph is easy to connect to its cause. The next steps depend on it too. You cannot run an incident well if you cannot see what is happening.

What to learn, in order

1. Metrics, logs and traces. Learn what each one is for (see the FAQ below). This tells you where to look first when something breaks.
2. What to measure. Google's SRE book names four golden signals: latency, traffic, errors and saturation (how full a resource is). Start with these, not with every number you can collect.
3. Collect metrics. Follow the Prometheus getting started guide. Add a /metrics page to your app using a client library, and run node_exporter to get CPU, memory and disk data.
4. Query them. Learn a little PromQL, the Prometheus query language. Start with rate(), because a count that only goes up tells you little until you turn it into a per-second rate.
5. Build one dashboard. Put rate, errors and latency at the top. Use percentiles for latency: p95 means 95% of requests were faster than this number. An average hides your slowest users.
6. Structured logs. Write logs as JSON, with fields such as level, path and request_id. Send them to one place, such as CloudWatch Logs or Loki, so you can search without using SSH.
7. One alert that matters. Prometheus's own advice is to alert on symptoms linked to user pain, and to have as few alerts as you can. "Error rate above 5% for 5 minutes" is a symptom. "CPU above 80%" often is not. Prometheus only decides that an alert should fire. A separate program, Alertmanager, sends the message to you.
8. One SLO. An SLO (service level objective) is a target for how reliable your app should be. It turns "is this bad?" into a question you can answer with a number.

What to skip for now

• Running OpenTelemetry pipelines at scale. Learn what OpenTelemetry is (a standard way to produce metrics, logs and traces), and leave it there.
• Thanos, Mimir and other tools for storing metrics at large scale.
• Running your own Elasticsearch cluster for logs.
• Comparing paid vendors such as Datadog. The ideas you learn here carry over.
• Distributed tracing, until you have more than one service to trace.

Common mistakes

• Alerting on everything. When most alerts need no action, people stop reading them. Every alert should need a person to do something. If it does not, make it a dashboard panel.
• Never testing the alert. An alert you have not seen fire may not work at all. Break the app and check that the message arrives.
• Running the monitor on the machine it watches. If that server dies, your alerts die with it. Add one check from outside, such as the health-check tool from step 4 running somewhere else.
• Putting user IDs in metric labels. Each new label value creates a new time series in Prometheus, so memory use climbs fast. Put those details in logs. The naming guide has more on labels.
• Logging secrets. Full request bodies can hold passwords and tokens. Log only the fields you need.

Your project

Add metrics, logs and one alert to your app. Then run a load test with a tool such as k6, raising traffic until something gets slow. Save a screenshot of the graph and write a few lines on what ran out first and how you know. Then stop the database and time how long the alert takes to reach you. Add those notes to your repo, as you will use this setup at the incident response step.