Monitoring and Logging with CloudWatch

If you run anything on AWS, CloudWatch is already collecting data about it whether you asked or not. It is the default observability service - metrics, logs, alarms, and dashboards in one place, wired into nearly every other AWS service out of the box. That "already there" quality is its biggest strength and its biggest trap: the automatic parts lull you into thinking you have monitoring, while the things that actually page you at 3am (memory pressure, application errors, cost from unbounded log retention) are exactly the parts you have to set up yourself. This guide walks the four pieces you will use - metrics, logs, alarms, dashboards - and points out where the defaults quietly leave a gap. It assumes you know the basics from the AWS fundamentals guide.

Metrics: what AWS collects for free, and what it does not

A metric in CloudWatch is a time-ordered series of numbers - CPU utilization, request count, queue depth - each point a value at a timestamp. Almost every AWS service publishes its own metrics automatically, at no extra setup. Launch an EC2 instance and you get CPUUtilization, NetworkIn, DiskReadOps, and more, every five minutes (or every minute if you enable detailed monitoring). Put an Application Load Balancer in front and you get RequestCount, TargetResponseTime, HTTPCode_Target_5XX_Count. RDS, SQS, Lambda, DynamoDB - all of them publish. You did not instrument anything; the service did it for you.

Metrics are organized by namespace and dimensions. A namespace is a bucket that keeps metrics from different services from colliding - AWS/EC2, AWS/ApplicationELB, AWS/Lambda, and so on (custom namespaces do not start with AWS/). Dimensions are name-value pairs that pin a metric to a specific resource: CPUUtilization in AWS/EC2 with dimension InstanceId=i-0abc123 is one instance's CPU; the same metric with AutoScalingGroupName=web-asg is the group's. Dimensions are how you slice from "all EC2 CPU" down to the one box you care about.

Here is the gotcha that catches everyone. AWS metrics only see what the hypervisor can see from outside the guest OS. For EC2 that means CPU, network, and disk I/O at the volume level - but not memory usage and not disk space used inside the filesystem. There is no MemoryUtilization metric for EC2 by default, and people burn real time looking for it. To get memory, disk-used percentage, or any in-guest number, you install the CloudWatch agent on the instance, which reads them from inside the OS and publishes them as custom metrics.

# Publish a custom metric directly from the CLI (the agent does this for you at scale)
aws cloudwatch put-metric-data \
  --namespace "MyApp" \
  --metric-name QueueDepth \
  --dimensions Service=worker,Env=prod \
  --value 42

Custom metrics - whether from the agent or your own put-metric-data calls - are billed per metric, so do not blindly emit a high-cardinality dimension (a metric per user ID will explode both your bill and the console). Use dimensions for things you have a bounded, sensible number of: environment, service, instance, region.

Logs: groups, streams, and the retention bill

CloudWatch Logs is where text output goes - application logs, system logs, Lambda output, VPC flow logs. The model is two levels. A log group is a named container for one logical source, usually one application or service (for example /aws/lambda/checkout or /var/log/nginx). A log stream is a single sequence of events within that group, typically from one instance, container, or function invocation. So a service running on ten instances has one log group with ten (or more) streams; you set retention and metric filters on the group, and read individual streams underneath.

How logs get in depends on the source:

• Lambda ships stdout/stderr to CloudWatch Logs automatically - a log group per function, zero setup. This is the one that is truly free-of-effort.
• EC2 and on-prem servers need the CloudWatch agent (the same agent that publishes memory metrics). You give it a config file listing which files to tail and which log group to send them to.
• Applications can write directly via the SDK's PutLogEvents, but in practice you almost always let the agent, a container log driver, or the platform handle shipping rather than calling the API from app code.
• ECS/EKS use the awslogs (or FireLens) log driver to route container stdout into log groups.

The setting people forget is retention. By default a new log group keeps logs forever (Never expire), and CloudWatch Logs is billed for storage. Left alone, chatty services quietly accumulate gigabytes for years and the bill creeps up with no obvious cause. Set a retention period on every log group deliberately - short (7-14 days) for noisy debug logs, longer where compliance or audit actually requires it. It is both a cost decision and a compliance one, and "Never expire" is rarely the right answer.

# Set retention to 14 days - do this on every group you create
aws logs put-retention-policy \
  --log-group-name /aws/lambda/checkout \
  --retention-in-days 14

Querying with Logs Insights

Scrolling raw log streams does not scale past a handful of lines. CloudWatch Logs Insights is a purpose-built query language over your log groups - you pick one or more groups, write a query, and it scans the matching time range. It is how you actually answer "which requests errored in the last hour and how slow were they."

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

If your logs are structured JSON, Insights parses the fields automatically and you can query them by name - filter on a statusCode, group by path, compute percentiles:

fields @timestamp, path, duration
| filter statusCode >= 500
| stats count() as errors, avg(duration) as avg_ms by path
| sort errors desc

Insights is billed by the amount of data scanned, so scope queries to a tight time range and the specific groups you need rather than "everything, last 30 days." Structured (JSON) logs pay off enormously here - they turn free-text grep into real queries.

Alarms: turning metrics into action

A metric nobody watches is just a chart. An alarm watches a metric (or a Logs Insights-derived metric) against a threshold and changes state, and that state change is what drives notification and automation.

A metric alarm has three states:

• OK - the metric is within the threshold.
• ALARM - the threshold was breached for the configured number of evaluation periods.
• INSUFFICIENT_DATA - not enough data points to decide (a new alarm, or a metric that stopped reporting).

You define it as: this metric, this statistic (average, sum, p99...), this comparison and threshold, sustained over this many periods. The "number of periods" part matters - alarming on a single spiky data point produces noise, so most real alarms require the breach to persist (for example, 3 out of 3 five-minute periods) before firing.

# Alarm when average CPU on an instance stays above 80% for 15 minutes
aws cloudwatch put-metric-alarm \
  --alarm-name web-high-cpu \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abc123 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 3 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:123456789012:ops-alerts

An alarm's actions are where it earns its keep. When it enters ALARM (or OK, or INSUFFICIENT_DATA) it can:

• Notify by sending to an SNS topic, which fans out to email, Slack, PagerDuty, or a Lambda for custom handling. This is the classic "page the on-call" wiring.
• Scale by triggering an Auto Scaling policy - CPU high for 10 minutes adds instances, low for 20 removes them. The alarm is the trigger for elasticity.
• Remediate by invoking a Lambda (via SNS) or an EC2 action (reboot, stop, recover) to fix a known failure mode automatically.

A composite alarm combines several alarms with boolean logic (ALARM(A) AND ALARM(B)) so you only page when a real problem is present. The everyday use is suppressing noise: alert on "5xx errors high AND healthy-host count normal" so a routine deploy that briefly drops hosts does not fire, and route a single meaningful page instead of five separate ones for the same incident.

Dashboards: the at-a-glance view

A dashboard is a saved, shareable layout of widgets - line graphs, numbers, alarm status, log tables - pulling from any metrics and log groups in the account (and across accounts and regions). The point is one screen that answers "is the system healthy right now?" without hunting through per-service consoles.

Build dashboards around a service or user journey, not around AWS's internal boundaries. A good service dashboard puts request rate, error rate, latency percentiles, and saturation (CPU, memory, queue depth) side by side, so during an incident you read one page instead of clicking between EC2, ELB, and RDS consoles. Keep them focused - a dashboard with forty widgets is as useless as no dashboard because nobody can scan it. You can define dashboards as JSON and commit them to your infrastructure repo, which keeps them versioned and reproducible alongside the resources they watch.

Tracing and the golden signals

CloudWatch metrics and logs tell you that something is wrong and where in aggregate, but not the path of a single slow request across services. That is tracing, and on AWS it is X-Ray (now surfaced under the CloudWatch console as part of its wider observability suite). X-Ray follows one request through the load balancer, your service, the database, and any downstream calls, showing which hop ate the latency. Reach for it when a request is slow but no single metric is obviously red - it turns "the checkout is slow sometimes" into "the payment service's call to DynamoDB is the 300ms." Metrics, logs, and traces are the three complementary signals; CloudWatch covers all three.

A useful frame for what to monitor, regardless of tool, is the golden signals from Google's SRE practice. Applied to CloudWatch:

• Latency - how long requests take. TargetResponseTime on an ALB, or a custom timing metric; watch percentiles (p95, p99), not just the average, because the average hides the tail.
• Traffic - how much demand. RequestCount, messages per second, invocations - your load, so you can tell a real incident from a traffic surge.
• Errors - the rate of failed requests. HTTPCode_Target_5XX_Count, Lambda Errors, or an Insights metric filter counting ERROR lines. Usually your most important alarm.
• Saturation - how full the system is. CPU, memory (agent metric), disk, queue depth, connection-pool usage - the leading indicator that you are about to run out of headroom.

Alarm on these four for every service and you catch the large majority of real problems, without drowning in alerts on metrics that do not map to user pain.

The shape of it

CloudWatch gives you observability in four moving parts. Metrics arrive automatically from AWS services, sliced by namespace and dimensions - but anything inside the guest OS (memory, disk-used) needs the CloudWatch agent publishing custom metrics. Logs land in log groups and streams; Lambda ships them free, EC2 and apps need the agent, and every group needs a deliberate retention setting for cost and compliance, with Logs Insights for real querying. Alarms turn a metric crossing a threshold into an action - notify via SNS, scale an Auto Scaling group, or remediate - and composite alarms cut the noise. Dashboards put it on one screen, X-Ray adds request-level tracing, and the golden signals - latency, traffic, errors, saturation - tell you what to watch in the first place. The defaults hand you the easy 80 percent; the value is in setting up the parts AWS deliberately leaves to you.