Metrics, logs, and traces, so you hear about a failure before your users do.
Host, app, and the dependency in between. A green dashboard that hides the failure is useless.
I put the few graphs you'd check during an incident on one screen.
The alarm fires on the thing that breaks the app, not on every CPU twitch.
Once the graphs exist, the slow path and the wasted capacity are easier to see.
These show up when they fit the job.
Prometheus
Grafana
ELK Stack
Datadog
New Relic
Splunk
Nagios
Zabbix
Jaeger
Zipkin
AWS CloudWatchPrometheusGrafanaELK StackDatadogNew RelicSplunkNagiosZabbixJaegerZipkinAWS CloudWatchOn an ALB I start with HTTPCode_Target_5XX_Count and TargetResponseTime. Errors and latency come before CPU, because users feel those first.
An average latency can look fine while one request in a hundred takes ten seconds. I alarm on p99, or at least p95.
A CloudWatch alarm treats missing data as missing by default. A service that dies and stops sending metrics can sit in INSUFFICIENT_DATA instead of ALARM. For heartbeat metrics I set missing data to breaching.
A Prometheus label holding a user ID or a raw URL creates a new series for every value. In CloudWatch, each dimension combination is a separate custom metric. Log groups also never expire unless someone sets retention.
OpenTelemetry passes the W3C traceparent header over HTTP. A queue drops it unless the producer writes it into the message attributes and the consumer reads it back.
Writing
Posts that sit next to this work.
Use the Cloud Watch agent to send logging and debugging to Cloud Watch. You can check logs without logging into instances. It's accessible even after the instance is terminated. Life is far from perfect, and so does Amazon Lightsail instances. It can...
How I connect Langfuse, Lambda, and CloudWatch so I can trace prompt behavior, cost, retries, and failures in a production AWS app
