AWS observability combines metrics, logs, traces, events, and API audit history to explain workload behavior.
CloudWatch handles service metrics, custom metrics, log groups, dashboards, and alarms. CloudTrail records AWS API activity, while X-Ray and OpenTelemetry-compatible tooling can trace requests across services.
Useful monitoring starts from user symptoms and service objectives, then connects those outcomes to dependency and resource signals.
Track request rate, error rate, latency, and saturation for each important request path. Add service-specific signals such as queue age, database connections, Lambda throttles, or load balancer unhealthy hosts.
Emit structured records with timestamps, severity, request IDs, and safe business context. Centralize log groups, set retention deliberately, and prevent secrets or personal data from entering routine diagnostics.
An alarm should identify an owner and response, not merely state that a metric crossed a line. Composite alarms and metric math can reduce noise when one dependency causes many secondary symptoms.
Use CloudTrail to answer who changed a resource and when, then align that event with deployments, configuration history, application logs, and metrics. Preserve incident evidence before short retention periods remove it.
Begin with one user or business journey and define what healthy means: availability, latency, correctness, freshness, and throughput within an agreed window. A service-level indicator measures that outcome; an objective sets the acceptable target. Resource telemetry then explains why the objective is at risk rather than replacing it.
Use metrics for numeric behavior over time, logs for detailed events, traces for causality across components, and change records for control-plane activity. Events and profiles can add state changes and code-level cost. Each signal has different cardinality, retention, query, privacy, and price characteristics, so collect it because it answers an operational question.
Map every request path to entry point, compute, dependency, data store, queue, and external service. Define the correlation fields that connect those layers. A dashboard full of unrelated service defaults is less useful than one view that follows customer impact into the constrained dependency.
AWS services publish metrics into namespaces with dimensions that identify resources or variants. A metric is determined by namespace, name, and exact dimension set; adding a dimension creates another time series. Choose period, statistic, and unit from the decision being made. Percentiles often reveal user latency that an average hides.
Publish custom metrics for domain outcomes such as completed orders, rejected payments, queue work age, or stale data, but control dimension cardinality. Request IDs, user IDs, and object keys belong in logs or traces, not as unbounded metric dimensions. Use embedded metric formats or approved telemetry libraries where they improve consistent context.
Metric math can derive error ratios, utilization, or burn rate from several series. Validate missing-data and low-sample behavior before using a derived alarm. Keep dashboards and alarms in infrastructure code so a new environment receives the same observability contract as the workload.
Emit structured records with timestamp, severity, service, environment, release, event name, correlation ID, safe dimensions, and classified error data. Keep the human message, but make fields stable enough for Logs Insights queries and metric filters. Log an error once at the layer that owns the outcome and preserve its cause.
Redact credentials, authorization headers, cookies, secrets, private request bodies, payment data, and personal information before serialization. A field allowlist is safer for high-risk events than attempting to mask every possible secret afterward. Set log-group retention from operational and legal need instead of accepting indefinite retention.
Design for burst volume, multiline stack handling, subscription delivery failure, and query cost. Create saved queries for known incidents and verify logs after they reach the actual destination. Local console output can appear correct while parsing, truncation, permissions, or retention make production evidence unusable.
A CloudWatch alarm evaluates datapoints across periods and changes state according to threshold, evaluation window, and missing-data treatment. Configure those values from the failure duration the system can tolerate. A one-minute spike and a fifteen-minute outage should not accidentally produce the same response.
Choose whether missing data means breaching, not breaching, ignored, or missing based on the metric. Absence of successful request data can indicate an outage, while absence of error events may be normal. Composite alarms can suppress dependent noise or require several conditions, but their logic must remain understandable during an incident.
Route alarms through SNS or the incident platform to an accountable team with severity, dashboard, runbook, and correlation context. Test delivery and escalation, including outside working hours. An alarm that nobody receives or knows how to act on is only stored configuration.
CloudTrail records account activity such as console, CLI, SDK, and service API calls. Event history supplies a regional view of recent management events, while trails deliver selected ongoing events to S3 and optional integrations. CloudTrail Lake event data stores support selected event capture and query. Choose among them from retention, account scope, event type, query, and cost needs.
Management events describe control-plane operations. Data events cover high-volume resource operations for supported services and must be selected deliberately. Insights, network activity events, and other event categories solve narrower questions. Centralize organization evidence, protect its bucket and keys from workload administrators, monitor delivery, and synchronize timestamps in UTC.
During an incident, search by event source, event name, user identity, resource, request ID, source address, and time. A CloudTrail event explains an API attempt; it does not prove the application result or packet path. Correlate it with AWS Config history, deployment markers, workload logs, metrics, and traces.
Use X-Ray or OpenTelemetry-compatible instrumentation to propagate trace context through HTTP, functions, queues, and supported services. Sampling controls cost and volume but can miss rare failures, so preserve error and high-latency evidence according to policy. Do not put secrets or unrestricted payloads in span attributes.
Mark deployments and configuration changes on dashboards. During diagnosis, move from customer symptom to trace, dependency metric, structured log, and control-plane change without relying on one engineers memory. Keep a runbook for common failures with exact evidence locations and safe first actions.
Test observability by injecting a controlled application error, slow dependency, denied API request, failed queue message, and stopped telemetry destination. Confirm that alarms fire, the operator can identify the release and request, protected data is absent, and evidence remains available for the required period. Track telemetry ingestion and query cost as part of the service budget.
CloudWatch cost can come from custom metric series, detailed monitoring, log ingestion and storage, queries, alarms, dashboards, traces, and cross-account or cross-Region movement. Group spend by telemetry type and workload, then connect each high-volume signal to a diagnostic or compliance question. Do not delete evidence blindly to make the bill smaller.
Reduce cost by lowering unnecessary cardinality, filtering verbose success logs, choosing intentional retention, sampling traces, and avoiding repeated full-range queries. Preserve security, audit, and recovery evidence according to policy. After every change, repeat a failure drill to prove operators can still find the signal that justified collection.
aws logs filter-log-events \
--log-group-name "/aws/lambda/orders-api" \
--filter-pattern "ERROR" \
--start-time 1717000000000Users report slow checkout even though average latency looks normal.
Constraints: The service objective is based on p95 latency and error rate; low traffic must not create noisy missing-data alerts.
Decision: Alarm on sustained p95 breach with a minimum request volume and link the notification to logs, traces, and a runbook.
Verification: A controlled slow dependency produces one routed alarm whose dashboard and trace identify the dependency.
Failure test: Stop traffic completely and confirm the alarm follows the chosen missing-data policy.Expected evidence: A controlled slow dependency produces one routed alarm whose dashboard and trace identify the dependency.Explore 500+ free tutorials across 20+ languages and frameworks.