Cloud Monitoring collects platform and application metrics, while Cloud Logging stores structured records from services and workloads. Traces, error groups, audit logs, uptime checks, dashboards, and alert policies connect those signals to investigation and response.
Useful observability answers whether users are succeeding, which dependency explains a failure, and what changed. Collecting every available field without retention and routing decisions can produce noise and cost without improving diagnosis.
For an API, begin with successful request ratio and latency at a meaningful percentile. Add traffic, saturation, dependency failures, and backlog age only when they explain the user-facing objective.
An alert needs a condition, duration, notification route, owner, severity, runbook, and recovery behavior. Page on actionable symptoms; route slower capacity or cost trends to planned work.
Emit structured fields for severity, service, revision, trace or correlation ID, operation, outcome, and safe identifiers. Never log passwords, tokens, signed URLs, payment data, or unrestricted request bodies.
Log-based metrics can count a stable event that is not already a native metric. Keep label cardinality bounded; user IDs, random request IDs, and raw error messages make expensive metric labels.
gcloud logging metrics create orders_api_5xx \
--description="Cloud Run 5xx responses for orders-api" \
--log-filter='resource.type="cloud_run_revision"
resource.labels.service_name="orders-api"
httpRequest.status>=500'
gcloud logging metrics describe orders_api_5xxLog exclusions reduce stored volume but should not remove evidence required for security, compliance, or incident response. Log buckets, retention, views, sinks, and analytics routing determine where records live and who can inspect them.
Test alerts by creating a known failure, then verify detection, notification, runbook accuracy, and automatic closure. A dashboard that no one opens during an incident is documentation, not a response system.
Cloud Monitoring stores numeric time series identified by metric type, resource type, labels, and time. Cloud Logging stores structured log entries, Cloud Trace follows request spans, Error Reporting groups application errors, and Profiler samples runtime behavior for supported environments. These products overlap during investigation but answer different questions; collecting one does not automatically create the others.
Start from a user journey and propagate a trace or correlation identifier through ingress, service calls, events, and jobs. Emit structured fields for service, version, environment, tenant-safe identifier, operation, result, duration, and retry count. Keep sensitive data, credentials, and unbounded payloads out of labels and logs. High-cardinality metric labels can create cost and operational problems, while logs are more suitable for detailed identifiers.
Use OpenTelemetry or supported agents when portability and consistent instrumentation matter, but define semantic conventions and ownership before deploying collectors. A telemetry pipeline also has buffers, quotas, credentials, sampling, and failure modes. Monitor dropped spans, rejected log entries, delayed metrics, and collector saturation so an outage does not silently erase the evidence needed to diagnose it.
A service-level indicator measures behavior users care about, such as successful checkout ratio or request latency below a threshold. A service-level objective sets the acceptable target over a window, and the error budget represents tolerated unreliability. Availability of a VM is not a substitute for success of the user journey when DNS, load balancing, code, database, and payment dependencies can fail independently.
Alert on symptoms that require action. Multi-window burn-rate alerts detect both rapid budget consumption and slower sustained degradation. Pair them with capacity and dependency alerts that warn before impact when operators have a clear response. Every page should identify the service, condition, user effect, dashboard, recent change, runbook, and escalation owner.
Test alert logic by injecting known failures and checking detection time, routing, grouping, silence behavior, and recovery notification. Avoid thresholds copied from another service without a baseline. For low-volume services, a ratio can swing wildly from one request, so use minimum-event conditions, synthetic checks, longer windows, or absolute error evidence as appropriate.
Log buckets define location, retention, and access behavior. Sinks route matching entries to another log bucket, BigQuery, Cloud Storage, or Pub/Sub. Centralize security and platform evidence in a protected project where workload administrators cannot erase the only copy. Sink writers need explicit destination access, and an exclusion at the wrong scope can remove data before it reaches a downstream destination.
Choose retention by investigation, legal, privacy, and cost requirements. Keep high-volume debug logs for a shorter period than audit evidence when appropriate. Log-based metrics turn matching entries into time series but inherit the quality and cardinality of the filter and labels. Test the filter against positive and negative fixtures before using it for a security alert.
During incident review, correlate deploy markers, audit changes, user-facing SLI changes, dependency latency, resource saturation, and representative traces. Preserve raw evidence before transforming it. A dashboard should accelerate a hypothesis, not hide query definitions or replace a timeline. Record timezone, query window, project, resource type, and filters so another operator can reproduce the result.
A useful dashboard follows the system path: traffic, errors, latency, saturation, release, dependencies, backlog, and business outcome. Separate overview, service diagnosis, and capacity planning views instead of placing every metric on one screen. Annotate deployments and policy changes, and keep units and aggregation explicit. Averages can hide a failed zone or a small group of users, so include distributions and breakdowns that support action.
At incident start, establish impact, scope, start time, current hypothesis, incident commander, communications owner, and next decision. Stabilize the service through rollback, traffic shift, capacity change, feature disablement, or dependency isolation, then preserve evidence. Do not make several untracked configuration changes at once; each change should have an expected signal and rollback.
After recovery, build a factual timeline, identify contributing technical and organizational conditions, and assign corrective actions with owners and verification. Track time to detect, acknowledge, mitigate, and fully recover. Confirm that the new alert or runbook would have changed the outcome by replaying the scenario; otherwise the action merely creates documentation.
Observability volume is an architecture input. Estimate log bytes, metric time series, trace spans, retention, exports, and query patterns per request or job. A burst of repeated stack traces or an unbounded tenant label can increase cost precisely during an incident. Apply sampling and rate limits without removing rare security or failure evidence.
Create a telemetry quality dashboard for rejected entries, ingestion delay, collector drops, missing release labels, malformed structured logs, trace sampling, and noisy sources. Review the top volume and cardinality contributors with service owners. Reduce redundant success logs before shortening all retention indiscriminately.
Test that dashboards and alerts survive a project move, service rename, new resource type, or label change. Queries that silently return no data are more dangerous than visibly broken charts. Add synthetic telemetry fixtures or post-deployment assertions for critical alert paths.
Set ownership for custom metric descriptors and log-based metrics. Removing a label or changing its type can break dashboards and alerts, while abandoning obsolete descriptors leaves confusing series. Treat telemetry schema changes like API changes: review consumers, deploy compatibly, observe both versions, and retire the old form deliberately.
A service meets average availability but consumes its monthly error budget during short incidents.
Constraints: Alerts should detect fast and slow burns without paging on isolated errors.
Decision: Define a request-based SLI and combine short and long windows for burn-rate notification and paging.
Verification: Controlled failures trigger the intended severity and the alert links to traces, logs, and the service runbook.
Failure test: Generate one isolated failure and confirm it does not page.Expected evidence: Controlled failures trigger the intended severity and the alert links to traces, logs, and the service runbook.No. Alert on sustained or high-impact symptoms. Individual expected failures are usually better investigated through aggregated metrics, logs, and traces.
Many services publish platform metrics without an agent. VM and application telemetry may require the Ops Agent, OpenTelemetry, or explicit instrumentation.
Explore 500+ free tutorials across 20+ languages and frameworks.