Google Cloud Architecture: Make Workload Decisions Explicit

Architecture Decisions

A Google Cloud architecture connects business goals to project boundaries, identities, entry paths, compute, data, events, operations, recovery, and cost ownership. Product names are implementation choices inside that larger system.

Good diagrams show trust and failure boundaries as well as arrows. Good decision records explain assumptions, rejected alternatives, measurable objectives, and the condition that would trigger a redesign.

Platform Boundaries

Separate projects when environment, team, policy, quota, billing, data sensitivity, or blast radius requires it. Centralized folders, Shared VPC, logging sinks, and policy can govern many workload projects without putting every resource in one project.

Choose a region from users, data residency, service availability, network transfer, dependency location, and recovery needs. Multi-region is not automatically more reliable if the application or its data path still has a single dependency.

Managed Service Selection

Cloud Run removes most host and cluster work for compatible containers. GKE exposes Kubernetes behavior. Compute Engine exposes the operating system. The correct choice depends on workload constraints and the team capabilities needed to operate it at 3 a.m.

Apply the same reasoning to data: query model, consistency, transaction boundary, scale, backup, restore, and regional topology matter more than a familiar product category.

Decision	Evidence to collect	Failure question
Availability target	User journey SLI and allowed downtime	Which dependency can consume the error budget?
Recovery target	RTO, RPO, backup, replication, restore drill	Can operators recover after deletion or corruption?
Scaling model	Load shape, quota, downstream limits, unit cost	Where does backpressure appear first?
Security boundary	Principal, policy, network, data, audit trail	How is misuse blocked and investigated?

Change and Failure Review

Design reviews should test zone loss, dependency latency, quota exhaustion, duplicate events, credential rotation, bad deployment, schema change, data restoration, and sudden demand. Game days expose assumptions that static diagrams hide.

Cost is an architecture signal. Track cost per business unit, request, job, or tenant; then decide whether caching, batching, retention, topology, commitments, or a different service changes unit economics without weakening objectives.

Record one architecture decision

Decision: Run the catalog API on Cloud Run in us-central1
Context: HTTP container, bursty traffic, no host dependencies
Objectives: 99.9% availability; p95 under 300 ms
Data: Regional Cloud SQL with tested point-in-time recovery
Identity: Dedicated catalog-api service account
Trade-off: Regional dependency limits disaster recovery
Trigger to revisit: RTO below 60 minutes or multi-region requirement
Evidence: Load test, restore drill, canary dashboard, monthly unit cost

A decision record is short enough to maintain and precise enough to challenge.
Dates, owners, and links to test evidence should be included in the real repository.

Requirements and Decision Records

Translate business goals into user journeys, load shape, latency percentiles, availability, RPO, RTO, consistency, residency, security, compliance, and cost-per-unit objectives. Include growth, seasonality, team skill, migration constraints, and support hours. Without measurable requirements, service selection becomes a collection of preferences that cannot be reviewed after production evidence arrives.

An architecture decision record states context, options, decision, trade-offs, evidence, owner, date, and a trigger to revisit. Record why Cloud Run, GKE, Compute Engine, Cloud SQL, Spanner, Pub/Sub, or another product fits this workload instead of listing product features. A rejected alternative can become correct when scale, organization, or regulation changes.

Keep diagrams aligned with deployed reality. Show project and folder boundaries, regions and zones, identities, ingress, network paths, compute, authoritative data, caches, event flows, administrative paths, logging sinks, recovery stores, and external dependencies. Label protocols and trust boundaries so reviewers can reason about failure and misuse, not only connectivity.

Make every major service choice traceable to a requirement.
Assign owners to decisions and review triggers.
Include administrative and recovery paths in diagrams.
Validate diagrams against infrastructure and telemetry.

Reliability and Recovery Design

Map every user journey to dependencies and determine which failures consume its error budget. Zonal distribution helps only when load balancing, state, quotas, deployments, and dependencies can continue through zone loss. Regional or multi-regional services still need an application plan for control-plane operations, client retries, data conflicts, and dependencies with narrower topology.

Use timeouts derived from an outer request deadline, bounded retries with jitter, idempotency, circuit breaking, load shedding, and backpressure. Retries multiply load during failure and can turn a slow dependency into a wider outage. Define which layer retries each operation and how ambiguous writes are reconciled.

Recovery design distinguishes high availability, backup, point-in-time recovery, replication, failover, and disaster recovery. Set RPO and RTO per data class and workflow, protect recovery credentials and copies, and exercise restoration. A document claiming a one-hour RTO is not evidence if DNS, secrets, infrastructure, data validation, and backlog reconciliation have never been timed together.

Test zone, region, quota, and dependency failures separately.
Budget retries across the entire call graph.
Keep recovery control outside ordinary workload compromise paths.
Measure RPO and RTO in full-system exercises.

Security and Operational Ownership

Design workforce, deployment, runtime, data, network, key, and audit roles as separate capabilities. Apply least privilege at stable resource boundaries and prefer short-lived federation over exported keys. Use organization policy and service perimeters where they match the threat, but retain service-level authorization and application checks for every sensitive action.

For each managed service, write what Google operates and what the team still owns. Cloud Run removes node management but not concurrency, identity, data, or release design. GKE manages a control plane but not application disruption behavior. Cloud SQL automates infrastructure tasks but not query design, connection pressure, schema safety, or restore validation.

Assign service owner, data owner, security contact, cost owner, incident escalation, and recovery operator. Verify access and documentation through handoff exercises. Architecture that depends on one expert or one privileged laptop has an availability risk even when every cloud resource is redundant.

Record the remaining duty for every managed service.
Test unauthorized paths as part of architecture review.
Use groups and runbooks that survive personnel changes.
Protect audit and recovery administration independently.

Performance, Cost, and Evolution

Build a capacity model from arrival rate, concurrency, service time, payload size, data growth, cache behavior, fan-out, quotas, and downstream limits. Load tests should preserve skew, burst, and failure behavior. Measure p50, p95, p99, errors, saturation, queue age, and unit cost; an average response time can hide the users who experience overload.

Estimate compute, database, storage, operations, network transfer, logging, backup, support, and engineering cost. Compare steady, burst, and disaster scenarios. A managed service may have a higher unit price but lower operating burden, while a commitment may reduce rate but create utilization risk. Optimize only within reliability and security objectives.

Review architecture after incidents, major releases, traffic or data growth, new regulation, team changes, product deprecations, and sustained cost shifts. Use fitness functions such as policy tests, restore drills, SLO checks, dependency inventories, and cost queries to detect drift. Evolution should be a sequence of reversible decisions rather than a periodic rewrite driven by fashion.

Test representative skew and burst, not uniform toy traffic.
Track cost per business outcome beside service cost.
Set measurable triggers for revisiting decisions.
Prefer reversible migration steps with verification gates.

External Dependency Review

Cloud architecture diagrams often stop at the Google Cloud boundary even though identity providers, payment services, package repositories, DNS registrars, email vendors, client networks, and on-premises systems determine availability. List each external dependency, owner, protocol, authentication, quota, data exchanged, latency objective, failure behavior, and support path.

Define a deadline, retry and circuit policy for each synchronous call. For asynchronous integrations, define acknowledgment, deduplication, replay, and reconciliation. Decide which user journeys degrade, queue, use cached data, or stop when the provider is unavailable. A fallback is valuable only when it is independent enough and tested with current data and credentials.

Monitor the boundary from both technical and business signals. Provider status pages are context, not proof of your impact. Use synthetic transactions where permitted, record correlation IDs, and retain request evidence without leaking sensitive payloads. Review contract limits and certificate or credential expiry before launches.

For a critical provider, compare contractual availability with the application objective and budget the dependency contribution explicitly. If the provider target is lower than the user journey target, caching, asynchronous acceptance, multiple providers, or a reduced-function mode may be necessary. Verify that the fallback does not depend on the same DNS, identity, network, or data source.

Include third-party and organizational services in failure diagrams.
Give each dependency an owner and escalation route.
Test degraded behavior and recovery reconciliation.
Track quota, credential, and contract renewal dates.
Retest fallbacks after provider contract changes.
Measure fallback cost during every resilience exercise.
Record who approves degraded operation.

Architecture Review Examples

Record an External Dependency Decision

A checkout service depends on a third-party tax API with variable latency.

Record an External Dependency Decision

Constraints: Checkout must finish within two seconds and tax results may be retried but not duplicated.
Decision: Set a bounded deadline, use idempotent request keys, isolate retries, and define a degraded response approved by the business.
Verification: Latency injection exercises the timeout and fallback while traces show bounded attempts and no duplicate charge.
Failure test: Return slow successes after the deadline and confirm late responses cannot mutate completed checkout state.

Output

Expected evidence: Latency injection exercises the timeout and fallback while traces show bounded attempts and no duplicate charge.

This is a worked engineering decision, so the result is operational evidence rather than terminal output.

Before you move on