A dependable delivery path builds an immutable artifact, records its provenance, scans it, deploys it with reviewed configuration, verifies health, and promotes the same artifact rather than rebuilding from a moving branch.
Cloud Build can execute builds, Artifact Registry stores versioned packages and images, Cloud Deploy manages delivery pipelines for supported targets, and Terraform can declare infrastructure. Their service accounts and approval boundaries are part of the security design.
Pin base images and dependencies, run tests in a clean environment, generate provenance, scan for known vulnerabilities, and push to a controlled repository. Deploy by digest or immutable version so a later tag change cannot alter the release.
The build service account should be able to read source and write artifacts, but it does not automatically need production deployment rights. Separate identities make approvals and audit logs meaningful.
steps:
- name: gcr.io/cloud-builders/docker
args: [build, -t, ${_REGION}-docker.pkg.dev/$PROJECT_ID/apps/orders-api:$COMMIT_SHA, .]
- name: gcr.io/cloud-builders/docker
args: [push, ${_REGION}-docker.pkg.dev/$PROJECT_ID/apps/orders-api:$COMMIT_SHA]
- name: gcr.io/google.com/cloudsdktool/cloud-sdk
entrypoint: gcloud
args: [run, deploy, orders-api, --image, ${_REGION}-docker.pkg.dev/$PROJECT_ID/apps/orders-api:$COMMIT_SHA, --region, ${_REGION}, --no-traffic]
substitutions:
_REGION: us-central1Declarative infrastructure produces a reviewable plan and reduces console drift. State storage, locking, provider versions, policy checks, imports, and destructive changes require the same care as application code.
Use expand-and-contract database migrations so old and new application revisions can operate during a canary or rollback. A deployment rollback cannot restore deleted data or reverse an incompatible schema operation.
Predeployment checks confirm artifact identity, configuration, policy, quota, and dependency compatibility. Postdeployment checks compare errors, latency, saturation, logs, traces, and business events against a baseline.
Keep an emergency path, but protect it with strong authentication, narrow permissions, peer review where time permits, and automatic audit capture. Reconcile emergency changes back into source immediately.
A delivery chain starts with reviewed source and produces an immutable artifact plus evidence. Cloud Build executes steps using a build identity; Artifact Registry stores container images and language packages. Pin builder images and dependencies where reproducibility matters, isolate untrusted pull-request builds, and grant the build service account only the repositories, secrets, and deployment targets required for that pipeline.
Tag names are convenient references but can move. Promote and deploy container image digests so the tested bytes are the released bytes. Generate software bills of materials and provenance where required, scan artifacts, and establish a policy for critical vulnerabilities, exceptions, and rescans. Do not rebuild separately for each environment because the resulting artifacts are no longer identical.
Build logs can expose environment values and command output. Keep secrets in an approved integration, prevent forked or untrusted code from accessing privileged triggers, and avoid passing credentials as plain arguments. Preserve source revision, dependency lock, builder identity, artifact digest, test result, approval, and deployment record for traceability.
Manage projects, APIs, IAM, networks, service configuration, and application infrastructure through reviewed declarative code when practical. Terraform, Config Connector, and service-specific tools have different state and ownership models. Define one authoritative controller for each resource to avoid two systems repeatedly overwriting policy or configuration.
A plan or preview is evidence of intended change, not a guarantee of runtime safety. Review replacements, IAM expansion, network exposure, retention changes, regional moves, and data deletion separately. Protect state, lock concurrent runs, pin provider versions, and import existing resources carefully. Detect drift and decide whether to reconcile it or document an emergency exception.
Separate environment-specific values from reusable modules without creating a maze of inheritance. Validate policy, schema, quotas, and required APIs before deployment. Destructive database and storage changes need backup, restore proof, staged migration, and approval outside an ordinary application rollout.
A promotion moves an already built artifact through test, staging, and production gates. Tests should include unit behavior, contract compatibility, policy, vulnerability, migration, load, and recovery evidence according to risk. An approval is useful only when the approver sees the artifact identity, change, evidence, target, risk, and rollback plan.
Use a rolling, canary, blue-green, or traffic-splitting strategy that the target platform supports and the application data model can tolerate. Compare canary and baseline using errors, latency, saturation, business outcomes, and data correctness. Low-volume paths may need synthetic transactions or a longer observation period. Stop automatically when guardrails fail.
Feature flags separate code deployment from feature exposure but add configuration and cleanup work. Protect flag administration, log changes, define safe defaults, and remove stale branches after rollout. A flag cannot make an incompatible schema rollback safe unless both versions were designed for coexistence.
Rollback means restoring a known application and configuration state, but it may not undo data mutation, published events, emails, or partner calls. Design database changes through expand, migrate, and contract phases and keep old readers functional until rollback risk ends. For irreversible business effects, use compensating actions and reconciliation rather than pretending an image rollback erases history.
A runbook should identify the last known good artifact, configuration, secrets, schema compatibility, traffic command, authorization, and verification query. Practice rollback from a failed startup, increased error rate, latent data corruption, expired secret, broken IAM binding, and unreachable dependency. Measure decision and recovery time under realistic approval controls.
After recovery, verify user journeys, background work, backlog, data invariants, monitoring, backups, and cost. Preserve failed artifact and telemetry for analysis. Fix the control that allowed the issue, then rerun the pipeline; do not patch production manually and leave the declared state inaccurate.
A delivery pipeline can fail before build, during artifact publication, between approval and deployment, or after only part of a rollout. Define durable state for each stage so a retry does not publish a different artifact, duplicate a migration, bypass approval, or deploy to the wrong project. Use a release identifier that joins source, artifact digest, evidence, target, and deployment operation.
Test loss of the build worker after artifact upload but before status reporting. The retry should discover the existing immutable artifact and verify it rather than silently replacing a tag. Test an approval that expires while queued, revoked deployment permission, unavailable region, quota failure, and a target whose actual state drifted after planning.
A partially completed multi-service release needs an explicit compatibility policy. Either components remain backward compatible while promotion resumes, or the orchestrator rolls back the completed subset in a verified order. Avoid a global script that assumes every service supports identical rollback semantics.
Protect pipeline availability without creating an unreviewed bypass. Maintain a documented emergency deployment path using the same artifact verification and audit requirements, with narrower eligibility and immediate post-use review. If the control plane is unavailable, the incident plan should state whether the system waits, uses a pre-authorized recovery mechanism, or shifts traffic elsewhere.
Preserve deployment evidence in a project and retention boundary that the delivery identity cannot rewrite after the fact. The record should include approval identity, policy result, artifact digest, target revision, timestamps, rollout observations, rollback decision, and final verification. This makes release reconstruction possible even when the pipeline database or target service is part of the incident.
A container passes staging but production deployment uses a newly rebuilt tag.
Constraints: Promotion must preserve the tested bytes and allow fast rollback.
Decision: Promote the image digest from staging, deploy configuration separately, verify health, and retain the previous revision.
Verification: Staging and production report the same digest and rollback restores the prior revision without rebuilding.
Failure test: Move the mutable tag after approval and confirm digest-based deployment still selects the reviewed artifact.Expected evidence: Staging and production report the same digest and rollback restores the prior revision without rebuilding.Rebuilding can change dependencies, timestamps, base images, or generated output. Promoting one digest preserves the evidence gathered in earlier environments.
Terraform is strong for declarative infrastructure. Service-specific release tools often handle traffic shifting and application rollout semantics more safely.
Explore 500+ free tutorials across 20+ languages and frameworks.