A useful agent may wait for an API, a human approval, a scheduled time, or a remote specialist. If its process crashes during that wait, the work should resume from recorded state instead of disappearing or starting blindly from the beginning.
Durability means the workflow can reconstruct what happened and continue safely after interruption. It requires persisted state, explicit lifecycle transitions, replay-aware code, idempotent side effects, bounded retries, cancellation, and enough trace data to explain recovery.
Durable execution does not make a model deterministic. It makes the runtime accountable for which decisions and side effects have already happened.
A checkpoint should contain the goal, current phase, validated inputs, important observations, completed step IDs, pending approvals, artifact references, budgets, and version metadata. It should be possible for an engineer to understand the record without reconstructing an in-memory framework object.
Keep large documents and binary artifacts in controlled storage and save references with checksums and access policy. Checkpoint payloads need schemas, migrations, encryption, retention, and tenant isolation just like any other production data.
Do not assume the full model transcript is the workflow state. Transcripts are verbose, may contain sensitive material, and often fail to distinguish a proposal from an authorized action.
Some durable runtimes rebuild state by replaying workflow logic. A model call, clock read, random value, email send, payment, or file write can produce a different result or repeat an external action when replayed. Isolate such work behind recorded activities or steps.
Once an activity succeeds, persist a stable result before the workflow advances. During recovery, reuse that result. If the outcome is uncertain because the process crashed after the external service accepted the request but before the checkpoint committed, query the external system using an idempotency key or operation ID.
A timeout, rate limit, malformed model output, permission denial, and rejected approval require different responses. A blanket retry loop can amplify cost, repeat writes, overload a dependency, or keep asking a model to solve an impossible task.
Classify failures as transient, input-related, policy-related, permanent, or unknown. Retry only transient failures with capped exponential backoff and jitter. Repair or request input for validation failures. Stop on authorization denials. Escalate unknown side-effect outcomes for reconciliation.
A retry budget belongs to each step and to the whole run. When either budget is exhausted, preserve partial progress and return a clear status instead of looping.
| Failure | Typical Response | Automatic Retry |
|---|---|---|
| Rate limit or brief outage | Backoff with jitter | Yes, within budget |
| Invalid structured output | Repair once or use fallback | Limited |
| Permission denied | Stop and explain | No |
| Unknown payment outcome | Reconcile by operation ID | No blind retry |
| Human rejection | Stop or re-plan from feedback | No |
Distributed systems rarely offer one transaction across email, payments, tickets, and SaaS APIs. When a later step fails, the workflow may need a compensating action such as voiding a reservation, closing a draft ticket, or marking an artifact invalid.
Compensation is a new business action with its own authorization, failure modes, and audit record. It may not restore the exact original state. Define which completed steps are reversible, who may approve compensation, and what happens when compensation also fails.
Human review and external callbacks should move the run into a named waiting state. Store the expected signal, deadline, reviewer scope, and resume transition. A callback must be authenticated and correlated to the exact workflow and pending step.
Cancellation is a workflow operation, not simply a deleted queue message. Stop scheduling new work, attempt to cancel active activities, preserve completed side effects, and communicate what could not be undone.
Timers should use durable scheduling rather than sleeping inside a worker. Define what happens when a deadline passes: remind, escalate, expire, compensate, or return partial progress.
Happy-path tests do not prove durability. Terminate a worker before and after each side effect, duplicate queue deliveries, delay callbacks, expire credentials, change a tool version, and resume an old checkpoint under the new deployment.
The test passes when the final business outcome is correct, no side effect is duplicated, the state history remains understandable, and operators receive an actionable status. Include these cases in release gates for workflow and schema changes.
Persist state at meaningful transition boundaries: before a side effect, after a committed result, and before a long wait such as approval or an external callback. A checkpoint should include workflow and schema versions, next node, validated state, completed action IDs, pending timers, usage, and trace references. Avoid serializing live clients, secrets, or opaque provider objects that cannot survive deployment.
Retries create at-least-once execution unless the domain provides stronger guarantees. Give each external mutation an idempotency key, record the committed result, and check it before repeating. Keep database state and outbox or workflow events consistent so a process crash cannot commit a payment but lose the fact that it completed.
Version long-lived work. New workers may receive checkpoints created by old code, and rolling back may send new checkpoints to an older reader. Use explicit migrations or route runs to compatible worker versions. Test crash points, duplicate delivery, timeout, cancellation, approval expiry, and recovery after a partial provider outage.
This small ledger demonstrates the contract: the same operation key returns the recorded result instead of executing twice.
from dataclasses import dataclass
@dataclass(frozen=True)
class Result:
operation_id: str
status: str
ledger: dict[str, Result] = {}
def submit_refund(idempotency_key: str, order_id: str) -> Result:
if idempotency_key in ledger:
return ledger[idempotency_key]
result = Result(operation_id=f"refund:{order_id}", status="submitted")
ledger[idempotency_key] = result
return result
first = submit_refund("run-42:refund", "ORD-9")
second = submit_refund("run-42:refund", "ORD-9")
print(first)
print("Same recorded result:", first == second)
print("External submissions:", len(ledger))Result(operation_id='refund:ORD-9', status='submitted')
Same recorded result: True
External submissions: 1A real implementation stores the ledger transactionally and reconciles uncertain remote outcomes by operation ID.
Retries are selected from the failure class rather than applied to every exception.
RECOVERY = {
"RATE_LIMIT": "RETRY_WITH_BACKOFF",
"INVALID_OUTPUT": "REPAIR_ONCE_THEN_FALLBACK",
"PERMISSION_DENIED": "STOP_AND_REPORT",
"UNKNOWN_WRITE_OUTCOME": "RECONCILE_BY_OPERATION_ID",
"HUMAN_REJECTED": "REPLAN_OR_STOP",
}
failures = ["RATE_LIMIT", "PERMISSION_DENIED", "UNKNOWN_WRITE_OUTCOME"]
for failure in failures:
print(f"{failure}: {RECOVERY[failure]}")RATE_LIMIT: RETRY_WITH_BACKOFF
PERMISSION_DENIED: STOP_AND_REPORT
UNKNOWN_WRITE_OUTCOME: RECONCILE_BY_OPERATION_IDThe unknown write outcome is deliberately not retried because the first request may have succeeded.
0 of 2 checked
Try this next
0 of 3 completed
No. A checkpoint helps reconstruct state, but safe recovery also needs idempotent side effects, failure classification, lifecycle rules, version handling, and tested resume behavior.
Do not rely on that promise across external systems. Design consumers and side effects to tolerate duplicate delivery through idempotency and reconciliation.
Explore 500+ free tutorials across 20+ languages and frameworks.