An AI feature is still an API integration. Network faults, overloaded services, invalid requests, duplicate jobs, and slow downstream dependencies are normal operating conditions. The application needs to recognise which failures may be retried, how long it may wait, and when it should show a useful fallback instead.
This lesson treats reliability as a contract at the edge of your application. You will build a retry classifier, an idempotent job key, and a concise trace record. The model request stays inside a bounded operation rather than becoming an untracked background risk.
A timeout is a product decision. A browser feature might wait a few seconds and offer a queued result; a background document job may have a longer budget but should still end. Pass a deadline through the entire workflow so a slow tool does not outlive the user request without anyone noticing.
Do not use an endless retry loop to hide a design problem.
Retry only failures that are plausibly transient and only when repeating the work is safe. Bad credentials, malformed parameters, blocked permissions, and invalid schemas need a corrected request, not more traffic. Use documented error codes and headers from the provider rather than guessing from a message string.
Keep the number of attempts small and make the final failure visible in telemetry.
Rate limits protect a shared service and give you a capacity signal. Queue background work, cap concurrency, respect retry-after guidance, and use load shedding when your own system is overwhelmed. A graceful response is better than flooding a dependency and making recovery slower for everyone.
Separate interactive traffic from long-running or batch traffic when the product allows it.
A model response may recommend an action, but the action belongs to your application. If an operation creates a ticket, stores a summary, or calls a billing system, attach an idempotency key before the side effect. Retries can then return the prior result rather than creating duplicates.
Log the state transition, not the entire customer conversation.
A request that works once is not an operational design. Your adapter needs a deadline, bounded retries for transient failures, an idempotency strategy for work your application initiates, and a stable error result for the caller. The customer should never have to infer whether the system is still working from a raw provider exception.
Keep retries close to the API boundary and actions behind a separate confirmation or job record. A retry can safely repeat a read-only generation request, but it must not accidentally repeat an external side effect such as sending an email or issuing a refund.
Client settings should be server-side configuration, not hidden defaults spread through route handlers.
from anthropic import Anthropic
client = Anthropic(
timeout=15.0,
max_retries=2,
)
# The surrounding application still owns its request deadline and error handling.This pure function keeps retry policy easy to test without making network calls.
def next_action(status_code: int) -> str:
if status_code in {408, 429, 500, 502, 503, 504}:
return "retry_with_backoff"
if status_code in {400, 401, 403, 404, 422}:
return "fix_or_reject"
return "investigate"
print(next_action(429))
print(next_action(401))retry_with_backoff
fix_or_rejectNot every failure deserves another attempt. This policy stops retries for request and authorization problems.
RETRYABLE = {429, 500, 502, 503, 529}
def should_retry(status_code: int, attempt: int) -> bool:
return status_code in RETRYABLE and attempt < 2
print(should_retry(429, 0))
print(should_retry(400, 0))True
FalseThe application records the request key before a downstream workflow can run twice.
seen_keys: set[str] = set()
def accept_once(key: str) -> bool:
if key in seen_keys:
return False
seen_keys.add(key)
return True
print(accept_once("support-104:reply-1"))
print(accept_once("support-104:reply-1"))True
FalseAn operations design lab
0 of 2 completed
No. Respect current provider guidance, back off with jitter, reduce concurrency, and queue work when appropriate.
At a well-tested integration boundary, not repeated ad hoc across every route or prompt.
Explore 500+ free tutorials across 20+ languages and frameworks.