Error Handling Design Patterns for External API Calls
Standard error handling misses silent failures where LLMs return 200 with wrong answers.
Error handling design patterns for external API calls must cover more than requests that visibly fail. Status codes, retries, and structured errors are still needed, but with LLM integrations, a response can succeed on the wire and still give a wrong answer. The design must handle all three layers of failure: transport errors, client/server faults, and silent semantic failures.
Why the standard API error playbook leaves gaps that matter
The standard playbook is incomplete for the systems developers ship in 2026. It handles deterministic APIs well because network failures, timeouts, and client/server faults produce exceptions or non-2xx status codes that software can classify. Retries address temporary trouble, while structured envelopes make failures debuggable.
Probabilistic systems add a different failure surface. LLM responses can be slow, partial, or substantively wrong without reporting an error. That creates three distinct layers, with transport errors below HTTP client/server faults and silent semantic failures beyond both. The engineering gap is tractable, but closing it requires checking both the answer and whether the request completed.
HTTP status codes as a communication contract, not just a lookup table
HTTP status codes let clients decide what happens next. Precision is the prerequisite for that decision. A 400 means the request has malformed syntax, invalid JSON, or missing fields, while 401 means credentials are missing or invalid and 403 means the caller is authenticated but lacks permission. A 404 identifies a missing resource, and 409 signals a conflict with its current state. For syntactically valid requests that fail business validation, 422 gives a more specific signal than a generic 400.
Most client faults need correction. A 429 requires separate handling because it signals a rate limit and should include Retry-After. On the server side, 500 signals an unhandled exception, 502 an invalid upstream response, 503 temporary unavailability, and 504 an upstream timeout. A 503 should also include Retry-After when recovery time can be estimated.
These distinctions are inputs to recovery logic. The practical rule is to use the most specific applicable code, including when 400 or 500 would be too generic. Returning 200 with an error buried in the body is worse because consumers must inspect every payload to discover whether it succeeded.
Structuring error responses so clients handle them once, not per-endpoint
A useful status code still needs a payload that has the same shape each time. Without one, consumers must write a parser for each error shape, sometimes for different endpoints in the same API. RFC 9457 Problem Details supplies a standard application/problem+json envelope. It succeeds RFC 7807 and is fully backward-compatible, so existing responses following that earlier standard are already compliant.
The envelope separates stable classification from occurrence-specific detail. Its type URI identifies the error type and links to documentation, while title provides a short, stable summary. The integer status repeats the HTTP status for convenience, and detail explains this particular failure. An instance URI identifies the specific request for log correlation.
Extensions make that shared shape more useful. They can include a machine-readable error_code, a request_id, and an errors array reporting every invalid field at once. That avoids the fix-one-submit-discover-another routine, which is a poor substitute for a validation interface. For a new API, the lesson is to adopt the standard across every endpoint. Reliable error shapes make retry classification and idempotency handling tractable without requiring clients to interpret human-readable prose.
Retry logic and exponential backoff: what to retry, what never to retry
Retry design starts with classification. A retry should address a transient failure, such as a network timeout, a temporary server fault, or a retryable 429. Repeating an unchanged request with bad credentials or invalid input only adds delay and can trigger rate limits. Persistence is not a repair strategy.
APIVerve illustrates the classification problem with two meanings of 429. One means requests arrived too quickly, so waiting and retrying can help. The other means the credit quota is exhausted, so retries should wait until renewal. A header or machine-readable response field must distinguish them. Parsing message text makes recovery depend on words written for people to read.
For retryable faults, exponential backoff doubles the delay after each attempt, while jitter stops clients from overwhelming a recovering service in lockstep. A supplied Retry-After overrides that calculation. LLM workloads call for longer base delays, more maximum attempts, and a higher maximum delay because providers can degrade in latency before reporting errors. But a hard deadline must limit the total elapsed time for the whole sequence, along with the timeouts for each attempt.
Write operations add a separate safety condition. They must be idempotent before retries are safe. Idempotency keys make retry-safe POST requests possible without duplicate side effects, including the classic duplicate charge. A timeout does not make a second charge acceptable.
Circuit breakers: stopping cascading failures before they propagate
Retries address individual requests, while circuit breakers address unhealthy dependencies. When a dependency’s failure rate exceeds a threshold over a rolling window, the circuit opens and rejects calls immediately. After a cooldown, a half-open state allows one test request through. Success closes the circuit, while failure keeps it open. This keeps a slow upstream from blocking threads, using up connection pools, and causing timeouts throughout the call chain.
For LLM integrations, counting HTTP errors is insufficient. A provider can slow down while its error rate stays at zero, so a conventional breaker stays closed as p95 latency climbs. An LLM-aware breaker therefore needs timeout events as well as HTTP failures.
If a configured percentage of responses fail validation over a rolling window, the circuit can open even when every call returns 200. That makes prompt drift, model checkpoint regressions, and context corruption visible to the recovery system when they cause schema failures.
Timeout design for LLM calls: why a single timeout is almost always wrong
Streaming LLM calls have two distinct latency phases. Time to first token, or TTFT, measures the wait before output begins, while total completion time measures how long the generation takes to finish. High TTFT points to provider-side queueing or capacity trouble, where aborting and retrying or failing over early is appropriate. A stream that runs for a long time may instead be an unusually long generation or a stall.
A short TTFT timeout detects requests that have not started, while a longer completion timeout gives legitimate generations room to finish. One timeout generous enough for worst-case completion leaves queueing failures undetected for too long. One short enough to catch those failures aborts valid long completions.
The calling code therefore needs a structural change to distinguish “hasn’t started” from “started but not finished. Even when a 200 response arrives on time, its answer can still be unusable.
Why HTTP 200 is not a quality signal for LLM responses
A 200 response with valid JSON can contain a hallucinated action, violate an expected schema, or present a well-formed but substantively wrong answer. None requires an exception or a non-2xx status. There is no Retry-After header for being confidently wrong. The conventional error-handling stack has no signal telling it to intervene.
In an agentic workflow, a hallucinated tool call or schema-violating response can produce real side effects before a human reviews it. The output needs a check after generation and before execution.
A single-model generation pass has the architectural limitation of lacking an independent adversarial check on its own output. HTTP error rates, p95 latency, and uptime dashboards cannot establish whether the answer is sound, so prompt drift or a checkpoint regression can degrade quality for hours without moving those metrics. Recovery patterns need an explicit source of evidence about the content itself.
Semantic quality gates: validating what a 200 response actually contains
Structural checks supply the cheapest first line of validation. Minimum response length, expected keys, and JSON parsability catch empty responses, missing fields, and obvious truncation before an output reaches downstream action. Rule-based heuristics can then flag known signs of failure, including apology phrases, refusal patterns, and answers to complex questions that are shorter than expected. These checks catch only recognizable defects.
For high-stakes outputs, the next layer is an LLM-as-validator. A binary YES/NO prompt sent to a small, cheap model provides a lightweight semantic check with latency and cost overhead low enough to be practical for most production workloads. Its purpose differs from schema validation. Instead of asking only whether the response has the expected shape, it checks whether the content meets the required condition.
The practical order is structural checks, heuristics, then model-based validation, stopping at the first failed check. For legal summaries, financial data, or medical information, the gate must distinguish acceptable recovery from an unsafe quality downgrade. A cached response or an honest user message may be preferable to silently switching models.
Multi-model fallback chains and the prompt compatibility trap
A single-provider integration makes provider incidents outages of its AI features. A tiered fallback chain supplies alternatives because another model can usually perform the same task, though at a different quality level. The chain can move from a primary model to a fallback on separate provider infrastructure, then another rate-limit pool, then a smaller on-premises model. A static or deterministic response, or human escalation, provides the final fallback.
Models handle system prompts, tool schemas, and structured output differently, so a prompt tuned for the primary can degrade on its replacement. An untested fallback is an untested code path scheduled for the worst possible moment. To make the chain dependable, teams should run their evaluation set against every fallback regularly and keep per-model prompt variants where quality differs materially. The quality restrictions for high-stakes outputs still apply during failover.
Adversarial multi-model review addresses a different part of the problem. Rather than waiting for failure to trigger a replacement, it runs multiple models in parallel against every output, comparing, critiquing, and voting on results before they reach the caller. Sequential fallback provides another route when one fails. Parallel review adds a check on the output even when the first route appears healthy.
Observability as the layer that catches what error handling misses
Patterns can respond only to signals they receive. Structured logging should capture the request ID, machine-readable error code, HTTP status, attempt number, retry delay, and whether the error was retryable. These fields turn isolated failures into queryable events. Passing correlation IDs through every hop connects downstream API trouble to the originating user action, making distributed debugging possible without guesswork.
LLM metrics must extend beyond HTTP health. TTFT distributions reveal waiting before generation starts, while schema and semantic quality gate failure rates expose rejected output. Per-model fallback trigger rates show when the primary route is being bypassed. Alerts should cover validation and semantic gate failures, while dashboards should expose half-open circuit states and gate failure counts from state machines.
Asynchronous jobs need a visible destination for exhausted retries. A dead letter queue preserves failed content jobs for manual inspection and replay. That extends recovery beyond the lifetime of a user-facing request and keeps unresolved failures available for investigation.
Observability rounds out the design by showing failures the patterns were not built to catch. A production-grade system needs both recovery mechanisms and evidence of their behavior at every layer, from transport errors through client/server faults to semantic failures. The final test is not whether the dashboard says the call succeeded, but whether the system can detect when that success was false.



