Monday, November 16, 2026
Cover illustration for “Circuit Breaker Patterns for Third-Party API Outages”
Writing for Technical FoundersCircuit Breaker Patterns for Third-Party API Outages

Circuit Breaker Patterns for Third-Party API Outages

Protect backends from cascading failures with strategic thresholds and bulkheads.

· 8 min read

Circuit breaker patterns protect a backend from third-party API outages by turning repeated waits into fast, predictable decisions. Their value depends on more than a switch that trips after errors. The three-state machine needs thresholds suited to the dependency, retries that do not hide failures, and bulkheads that stop waiting calls from swallowing shared resources.

Why a single slow API brings down an entire backend

A synchronous outbound call waits for a third-party response while holding a worker slot and, depending on the request, database connections and memory. As the API slows, more requests stay in flight, each holding resources that new requests need. The pool runs out before demand does. Unrelated endpoints then fail because they share the exhausted resources.

Daniel Dallas Okoye describes this cascade as user profile requests dropping while the server is paralyzed waiting for a third-party response. A fraud API can therefore disable much more than fraud checks. Payment gateways, SMS providers, and identity services present the same risk when calls wait synchronously. Catching a 500 and returning None handles the eventual error while leaving the worker occupied until then and the feature silently broken afterward.

What the three-state machine actually does

A circuit breaker wraps outbound calls and changes its behavior when failures cross a threshold. In Closed state, requests pass through while the breaker collects successes, failures, and latencies over a rolling window. The electrical metaphor fits, but developers still pay a small tax in words they have to learn.

In Open state, the breaker blocks calls immediately and returns a fallback without touching the network. Those requests free up the resources they would have held while waiting for the dependency. After a cooldown, Half-Open allows a controlled probe through to test recovery. A successful probe closes the breaker, while a failed probe opens it again. That controlled return to service distinguishes the breaker from a simple rate limiter and avoids flooding a dependency that is still fragile.

Viswamithra Vallabhaneni states, “Circuit breakers don't prevent failures. They prevent one failure from becoming everyone's failure.” The distinction sets the scope of the pattern. It protects the caller while leaving upstream repair to the service. The state machine itself is simple. The difficult decisions begin with which outcomes move it between states and how quickly those transitions happen.

The threshold decisions that determine whether the breaker actually works

A failure threshold is only useful if “failure” matches the dependency’s behavior. Server-side 5xx responses and connection exceptions, including timeouts, DNS failures, and ECONNREFUSED, count toward that assessment. Not every rejection means the API is down. Panker excludes 4xx responses, especially 429 rate-limit responses, from outage counting. APIs with non-standard response conventions may also need custom failure predicates to extend a generic HTTP rule.

Once failures are defined, the breaker needs a way to count them. A consecutive-failure threshold, such as 5 in a row, is simple and quick to trip but can overreact to a burst of errors in otherwise healthy traffic. A rolling failure rate, such as more than half of requests failing in the last minute, works better when traffic is high but may be slow to react when traffic is low. Neither approach wins by default. Okoye’s opossum example uses an error-threshold percentage with a fixed reset timeout, while Panker chose 5 failures per minute after tests showed the first threshold was too high.

Too short a wait sends probes into a service that has not recovered, producing repeated trips. Too long a wait leaves callers in fallback mode after the dependency is healthy. Recovery is not always a clean switch. Doubling the timeout after repeated Half-Open failures is one practical response to slow or intermittent recovery. Teams need to instrument and tune these settings against their traffic and upstream behavior.

Why retries without a circuit breaker make sustained outages worse

Retries give a transient fault another chance to clear, while a circuit breaker stops attempts during a sustained outage. Retrying during a sustained outage adds load just when the dependency cannot handle the requests it already has. Exponential backoff spaces attempts farther apart, but clients that fail together can still retry together. Jitter adds random variation to the delay so the recovering service does not receive a synchronized encore.

The retry decision also needs its own error rules. The retryable set includes 5xx responses, 408, network errors, and 429 with respect for Retry-After. By contrast, repeating 400, 401, 403, or 404 responses and data-validation failures does not change their cause. This differs from the breaker’s outage classification. A rate-limit response can justify a delayed retry without being counted as evidence that the whole API has failed.

The less obvious danger is retrying below the layer where the breaker observes results. The urllib3 configuration Retry(connect=3, read=3, total=3) can place retries beneath HTTP adapters, bypassing a breaker implemented at that layer. In the failure sequence in the notes, two intermediate 5xx responses are hidden by a third attempt that succeeds. The application sees only the 200. Its breaker records zero failures even though the upstream service has already shown signs of trouble.

A breaker calibrated to failed network attempts cannot work as intended if it receives only the final result of a retry sequence. Retries and the breaker need to be composed deliberately so failures reach the layer meant to count them. Otherwise, breaker metrics can look healthy even when the dependency takes several attempts to answer.

How bulkheads limit the blast radius a circuit breaker leaves exposed

A breaker needs evidence before it trips, and gathering that evidence consumes resources. Between the first slow call and the opening transition, enough concurrent requests can arrive to exhaust a connection pool. The breaker cannot close that gap alone. A bulkhead caps how many calls can reach one downstream service at once and queues or rejects excess requests to keep their use of shared resources within a bound.

The two patterns therefore control different dimensions of exposure. The breaker responds to failure history, while the bulkhead limits how much work can be tied up at once. Timeouts add a bound on how long an individual call can wait, and retries with backoff and jitter provide another chance for transient faults without synchronized load. A timeout does not stop the next call, a breaker does not bound pre-trip concurrency, and a retry does not make a sustained outage disappear.

Concrete implementation: a Redis-backed circuit breaker and Cockatiel in Node.js

A Redis-backed PHP/Laravel implementation makes the shared-state decisions explicit. Aasimghaffar stores state in circuit:external_api:state and a failure counter in circuit:external_api:failures. Each call checks the state first. If it is OPEN, the implementation skips the network request and returns fallback data immediately. That early branch is where the pattern saves the caller from another wait.

On failure, Redis::incr atomically increments the counter, and reaching the threshold causes Redis::setex to store OPEN with a TTL that serves as the cooldown. On success, the implementation deletes the failure counter. Redis’s atomic increment prevents concurrent workers from losing counter updates, which is essential when several workers contribute to the same failure history. The fallback replaces a 500 with the structured array ['status' => 'fallback_mode', 'data' => []], giving consumers an explicit degraded-mode signal.

The visible Redis structure exposes an important boundary in the example. It stores Open and Closed state and uses expiry for the cooldown, but the described code does not show a separate controlled Half-Open probe. Expiry is not the whole recovery protocol. The shared counter shows how workers track failures together, while the earlier three-state model shows what else a complete breaker needs to do as the service recovers. Its fixed Redis keys also make the scope of that shared tracking an architectural decision.

Cockatiel provides a composable approach for Node.js with a focus on typed code. The configuration halfOpenAfter: 10_000 makes the recovery delay explicit, while the retry policy uses 3 attempts with exponential backoff, a 128ms initial delay, and a 30s maximum. The composition is written as wrap(retryPolicy, shippingCircuitBreaker). Whether individual failed attempts reach the breaker or disappear behind the retry policy’s final result depends on this composition.

The same composition approach extends to bulkhead and timeout policies in the wrap chain. That makes the resilience stack visible in one place, but it does not remove the need to choose each policy’s behavior. Library choice is only part of the implementation. The 2026 library references include pybreaker for Python, resilience4j for Java/Kotlin, sony/gobreaker for Go, and opossum. Across those choices, the consequential decisions remain failure classification, counting, cooldown, and how the policies interact.

Per-service isolation: why one breaker per dependency is not optional

A shared breaker across independent APIs turns one service’s failure into a reason to block the others. A payment outage should not trip the breaker for address validation just because both use outbound HTTP. Each service that can fail on its own needs its own breaker instance, with thresholds that fit its traffic and the failure rate it can accept. In a Redis-backed design, the state and counter keys must preserve that separation rather than funnel every integration into the example’s fixed keys.

The same rule holds for internal services where some parts can fail and for dependencies whose latency varies and whose error rates are above zero. A critical service is not an exception just because no substitute exists. Fast failure still protects the calling system. Vallabhaneni describes the fallback in that case as “return 503 quickly”, avoiding a wait that lasts until the caller dies.

Observability and fallback design: the two things that make circuit breakers operational rather than theoretical

An Open breaker that produces an opaque error leaves operators guessing whether protection is working or the application has another bug. Its state transitions need to be visible. A Closed-to-Open event should include the service name and failure count, while the transition from Open to recovery probing should record that the cooldown elapsed. The probe result then explains whether the breaker closed or reopened. Aasimghaffar logs at emergency level when the threshold is reached, and that logged signal needs to trigger an alert.

Failure rate over time helps teams judge whether a breaker is responding to a real outage or nuisance-tripping because its threshold is poorly calibrated. One transition is not the whole diagnosis. The relationship between recorded failures and state changes is what makes tuning possible. Without that check, changing a threshold is like adjusting a smoke alarm because its noise is annoying.

Fallbacks need the same clarity for callers. Cached or localized data can stand in for the last known good response where staleness is acceptable. A structured degraded-mode response can identify the unavailable capability, allowing a caller to hide the shipping estimate and keep the checkout page working. No substitute means a fast 503 with Retry-After. A generic 500 hides that distinction and sends the on-call engineer looking for an application bug before discovering that the breaker is open.

Recovery deserves as much visibility as the initial trip. A Half-Open probe either restores normal calls or resets the cooldown, and those outcomes imply very different upstream conditions. The next operational step is to log which outcome occurred and when. Without that record, teams cannot see whether the service is recovering and may intervene manually at exactly the wrong moment.

Filed underAPI Reliability

More in API Reliability