Handling Third-Party API Rate Limits in Production

Rate limits get treated like a setting you tune: pick a retry count, add a sleep call, move on. That habit is the problem. A rate limit is a structural boundary, closer to a load-bearing wall than a speed bump, and teams that route around it with a retry loop find out the hard way that the wall was holding something up.

The naive retry loop doesn't just lose a request. It hits the provider again, harder, while the provider is already saying slow down. That extends the throttle window. The extended window triggers more retries. By the time anyone notices, the product sitting on top of the API call is what broke.

The rest of this piece works through why that happens and what actually holds up in production: distributed token buckets, backoff that respects what the provider tells you, circuit breakers, and routing across multiple providers.

How providers enforce limits

Before fixing anything, it helps to know what's actually being enforced, because it's rarely one number. Enterprise APIs stack multiple limiters on top of each other, and every provider signals the problem in its own dialect.

Stripe is the clearest public example of layering. Stripe runs at least four limiter types at once: token buckets for overall request rates, a separate limiter for concurrent requests, a fleet usage load shedder, and a worker utilization load shedder. Four different mechanisms, each watching for a different kind of overload, all running at the same time on the same traffic.

The big AI providers each enforce a different combination. OpenAI applies limits on requests per minute and tokens per minute together, with daily request and token caps mostly showing up on free-tier accounts and a handful of lower-tier model snapshots. Anthropic enforces requests per minute, input tokens per minute, and output tokens per minute. Google applies limits per project, with per-region limits layered on for some services like Vertex AI. None of these line up with each other, so a rate limiter built around one provider's shape will misread another's.

The signaling is where things get genuinely messy. Some providers return a clean 429 with a well-formed Retry-After header, telling the client exactly when to come back. Some return a 403 instead, with no timing information. Some return a 200 OK with the actual error buried in the response body. The request looks successful and the data quietly never arrives. That last one is worse than a rejection, because nothing in the response tells the client anything went wrong.

Even the header names refuse to agree. GitHub uses X-RateLimit-Reset as a Unix timestamp. Shopify uses Retry-After in seconds. HubSpot adds its own X-HubSpot-RateLimit-Daily-Remaining header on top of the standard ones. Three providers, three different ways of saying the same thing, none of them interchangeable with code written for another.

Coupa sits at the far end of the spectrum. It caps requests at 25 per second, with a burst queue whose depth isn't published, enforces that limit at the instance level, and returns no rate-limit headers whatsoever. There's nothing to parse and nothing to infer from. A team integrating with Coupa has to build its own backoff and circuit breaker logic from scratch, because the provider hands over no signal to build on.

When these limits stack, a single endpoint can be governed by daily, per-second, per-endpoint, and burst limits at once; the response rarely says which one just got tripped. A handler that assumes uniform behavior across providers will work for exactly one of them and fail, in different ways, for the rest.

Why horizontal scaling breaks naive rate limiting

A rate limiter that keeps its state in one process works fine until a second process shows up. Adding a second application server breaks the limiter's protection entirely, because it has no idea the other server exists.

Picture ten application servers, each running its own in-memory token bucket. Each one believes it has the full quota available, because each one only sees its own requests. Multiplying that false confidence across ten servers makes the actual outbound request rate hitting the provider ten times what any single server thinks it's sending. The provider doesn't care how the traffic was distributed internally. It only sees the total, and the total blows past the limit every time.

The fix is to move the bucket out of each process and into shared state. Redis is the standard choice here, mainly because it supports atomic operations: a Lua script can check the bucket and decrement it in a single, indivisible step, which closes the race condition where two servers both check the bucket, both see room, and both fire a request that together exceeds the quota.

That bucket needs to be keyed per customer and per provider, not per application instance. Without that distinction, one customer burning through their quota throttles every other customer sharing the same integration, which turns a single noisy tenant into an outage for everyone else on the platform.

AI agent workloads make this worse. An agent running a multi-step reasoning loop produces traffic that's naturally spiky and bursty, call, pause, call again, in a pattern that looks a lot like scraping behavior from the outside. The provider's limiter treats a legitimate agent doing its job the same way it treats something scraping the API.

A pattern from a NetSuite integration case shows the fix in practice: use a per-tenant semaphore, sized to leave 30 to 40 percent of headroom for whatever else that customer's integrations are doing, and wrap the semaphore around a whole logical operation. That distinction matters, because an operation that spans several requests needs to reserve capacity for all of them, not just the first one.

A distributed bucket stops the overrun at the source. It doesn't decide what happens after a request gets throttled anyway, and that's where backoff strategy takes over.

Backoff that works: why jitter and Retry-After headers are non-negotiable

Getting throttled is normal. What a system does in the next few seconds determines how quickly that throttle clears or whether it turns into an extended outage.

Fixed-interval retry is the mistake most systems start with, and it's the one that makes things worse. If a rate-limit error fires across many workers at roughly the same time, and every one of them waits the same fixed delay before retrying, they all come back at the same instant. That synchronized wave hits an endpoint that's already struggling harder than the original traffic did, extending the throttle.

Exponential backoff with full jitter desynchronizes the retries. Each worker picks a randomized wait time within a growing window, spreading retries out across time. The provider sees a trickle of recovery traffic.

When a provider includes a Retry-After header, that value overrides whatever backoff calculation the client would otherwise run. The provider is stating, directly, when it will accept traffic again, and that instruction beats a guess every time. Backoff math exists for the providers that give nothing else to go on, Coupa being the clear example from the previous section.

An OpenRouter-based academic pipeline shows the pattern applied consistently across six different LLM providers, spanning a range of model sizes and vendors. Each one got exponential backoff capped at five retries with a bounded maximum delay, applied the same way regardless of which model was behind the call. Consistency across architectures mattered more than tuning each provider individually.

Backoff strategy also needs to know what kind of limit it just hit. A per-second burst limit calls for a retry within seconds. A daily quota exhaustion calls for waiting until midnight UTC, or handing the work off to a backfill job that runs later. The response itself rarely says which tier got tripped, so the handler has to infer it from context, usually the specific limit values returned alongside the error.

Circuit breakers as a containment layer when backoff alone is not enough

Backoff handles one request's recovery path. It has nothing to say about what happens when the provider is down for an extended stretch and every new request runs the full retry cycle anyway, accumulating latency and tying up resources the whole time.

A circuit breaker sits in front of that problem with three states. Closed means requests flow through normally. Open means requests fail immediately, without ever reaching the provider, because the breaker has already decided the provider is unhealthy. Half-open means a single probe request goes out to test whether the provider has recovered, before the breaker commits to reopening fully. That transition logic, closed to open to half-open and back, is what separates an actual circuit breaker from a boolean flag someone set in a config file.

Without that layer, a provider outage spreads upstream on its own. Background workers crash under the retry load, which delays unrelated jobs that had nothing to do with the failing provider and back-pressures the API those workers feed. Database connections stay open longer than they should. Memory gets consumed. CPU cycles get burned retrying calls that were never going to succeed during the outage window. None of that is hypothetical: it's the direct, documented consequence of retry loops that don't respect backoff.

Coupa is the case that makes the circuit breaker non-optional. Twenty-five requests per second, no headers, no standard signaling, enforced at the instance level. There's no programmatic way to ask Coupa when it'll be ready again, so the circuit breaker has to make that call on its own, based on observed failure patterns.

Resilience4j is a commonly cited reference implementation for this pattern in production systems. The logic can live outside application code. NGINX and API gateway configurations can enforce circuit-breaking before a request ever reaches the backend, which keeps the containment layer outside the service that would otherwise bear the load.

The rate limiter's own backing store needs the same kind of failure planning. If Redis becomes unreachable, the system has to pick a default: fail open, allowing requests through, which makes sense for public APIs where denying legitimate users is the worse outcome, or fail closed, rejecting requests, which makes sense for internal systems protecting something critical. That choice has to be made in advance, per system, before an incident happens.

Circuit breakers contain the damage from one provider going down. Circuit breakers don't do anything about the quota ceiling that appears when a single provider simply isn't enough traffic for the job; that's a routing problem, not a containment one.

Multi-provider routing as a quota-sharing strategy, and the prompt consistency problem it introduces

Spreading requests across multiple providers multiplies the effective quota available to a pipeline. Combining the limits of several providers lets the system push more throughput than any one of them would allow alone. That's a real architectural strategy, not just a fallback for when the primary provider is down.

OpenRouter is the clearest example of this working as infrastructure. It offers a single API surface over more than 400 models from over 70 providers, with load balancing that weights routing decisions by price, built into one layer as quota-sharing.

The catch is that providers aren't interchangeable just because they answer the same API shape. Different models respond differently to the same prompt. A prompt tuned for one model can produce output that's structurally fine, valid format, right fields, but reads wrong: off brand voice, a different hallucination rate, phrasing that technically parses and still sounds off. Routing around a rate limit by switching providers mid-pipeline can quietly swap in a worse answer that looks identical on the surface.

Schema validation won't catch this, and neither will an automated style check. The actual prose underneath can drift even while structure passes clean. A pipeline that switches models to dodge a quota ceiling, without adapting the prompt for the model it switched to, introduces a quality variance that standard monitoring has no way to see.

Content production pipelines feel this from both directions at once. Lean on a single model and the pipeline hits a quota ceiling, plus the output risks becoming watermarkable or recognizable as machine-generated at scale. Spread the load across multiple models without adapting for each one, and the quota problem goes away while voice consistency falls apart instead.

One structural answer is an adversarial multi-model setup: several models draft, critique each other's output, and vote before anything is finalized. That spreads the quota load across the whole council of models, so no single provider's limit becomes the bottleneck, and it puts voice consistency in the hands of the critique and voting step instead of assuming any one model will produce the same voice call after call. The same mechanism keeps any single provider's limit from becoming the bottleneck while also stabilizing voice consistency.

Observability as a precondition for any of this working in production

None of the patterns above are worth much without visibility into whether they're actually doing their job. A distributed token bucket, a circuit breaker, a backoff routine, every one of them can quietly fail while looking fine from the outside, until a customer reports something broken and the postmortem starts from zero.

The minimum instrumentation needed covers per-policy hit rates, which keys are getting rate-limited most often, sampled logs of the actual throttling decisions, and correlation between those events and latency or error rates elsewhere in the system. That last piece matters because a rate limiter can be tuned too aggressively and start blocking legitimate traffic, which is its own failure mode and one that's easy to miss if the only thing being tracked is whether abuse is getting stopped.

Prometheus and Grafana are the standard open-source combination for this kind of visualization, showing request rates per endpoint, how often 429s are showing up, and how quota utilization trends over time.

For pipelines running across multiple providers, the gap that matters most is knowing which provider's limit was hit, which customer's quota ran out, and whether a given retry actually succeeded or the request was lost along the way. Without that per-tenant, per-provider breakdown, debugging an incident means piecing the call graph back together from logs after the fact, which is slow and often incomplete.

Long-running jobs, scrapes, content batches, anything that runs for a while and consumes quota steadily, need checkpointing against durable state. A mid-run 429 that forces the whole job to restart from the beginning is an observability gap: the system didn't know where it had gotten to, so starting over was the only option it had left. Progress checkpointing closes that gap, turning a rate limit interruption into a pause.