Webhook Reliability in Production Systems

Webhooks power nearly every modern software integration, from payment confirmations to shipping updates to user authentication events. When they work, they are invisible infrastructure. When they fail, they fail quietly, and that silence is the core problem. Most engineering teams build webhook consumers assuming the happy path: the event arrives, the handler processes it, the response goes back. But production traffic is not the happy path. It is race conditions, duplicate deliveries, silent failures hiding behind 200 OK responses, and cascading retry storms that turn a single provider hiccup into a platform-wide outage. The gap between a webhook system that works in staging and one that holds up under real production load comes down to a handful of specific engineering decisions. Understanding what actually breaks, and why, is the first step toward building something that does not.

What production data reveals about real failure rates

Nearly 20% of webhook deliveries fail silently during peak load. Not with an error. Not with a red banner in a dashboard. The provider sends the request, receives a 200 OK, marks the delivery successful, and moves on. Meanwhile, the database write, the invoice update, or the shipment status change never happened. Nobody logged an error because from the provider's point of view, nothing went wrong.

The broader trend reinforces this. Average API uptime slipped from 99.66% to 99.46% between Q1 2024 and Q1 2025, which translates to 60% more total downtime year over year. A 0.2% failure rate sounds negligible until you are running a million daily events, at which point it is two thousand broken transactions.

Carrier APIs illustrate the pattern well. Across eight major carrier platforms tracked over 30 days in late 2024, EasyPost's webhook delivery success rate dropped to 94.2% during European peak hours, with 3.8% of those failures returning confirmed silent failures. ShipEngine landed at 96.7% success but published no retry policy, leaving consumers with no reliable way to know what happens when delivery fails. Only 73% of services offer any retry mechanism at all, and many of those offer exactly one attempt. A single transient timeout on a service with no retry policy is not a recoverable hiccup. It is permanent data loss.

Black Friday 2024 made the stakes concrete: 58% of users hit technical issues that rippled through webhook-dependent systems, and three platforms suffered total webhook outages lasting two to six hours. That is not bad luck. It is the predictable outcome of under-engineered delivery infrastructure exposed on the one day everyone is watching.

The four failure modes that cause most outages

Timeout-induced silent failure. GitHub, like most providers, considers a delivery failed if the server takes more than 10 seconds to respond. That window disappears quickly when a handler is performing database writes, calling downstream services, and running business logic synchronously. Miss the timeout, and the provider assumes failure and retries, even if the work completed a moment too late. Worse, if the server crashes after completing the job but before sending the acknowledgment, the provider retries an event that already processed, creating a duplicate.

Retry storms. When a provider recovers from its own outage or runs a bulk job, a burst of events arrives at once. If the consumer is already under strain and begins failing, immediate retries double the incoming traffic on top of a system that is already overloaded. More failures trigger more retries, which cause more failures, in a loop that never allows the server to recover.

Duplicate delivery. Any system that guarantees delivery must retry when it is uncertain whether the first attempt succeeded. Network latency alone, a slow acknowledgment with no other fault, is enough to trigger a resend even when the original delivery worked. A consumer that treats every arrival as unique will double-charge customers, create duplicate records, or apply credits twice.

Authentication and schema breakage. Providers change things on their schedule. UPS moved to OAuth 2.0 in August 2025, and every downstream consumer had to update credential handling or start failing. Strict schema validation that throws on an unrecognized field is among the most common causes of pipelines breaking the day after a provider ships a new API version. Token rotation, HMAC secret changes, and new payload fields are routine on the provider's side, and routine outages on yours without proper handling.

Acknowledge fast and process separately

The fix for timeout failures is straightforward to describe but requires discipline to build consistently. The HTTP handler has one job: verify the signature, save the raw event, and drop it on a queue. Nothing else. No database writes, no downstream calls, no business logic inside that request path.

This allows the handler to acknowledge the webhook in under a second regardless of how backed up processing is. The provider receives its 200 OK and stops its retry clock. A separate background worker picks the event off the queue and performs the real work, outside the pressure of a ten-second window.

The benefits compound. Timeout risk from slow processing disappears because the handler was never doing the slow work. Traffic spikes are absorbed by the queue rather than dropped at the connection level. The receiving layer and processing layer scale independently, so adding workers to clear a backlog does not require touching the endpoint.

Signature verification must stay synchronous and run before the event touches the queue. Moving it to the background means building an efficient pipeline for processing forged payloads. On the schema side, processors should log a warning on a missing field and ignore an unrecognized one rather than throwing an exception. Providers add fields regularly, and a system that rejects unfamiliar data will break on every provider update.

Retry logic that does not make failures worse

Immediate retries at fixed intervals are how most retry-induced outages start. Hammering an endpoint that is already failing doubles the load on it and delays recovery.

Exponential backoff with jitter is the standard pattern. The delay grows exponentially with each attempt, something like immediate, then 30 seconds, 2 minutes, 8 minutes, 32 minutes, 2 hours, 8 hours, with a random jitter component added to each interval. The jitter matters as much as the exponential growth. Without it, every failed client retries at the same intervals, creating synchronized waves of traffic that recreate the thundering herd problem on a delay. Full jitter, randomizing the entire delay window rather than nudging a fixed value, spreads load across time. The maximum delay should be capped; eight hours is sufficient.

Not every failure warrants a retry. Retry on 5xx errors, connection failures, 408 Request Timeout, and 429 Too Many Requests. Most 4xx responses indicate a problem with the request itself, and resending the same request will not fix it. When a 429 includes a Retry-After header, that value overrides the backoff schedule entirely. Ignoring it risks getting blocked at the infrastructure level, which is a more severe outcome than a delayed delivery.

Set a hard cap on retry attempts before routing to a dead-letter queue. Retrying indefinitely does not look like resilience; it looks like a persistent failure that nobody noticed.

Circuit breakers to stop cascading failures

Retries alone leave a gap: a consistently failing endpoint continues consuming worker capacity on every attempt, even when there is no realistic chance of success. That is capacity unavailable to healthy endpoints.

Circuit breakers address this through three states. Closed is normal operation, with requests flowing through and failure rates monitored against a threshold. Open activates once that threshold is crossed, rejecting requests immediately and routing events to the retry queue, giving the failing endpoint time to recover rather than absorbing continued traffic. Half-open is the recovery probe, allowing a single test request through after a cooldown period. A success closes the circuit; a failure reopens it.

In multi-tenant systems, circuit breakers must be scoped per tenant and per endpoint, not applied globally across a provider. A single breaker covering an entire provider means one customer's broken integration can halt webhook processing for every other customer on the same provider.

The performance difference is measurable. Data from Black Friday 2024 found that centralized webhook services took 15 to 45 minutes to recover from an outage, while distributed systems with per-endpoint circuit breakers recovered in 3 to 8 minutes.

Idempotency as the defense against duplicate delivery

At-least-once delivery is not a design flaw. It is the cost of guaranteeing that events are not silently dropped. Idempotency is what makes that guarantee safe to accept.

The pattern is to store the provider's event ID before processing begins, check for that ID before any write or downstream call, and skip processing entirely if the event has already been handled. Without this, a retried payment event can double-charge a customer, generate a duplicate invoice, or apply a credit twice to the same account.

Doing it correctly requires a few specific things. The event ID must be stable and provider-assigned, not a timestamp or a derived value, either of which can collide or shift across retries. The check-and-store step must be atomic, a single transaction rather than a check followed by a separate write, because a gap between the two creates a window for race conditions. The idempotency record must be persisted to durable storage; an in-memory store resets on every deployment, which defeats the purpose.

Idempotency should also extend to the downstream calls the processor makes, not just the initial event receipt. If processing is interrupted midway and resumes, operations that already completed should not fire again.

Dead-letter queues for recovery not just logging

Without a dead-letter queue, an event that exhausts its retry budget simply disappears. No record, no audit trail, no path to recovery.

A DLQ for payment events should store the full payload, the error details, the retry count, and a status field for tracking investigation. Retention should be longer than the main queue; 14 days in the DLQ versus 4 days in the main queue gives teams room to investigate before records expire. Events should be archived to cold storage before deletion so that the record is available for compliance review and for issues that surface weeks after the fact.

A DLQ only provides value if it functions as an active workflow rather than a passive bin. Arrival in the DLQ should trigger an alert, not a buried digest email. Events should be assignable to a specific person for investigation. There must be a mechanism to replay events manually once the root cause is resolved, otherwise the queue is a filing cabinet with no retrieval process.

The retry count before DLQ routing matters too: 3 to 5 automatic attempts with exponential backoff is the appropriate range. Fewer risks sending recoverable failures to the DLQ prematurely. More delays recognition that a real problem exists. For payment systems, a missing DLQ is not just a reliability gap but a compliance gap.

HMAC verification and replay attack prevention

An endpoint that does not verify signatures accepts whatever arrives at its URL, including fake payment confirmations, spoofed order updates, and invented events that never occurred on the provider's side.

HMAC-SHA256 is the standard approach, used by Stripe, GitHub, and Shopify. It is fast, cryptographically sound, relies on a shared secret key, and has library support in every major language.

The verification steps must happen in a specific order. The raw request body must be captured before any JSON parsing, because a parser can reformat whitespace or reorder keys in ways that invalidate a valid signature. Recompute the HMAC using the shared secret, then compare it using a constant-time comparison function rather than standard string equality. Timing differences in a naive comparison can leak information about whether the signature is close to correct. If the signatures do not match, reject the request without processing it.

A valid signature does not confirm that a request is fresh. A legitimate request can be captured and replayed later. It will pass signature verification every time because the signature has not changed. The defense is a timestamp embedded in the payload and a tolerance window, typically a few minutes. Requests outside that window are rejected regardless of signature validity. Stripe's implementation includes exactly this approach, and it is worth replicating, because signature verification without replay protection leaves the door open to a straightforward attack.