API Key Rotation Without Downtime

Static API keys are the default choice because they're easy, but that convenience comes at a cost: they're the most common single point of failure in production credential breaches. This article is about the engineering pattern for rotating them without knocking anything offline: overlapping validity windows, staged rollover, and secret propagation that doesn't require a deploy.

The scale of the exposure problem is worth sitting with for a second. GitGuardian's State of Secrets Sprawl 2026 report found roughly 29 million secrets detected in a single year, a 34% jump from the year before and the largest single-year increase on record. Of those exposed secrets, GitGuardian's 2025 report found API keys made up 43% of everything found sitting in public repositories, by far the biggest category.

Here's the part that should actually keep someone up at night: exposure doesn't close on its own. Nearly 70% of credentials confirmed valid in 2022 were still valid as of January 2025, and by January 2026, that number had only dropped to just above 64%. Pair that with IBM's 2025 Cost of a Data Breach Report, which puts median detection time at 204 days, and you get a leaked key that often outlives its own rotation schedule. The Trello incident touched over 15 million users, and a GitHub exposure revealed close to 13 million API secrets — both from 2024, both tracing back to one static credential each.

None of that is really in question anymore. The harder question, and the one this piece actually answers, is how you rotate a key that's already load-bearing in production without breaking the thing it protects.

What actually breaks during a naive rotation — and why

Ask most engineering teams why a key that everyone knows is stale hasn't been rotated yet, and the answer is rarely "we forgot." Revoking a widely used key feels riskier, right now, today, than the vague future risk of leaving it alone. That trade-off is backwards, but it's understandable, because the failure modes are real and specific.

Incomplete inventory. A key doesn't live in one place, and that's the whole problem. It's in a backend service, a CI pipeline, a cron job somebody wrote two years ago, a frontend build, maybe a partner's integration nobody remembers approving. Rotate without mapping every location first, and you get partial failures that are miserable to trace, because half the system works fine and the other half is quietly throwing 401s.

Revoke-before-verify. This is the classic, where someone creates the new key, feels good about it, and kills the old one before actually confirming the new one works against production. One failed verification step later, there's an outage with no clean way back.

Deployment-gated secrets. If the app only picks up a new key on deploy because it's baked into a config file, rotation is now hostage to the release pipeline. That's slow and fragile, and it means a security task now carries the risk profile of a full deploy.

Big-bang cutover. Flip every consumer to the new key at once, and there's no safety net if even one of them hasn't caught up.

Each of those is a design gap, not a discipline problem, and nobody's being careless. The tools and the architecture just weren't built to make rotation safe, which is exactly the hole the rest of this piece fills in.

The overlap window: the foundational principle behind every zero-downtime pattern

One rule sits underneath every pattern that follows: generate and deploy the new key first, and don't touch the old one until you've confirmed the new one actually works. Downtime happens at exactly one moment, when a key gets disabled before every consumer has switched over, and the overlap window is the deliberate gap engineered to prevent that moment from ever arriving.

How long should that window run? A 24 to 48 hour window covers most internal service propagation cycles, but the real floor is set by your slowest consumer's realistic update time, not your own schedule. For JWT and JWKS rotation specifically, the window can be scoped tighter, sized to the access-token TTL plus a safety buffer, because what matters there is token expiry, not service-level propagation.

What the window buys you: time to watch log traffic split across both keys, confirmation that migration actually finished before anyone commits to revocation, and the ability to roll back cleanly if the new key fails validation somewhere unexpected. This is the mechanism, and the next few sections are just that mechanism, applied differently depending on who controls the key and how many consumers are on the other end.

Step-by-step: the dual-key rollover for a single service you control

This pattern is for the cases where one team owns both ends: internal service-to-service keys, a self-managed API, anything where nobody outside the org needs a memo.

Step 1, generate. Create the new key, and leave the old one alone for now, since touching both at once is how mistakes happen.

Step 2, push to the secret store. Write the new key into the secrets manager or runtime injection layer, not into a config file, and the app should pick it up on its next request, no deployment involved.

Step 3, verify before doing anything else. Run a health check or smoke test against production with the new key. A failed check is a hard stop, full stop, not a "we'll deal with it later" warning.

Step 4, open the overlap window. Both keys are valid at the same time now, so watch the logs to see which key each consumer is actually using.

Step 5, confirm migration. Keep watching until the old key's request count drops to zero, or close enough to zero that it's just noise.

Step 6, revoke. Only after confirmation, since revocation is the one step you can't undo, and it deserves either a human's explicit go-ahead or an automated process with a real assertion behind it, not a cron job on a timer.

There's a variant worth knowing: code the client to try the new key first and fall back to the old one if that fails. It closes the one remaining gap, the brief window where propagation lag could otherwise cause a failed request. None of this works, though, if the application reads its credentials from a file baked at build time, and if that's the setup, fix the architecture before touching the rotation schedule.

The atomic roll-key operation: when the provider gives you a single endpoint for the whole swap

Some providers hand you a single "roll" endpoint that creates the new key and schedules the old one's expiry in one atomic call. It's a smaller surface for error: there's no gap between "created the new key" and "remembered to flag the old one," because the provider handles the overlap internally instead of leaving two keys for a human to track by hand.

It's not a full solution on its own, though, since the new key still has to reach every consumer before the provider's grace period runs out, and atomicity on the provider's end does nothing to speed up how fast your consumers actually update. This works well when the provider supports it and the consumer list is short enough to update inside that grace window. Once the consumer base gets big and you need actual visibility into who's migrated and who hasn't, this approach runs out of road, and that's where the phased method below takes over.

Phased migration for APIs with a large external consumer base

A single overlap window falls apart once you can't personally verify that every external consumer has switched over. Revoke on a fixed date and you will catch stragglers, guaranteed, because someone always misses the memo.

Phase 1, generate. New keys for every consumer, with old keys left untouched.

Phase 2, notify. Send migration notices with a real deadline attached, plus a direct link to the new key in the developer portal.

Phase 3, monitor adoption. Track traffic per consumer against old key versus new key. A dashboard here saves someone from running the same log query fifty times a week.

Phase 4, deprecate selectively. Set expiration dates only for consumers you've confirmed have migrated, and everyone else stays on the old key until they get handled one by one.

Phase 5, revoke, then follow up individually. Revoke after the grace period ends, and reach out to stragglers before the deadline hits, not after the fact when it's already too late to help them gracefully.

What this buys over a hard cutover is visibility: you know exactly who's moved and who hasn't, and that record is what lets you defend the decision to revoke when someone asks why their integration broke. The grace period itself should come from the adoption data in Phase 3, not from whatever date looked clean on the calendar.

JWKS rotation for JWT-issuing services: using key IDs to make token validation overlap-aware

JWT rotation plays by different rules, because the key isn't handed to a caller per request the way an API key is. It signs tokens at the moment they're issued and verifies them later, whenever they get used, which means both the old and new key have to stay valid until every token signed with the old one has actually expired.

The header field doing the real work here is kid, the key ID, and it's what makes the whole handoff possible. A properly configured JWT carries the kid of whichever key signed it, right there in the header, and verifiers use that field to look up the exact right key, instead of guessing by trying keys in order, which is fragile and depends on an ordering nobody guaranteed would stay stable.

The sequence: add the new key to the JWKS endpoint, keep the old one published right alongside it, then switch the issuer over to signing new tokens with the new key. Old tokens still verify fine, because the old key hasn't gone anywhere. The overlap window here equals the access-token TTL plus a safety margin, and the old key only comes off the JWKS endpoint once its longest-lived signed token has actually expired. During rotation, the JWKS should always be publishing at least two keys, the active signer and the one before it, and you should never prune down to a single key mid-rotation. It's the same overlap principle from earlier, just measured in token lifetime instead of how fast a service can update.

Runtime secret injection: the architectural prerequisite that makes rotation work

None of the patterns above work if the app reads its API key from a config file baked in at build time. That setup ties rotation to a deployment, which makes the whole thing slower, more fragile, and coupled to release risk that has nothing to do with the key itself.

Runtime injection means the app fetches its credentials from a secrets manager at startup or on each request, so a rotation updates the store, not the binary. Three tools are worth naming here.

AWS Secrets Manager encrypts with KMS, gates access through IAM, and has built-in Lambda-based rotation for RDS, Redshift, and DocumentDB, priced at $0.40 per secret per month. For credentials outside those native integrations, rotation logic requires a Lambda function.

HashiCorp Vault works differently by design: it generates short-lived credentials on demand and revokes them the moment the lease expires, rather than rotating a long-lived secret on a schedule. For static-key targets, Vault's auto-rotation policies (rotate every N days) do roughly what the dual-key pattern does, just automatically. The dynamic model is the real differentiator, though, since rotation becomes continuous instead of something scheduled on a calendar.

Kubernetes and the External Secrets Operator sync values from an outside provider into native Kubernetes secrets, and the refreshInterval setting controls how often the operator checks for changes. Rotate at the provider, and it reaches every pod without a redeploy.

The distinction between AWS Secrets Manager and Vault matters: AWS swaps one long-lived secret for another on a timer, while Vault's dynamic secrets simply stop existing when the lease runs out. Teams already on AWS rotating scheduled credentials fit the first model fine, while teams that need genuinely short-lived, per-service credentials at scale want Vault's dynamic approach instead.

Rotation frequency by key type: matching the schedule to the blast radius

Rotation frequency should track the damage a compromised key can do, not some flat rule applied to every credential equally. A leaked payment processor key and a leaked analytics key are not the same emergency, so the interval shouldn't be the same either.

• Production API keys (payment processors, cloud providers): every 30 to 90 days. Highest blast radius, and the first thing an auditor checks.
• Database credentials: 30 days for static secrets, or dynamic secrets with TTLs measured in hours. This is the category that gains the most from Vault's ephemeral model.
• Third-party SaaS keys (analytics, email, monitoring): every 90 days.
• Internal service-to-service tokens: mutual TLS with certificate rotation every 90 days, or short-lived JWTs where the TTL does the rotation work without anyone touching it.
• CI/CD pipeline tokens: every 30 days. Broad access, scattered across multiple systems, high-risk despite feeling like background plumbing nobody thinks about.

Alongside the fixed schedule, event-driven revocation needs to sit as its own trigger: a contractor leaves, a secret turns up in a commit or a support ticket, a service gets replatformed, monitoring flags API usage that doesn't look right, and any of those should trigger an immediate rotation, calendar be damned. That 204-day median detection time from IBM's report is the strongest argument against long rotation intervals that exists. If detection takes seven months on average, a 30-day rotation caps an attacker's window at one rotation cycle, even in the worst case where the compromise never gets noticed at all.

Building the rotation into the team's normal workflow rather than treating it as an incident

Rotation stops being scary the moment it stops being an event. Treat it like a deploy, something that happens on a schedule with a known procedure and a rollback plan, and nobody has to have a meeting about it. Treat it like an incident, something that only happens when someone's already worried, and it'll keep getting pushed to next quarter, forever.

The mechanics covered above, the overlap window, the dual-key rollover, the phased plan for external consumers, the kid-based JWKS handoff, only work if the architecture underneath supports them. That means runtime secret injection instead of baked config, a real inventory of where every key lives, and a rotation calendar that matches interval to blast radius instead of picking one number and using it everywhere. Once that's in place, rotation is a routine task with a checklist, not a weekend project that requires everyone to be on call. The keys were never the hard part; the plumbing underneath them was.