Secrets Rotation Runbook

Overview

A database password that hasn’t changed in 400 days is an incident waiting for a timestamp. When a secret leaks — committed to git, pasted into a chat, left behind by a departing contractor — the response is the same: rotate it fast, without taking production down. This runbook covers that procedure end to end: building the inventory, rotating without downtime, automating rotation in AWS Secrets Manager and HashiCorp Vault, handling JWT signing keys, and running an emergency rotation under pressure.

For the strategy and tooling decisions behind these procedures, see the Complete Guide to Secrets Management.

1. Secret Inventory

Rotation fails when you don’t know what exists. Teams routinely discover a hardcoded API key in a legacy service only after the leaked credential shows up in a breach dump. Before rotating anything, build a complete inventory of every secret, its owner, and its rotation deadline.

1.1 Secret Types

Secret type	Rotation frequency	Storage
Database passwords	90 days	Secrets Manager / Vault
API keys (third-party)	90 days	Secrets Manager / Vault
JWT signing keys	30 days	Secrets Manager / Vault
TLS certificates	90 days before expiry	ACM / cert-manager
Encryption keys (KMS)	365 days	AWS KMS / Vault Transit
Service account tokens	30 days	Secrets Manager / Vault
SSH keys	180 days	Vault SSH CA
OAuth client secrets	90 days	Secrets Manager / Vault

Frequencies above are a sane default, not a universal rule. High-value secrets — production database credentials, signing keys, anything that can move money — deserve tighter windows. NIST SP 800-57 ties cryptoperiod length to the sensitivity of the protected data; a 90-day baseline for credentials and 30 days for signing keys is a defensible middle ground that most compliance frameworks accept.

1.2 Inventory Template

Secret ID	Type	Service	Owner	Last rotated	Next due
db-prod-pg01	DB password	payment-service	team-payments	2026-04-01	2026-07-01
api-stripe	API key	billing-service	team-billing	2026-05-15	2026-08-15
jwt-signing	JWT key	auth-service	team-auth	2026-06-01	2026-07-01
tls-api-prod	TLS cert	api-gateway	team-infra	2026-03-01	2026-06-01

A ready-to-fill CSV version of this inventory is included in the companion resources.

Every row needs an owner — a team, not a person. Orphaned secrets are the ones that get forgotten until an auditor or an attacker finds them. The Next due column feeds your alerting: anything past due should page the owning team, not sit silently in a spreadsheet. The Secrets Rotation Template gives you a ready-to-copy version of this inventory, and the Secret Rotation Schedule Template covers the calendar side.

2. Zero-Downtime Rotation Strategy

The naive approach — generate a new password, update it everywhere, restart — causes an outage window where old instances hold the old secret and new instances hold the new one. Zero-downtime rotation avoids that window by letting both secrets coexist during the transition.

2.1 Dual-Key Rotation

Phase 1 — Add new secret (no removal)

• Generate new secret (v2)
• Store v2 in secrets manager alongside v1
• Deploy application with support for both v1 and v2
• Application reads v2 for new operations, accepts v1 for existing

Phase 2 — Switch to new secret

• Deploy application to use v2 as primary
• Monitor for errors and authentication failures
• Wait 24-48 hours for stability

Phase 3 — Remove old secret

• Verify no service is using v1
• Delete v1 from secrets manager
• Log rotation completion in audit trail

The 24-48 hour stability window in Phase 2 isn’t optional — it’s where you catch the cron job that only runs nightly, the batch worker that authenticates weekly, and the integration nobody documented. Deleting v1 early is the most common cause of self-inflicted rotation outages. If anything still uses v1 when you remove it, you’ve just built your own incident.

2.2 Database Password Rotation

import boto3
import json
import psycopg2
import secrets
from datetime import datetime, timezone
from psycopg2 import sql

def rotate_database_password(secret_id: str):
    client = boto3.client('secretsmanager')

    # Step 1: Get current secret
    current = client.get_secret_value(SecretId=secret_id)
    current_creds = json.loads(current['SecretString'])

    # Step 2: Generate new password
    new_password = secrets.token_urlsafe(32)

    # Step 3: Connect to database with current credentials
    conn = psycopg2.connect(
        host=current_creds['host'],
        user=current_creds['username'],
        password=current_creds['password'],
        database=current_creds['dbname'],
    )

    # Step 4: Update password in database (ALTER USER)
    with conn.cursor() as cur:
        cur.execute(
            sql.SQL("ALTER USER {} WITH PASSWORD %s").format(
                sql.Identifier(current_creds['username'])
            ),
            (new_password,),
        )
    conn.commit()
    conn.close()

    # Step 5: Update secret in Secrets Manager
    new_creds = current_creds.copy()
    new_creds['password'] = new_password
    client.put_secret_value(
        SecretId=secret_id,
        SecretString=json.dumps(new_creds),
    )

    # Step 6: Verify new credentials work
    conn = psycopg2.connect(
        host=new_creds['host'],
        user=new_creds['username'],
        password=new_password,
        database=new_creds['dbname'],
    )
    conn.close()

    print(f"Rotation complete for {secret_id} at {datetime.now(timezone.utc)}")

Two details matter here. Step 4 uses psycopg2.sql.Identifier instead of f-string interpolation — identifiers can’t be parameterized, and in a security runbook you don’t model the unsafe version. Step 6 isn’t decorative: verify the new credentials actually connect before declaring victory. A rotation that updates the secrets manager but leaves a broken password is worse than no rotation.

3. AWS Secrets Manager Automatic Rotation

Secrets Manager rotates on a schedule through a Lambda function you provide. The managed rotation path handles most RDS, Redshift, and DocumentDB credentials without custom code; anything else needs a rotation function.

3.1 Configure Rotation

import boto3

client = boto3.client('secretsmanager')

# Enable automatic rotation — every 90 days
client.rotate_secret(
    SecretId='prod/payment-service/db-credentials',
    RotationLambdaARN='arn:aws:lambda:us-east-1:123456789012:function:secrets-rotation',
    RotationRules={
        'AutomaticallyAfterDays': 90,
    },
)

# Trigger immediate rotation (for testing or emergency)
client.rotate_secret(
    SecretId='prod/payment-service/db-credentials',
    RotateImmediately=True,
)

For RDS databases, prefer the alternating users strategy over the single-user strategy shown here. AWS keeps two user accounts (user and user_clone) and alternates which one is active, so the application never loses connectivity mid-rotation. The single-user strategy works but introduces a brief window where the old password is invalid and the new one isn’t yet live.

3.2 Lambda Rotation Function

AWS calls the function once per step: createSecret, setSecret, testSecret, finishSecret. The contract is strict — each step is idempotent and can be retried by Secrets Manager.

import boto3
import json
import secrets
import psycopg2
from psycopg2 import sql

def lambda_handler(event, context):
    secret_arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']

    client = boto3.client('secretsmanager')

    if step == 'createSecret':
        # Generate new secret version
        current = json.loads(client.get_secret_value(SecretId=secret_arn)['SecretString'])
        new_password = secrets.token_urlsafe(32)
        new_secret = {**current, 'password': new_password}
        client.put_secret_value(
            SecretId=secret_arn,
            ClientRequestToken=token,
            SecretString=json.dumps(new_secret),
        )

    elif step == 'setSecret':
        # Update database with new password
        pending = client.get_secret_value(
            SecretId=secret_arn,
            VersionStage='AWSPENDING',
            VersionId=token,
        )
        pending_creds = json.loads(pending['SecretString'])
        current = json.loads(client.get_secret_value(SecretId=secret_arn)['SecretString'])

        conn = psycopg2.connect(
            host=current['host'],
            user=current['username'],
            password=current['password'],
        )
        with conn.cursor() as cur:
            cur.execute(
                sql.SQL("ALTER USER {} WITH PASSWORD %s").format(
                    sql.Identifier(pending_creds['username'])
                ),
                (pending_creds['password'],),
            )
        conn.commit()
        conn.close()

    elif step == 'testSecret':
        # Test new credentials
        pending = client.get_secret_value(
            SecretId=secret_arn,
            VersionStage='AWSPENDING',
            VersionId=token,
        )
        creds = json.loads(pending['SecretString'])
        conn = psycopg2.connect(
            host=creds['host'],
            user=creds['username'],
            password=creds['password'],
        )
        conn.close()

    elif step == 'finishSecret':
        # Promote pending to current — find the AWSCURRENT version id first
        versions = client.describe_secret(SecretId=secret_arn)['VersionIdsToStages']
        current_version_id = next(
            vid for vid, stages in versions.items() if 'AWSCURRENT' in stages
        )
        client.update_secret_version_stage(
            SecretId=secret_arn,
            VersionStage='AWSCURRENT',
            MoveToVersionId=token,
            RemoveFromVersionId=current_version_id,
        )

The finishSecret step looks up the current version via describe_secret because the rotation event only carries SecretId, ClientRequestToken, and Step — reading event['CurrentVersionId'] would raise a KeyError and kill the rotation at the last step. Also note that psycopg2 isn’t in the Lambda runtime; package it as a Lambda layer or use psycopg2-binary in the deployment bundle.

4. HashiCorp Vault Rotation

Vault takes a different approach: instead of rotating a shared static credential, it issues short-lived dynamic credentials per request. A leaked credential expires on its own within the TTL — which shrinks the blast radius of any single leak dramatically. For implementation details, see the Vault Dynamic Credentials recipe.

4.1 Database Secrets Engine

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection — pull the admin password from the environment,
# never paste a literal credential into the config
vault write database/config/payment-postgres \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@db-prod:5432/payment?sslmode=require" \
    allowed_roles="payment-service" \
    username="vault-admin" \
    password="$VAULT_DB_ADMIN_PASSWORD"

# Create role with rotation
vault write database/roles/payment-service \
    db_name=payment-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# Application requests credentials (auto-rotated)
vault read database/creds/payment-service
# Returns: username=v-token-payment-123, password=auto-generated, lease_duration=1h

With dynamic secrets, “rotation” stops being a scheduled procedure — every lease renewal is a rotation. The operational work shifts to renewal and revocation: applications must handle credential expiry gracefully, and you must be able to revoke a lease prefix (vault lease revoke -prefix database/creds/payment-service) to invalidate an entire class of leaked credentials at once.

4.2 Transit Key Rotation

# Create encryption key
vault write -f transit/keys/payments

# Rotate the key (creates new version, old version still decrypts)
vault write -f transit/keys/payments/rotate

# List key versions
vault read transit/keys/payments
# latest_version: 2

# Encrypt with latest version
vault write transit/encrypt/payments plaintext=$(base64 <<< "sensitive data")

# Decrypt (automatically uses correct version)
vault write transit/decrypt/payments ciphertext=vault:v2:...

Transit keys version natively — vault:v2: ciphertext tells Vault which key version to decrypt with, so old data stays readable after rotation. Set min_decryption_version to cut off old versions once you’ve re-encrypted legacy ciphertext, or you’ll carry weak keys forever.

5. JWT Signing Key Rotation

JWT keys need the dual-key treatment with one extra wrinkle: the kid (key ID) header tells verifiers which key signed each token. During rotation, both keys must be published — typically through a JWKS endpoint — until every outstanding v1 token expires.

5.1 Dual-Key Strategy

import jwt
from datetime import datetime, timedelta, timezone

# Current and new signing keys
SIGNING_KEYS = {
    'v1': load_private_key('jwt-signing-v1'),
    'v2': load_private_key('jwt-signing-v2'),
}
CURRENT_VERSION = 'v2'

# Issue tokens with current key
def issue_token(user_id: str) -> str:
    payload = {
        'sub': user_id,
        'exp': datetime.now(timezone.utc) + timedelta(hours=1),
        'iat': datetime.now(timezone.utc),
    }
    return jwt.encode(
        payload,
        SIGNING_KEYS[CURRENT_VERSION],
        algorithm='RS256',
        headers={'kid': CURRENT_VERSION},
    )

# Verify tokens — accept both v1 and v2 during transition
def verify_token(token: str) -> dict:
    # kid lives in the JOSE header, not the payload
    kid = jwt.get_unverified_header(token).get('kid', 'v1')

    if kid not in SIGNING_KEYS:
        raise ValueError(f"Unknown key ID: {kid}")

    public_key = get_public_key(SIGNING_KEYS[kid])
    return jwt.decode(token, public_key, algorithms=['RS256'])

The kid is a JOSE header parameter, not a payload claim — read it with jwt.get_unverified_header(), not by decoding the payload. Tokens issued before your switch had kid in the header only, and looking for it in the payload is how “mystery” verification failures happen during real rotations. The overlap window should match your token lifetime: with 1-hour tokens, 24-48 hours of dual-key acceptance covers every outstanding token plus slack for clock skew.

6. Emergency Rotation

Scheduled rotation is hygiene; emergency rotation is incident response. When a secret leaks, the clock starts at discovery, not at your convenience. Follow your Security Incident Response Template for the communication and evidence-preservation work that runs alongside the technical steps below.

6.1 Emergency Triggers

Trigger	Action	Timeline
Secret committed to git	Rotate immediately, scan git history	< 1 hour
Secret exposed in logs	Rotate immediately, purge logs	< 1 hour
Secret shared via email/chat	Rotate immediately	< 1 hour
Suspected breach	Rotate all secrets in blast radius	< 4 hours
Employee departure with access	Rotate within 24 hours	< 24 hours
Third-party breach notification	Rotate affected API keys	< 24 hours

“Blast radius” is doing real work in that table. A leaked database password means rotating every secret that database credential could reach — read replicas, reporting users, the analytics pipeline that connects with the same account. Scope the blast radius before you start rotating, or you’ll close one door while three identical ones stay open.

6.2 Emergency Rotation Procedure

1. Identify compromised secret(s) and blast radius
2. Generate new secret(s) immediately
3. Update secrets manager with new value(s)
4. Trigger application reload / restart to pick up new secrets
5. Verify all services are using new secret(s)
6. Revoke / invalidate old secret(s) at the source (DB, API provider)
7. Audit access logs for misuse of old secret(s)
8. Document incident in security incident log
9. Notify affected teams and stakeholders
10. Schedule post-incident review within 48 hours

Step 6 is where teams slip: rotating in the secrets manager doesn’t revoke the old credential at the source. The old password still works against the database until you explicitly invalidate it. Rotate first, verify services are healthy, then revoke — in that order, or you’ll lock yourself out mid-incident.

7. Monitoring and Verification

A rotation isn’t done when the new secret exists — it’s done when every consumer proves it works. Add these checks to the rotation procedure itself, not as an afterthought.

Run this verification checklist after every rotation:

• All application instances reloaded the secret — check version labels in Secrets Manager or Vault lease counts, not just “no alarms”
• Zero authentication errors in the last N minutes per service
• Health checks pass with the new credentials (DB connect, API call)
• No service still authenticates with the old secret — query the source: pg_stat_activity for DB users, Vault lease list, provider dashboards
• Audit trail entry written: who rotated, when, which version, ticket ref

Two metrics tell you whether the program works: rotation coverage (percentage of inventoried secrets within their rotation window — target 100%, alert at anything overdue) and emergency rotation time (discovery to full revocation — the number you’ll be judged on after a real leak). Track both per secret type; the one you rotate manually is always the one that breaks the trend line.

8. Rollback Procedure

Sometimes rotation breaks a consumer you didn’t know existed. Decide the rollback path before rotating, while there’s no pressure.

Symptom	Response
Auth failures spike after Phase 2	Revert primary to v1, keep v2 stored; fix the failing consumer
Old secret already revoked	Restore v1 value at the source if possible; otherwise rotate again to a fresh v3 and deploy fix first
Single service broken, rest healthy	Pin that service to v1 via a service-specific secret path; don’t roll back the whole fleet
Lambda rotation failed mid-step	Check which step failed in CloudWatch; AWSPENDING version is safe to delete and retry

The worst rollback scenario is discovering you can’t go back — the old credential is revoked and a critical consumer only knows v1. That’s the argument for the 24-48 hour Phase 2 window: cheap insurance that turns “can’t rollback” into “noticed early.”

9. Kubernetes Secrets Rotation

Kubernetes Secret objects are base64-encoded ConfigMaps, not a secrets manager — treat them as a delivery mechanism, not a source of truth. The standard pattern is the External Secrets Operator syncing from AWS Secrets Manager or Vault into the cluster.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payment-db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/payment-service/db-credentials
        property: password

The refreshInterval controls how quickly a rotation propagates — but the synced Secret only updates the object. Pods still hold the old value until they restart, unless your application watches the mounted file or queries the API. For mounted secrets, kubelet updates the file on the sync interval (minutes), and the app must re-read it; for environment variables, only a pod restart picks up new values. Budget for a rolling restart controller (e.g., Reloader) or design apps to reload credentials. For mounting patterns, see the Kubernetes ConfigMap and Secret Mounting recipe.

10. Testing Your Rotation

A rotation procedure that’s never been run is a hypothesis. Test it the way you test backups: regularly, against the real path, before you need it.

• Rotate a non-production secret end-to-end monthly — same tooling, same runbook, not a simplified “test path”
• Kill an app instance mid-rotation and verify recovery behavior
• Measure time from “start rotation” to “all services on new secret” — that’s your real emergency rotation estimate
• Simulate a failed Phase 2: force an auth error, verify alerting fires
• Run the full emergency procedure quarterly as a game day

The first real rotation in production will surprise you — a consumer you forgot, a TTL that doesn’t reload, a Lambda timeout in setSecret. Better to meet those surprises on a Tuesday afternoon in staging than during a breach.

Common Mistakes

• Rotating the secret in the manager but not at the source. Secrets Manager is a store, not an enforcer — the old password keeps working until the database or provider invalidates it.
• Deleting the old secret before the stability window ends. The nightly cron and weekly batch job only appear when they fail. Wait the full 24-48 hours.
• Reading kid from the JWT payload. It’s a header parameter; payload claims are for the token body. Verify with get_unverified_header.
• Rotating without an inventory. You can’t rotate what you haven’t catalogued — the inventory in section 1 is the prerequisite, not paperwork.
• Hardcoding the rotation scripts’ own credentials. A Lambda that rotates secrets using an embedded admin password creates a new stale secret to fix the old one. Use IAM roles and environment references.
• Forgetting cron and batch consumers. They authenticate rarely, so they fail late — long after you’ve declared the rotation done.
• Treating Kubernetes Secrets as a secrets manager. Base64 is encoding, not encryption. Sync from a real secrets manager and restrict RBAC on Secret objects.

See Also

• Complete Guide to Secrets Management
• Secrets Rotation Template
• Security Incident Response Template
• Live Database Credentials with HashiCorp Vault
• Python JWT Refresh Token Rotation
• AWS Secrets Manager — Rotation
• HashiCorp Vault — Database Secrets Engine
• NIST SP 800-57 — Key Management Recommendations
• OWASP Secrets Management Cheat Sheet

Companion code: secrets rotation runbook resources — the rotation Lambda, the database rotation script, the Vault setup commands, and the ready-to-fill secret inventory CSV from this page.