Runbook de Rotacion de Secrets

Overview

Este runbook cubre procedures para rotar secrets sin downtime. Secrets incluyen API keys, database passwords, TLS certificates, JWT signing keys y encryption keys. Stale secrets increasean el risk de credential compromise. Este documento cubre secret inventory, rotation schedule, zero-downtime strategies, automated rotation y emergency procedures.

---

1. Secret Inventory

1.1 Secret Types

Secret type              | Rotation frequency | Storage
─────────────────────────┼───────────────────┼──────────────────────────
Database passwords       | 90 days            | Secrets Manager / Vault
API keys (third-party)   | 90 days            | Secrets Manager / Vault
JWT signing keys         | 30 days            | Secrets Manager / Vault
TLS certificates         | 90 days before exp | ACM / cert-manager
Encryption keys (KMS)    | 365 days           | AWS KMS / Vault Transit
Service account tokens   | 30 days            | Secrets Manager / Vault
SSH keys                 | 180 days           | Vault SSH CA
OAuth client secrets     | 90 days            | Secrets Manager / Vault

1.2 Inventory Template

Secret ID    | Type        | Service         | Owner       | Last rotated | Next due
─────────────┼─────────────┼─────────────────┼─────────────┼──────────────┼──────────
db-prod-pg01 | DB password | payment-service | team-payments| 2026-04-01   | 2026-07-01
api-stripe   | API key     | billing-service | team-billing | 2026-05-15   | 2026-08-15
jwt-signing  | JWT key     | auth-service    | team-auth    | 2026-06-01   | 2026-07-01
tls-api-prod | TLS cert    | api-gateway     | team-infra   | 2026-03-01   | 2026-06-01

---

2. Zero-Downtime Rotation Strategy

2.1 Dual-Key Rotation

Phase 1 — Add new secret (no removal)
  - Genera new secret (v2)
  - Storea v2 en secrets manager alongside v1
  - Deployea application con support para both v1 y v2
  - Application readea v2 para new operations, acceptea v1 para existing

Phase 2 — Switch a new secret
  - Deployea application para usar v2 como primary
  - Monitora para errors y authentication failures
  - Wait 24-48 hours para stability

Phase 3 — Remove old secret
  - Verifica que no service estea usando v1
  - Deletea v1 del secrets manager
  - Loggea rotation completion en audit trail

2.2 Database Password Rotation

import boto3
import psycopg2
from datetime import datetime

def rotate_database_password(secret_id: str):
    client = boto3.client('secretsmanager')

    # Step 1: Get current secret
    current = client.get_secret_value(SecretId=secret_id)
    current_creds = json.loads(current['SecretString'])

    # Step 2: Genera new password
    new_password = generate_secure_password(length=32)

    # Step 3: Conecta a database con current credentials
    conn = psycopg2.connect(
        host=current_creds['host'],
        user=current_creds['username'],
        password=current_creds['password'],
        database=current_creds['dbname'],
    )

    # Step 4: Updatea password en database (ALTER USER)
    with conn.cursor() as cur:
        cur.execute(f"ALTER USER {current_creds['username']} WITH PASSWORD %s", (new_password,))
    conn.commit()
    conn.close()

    # Step 5: Updatea secret en Secrets Manager
    new_creds = current_creds.copy()
    new_creds['password'] = new_password
    client.put_secret_value(
        SecretId=secret_id,
        SecretString=json.dumps(new_creds),
    )

    # Step 6: Verifica que new credentials funcionen
    conn = psycopg2.connect(
        host=new_creds['host'],
        user=new_creds['username'],
        password=new_password,
        database=new_creds['dbname'],
    )
    conn.close()

    print(f"Rotation complete for {secret_id} at {datetime.utcnow()}")

---

3. AWS Secrets Manager Automatic Rotation

3.1 Configure Rotation

import boto3

client = boto3.client('secretsmanager')

# Enablea automatic rotation — every 90 days
client.rotate_secret(
    SecretId='prod/payment-service/db-credentials',
    RotationLambdaARN='arn:aws:lambda:us-east-1:123456789012:function:secrets-rotation',
    RotationRules={
        'AutomaticallyAfterDays': 90,
    },
)

# Triggerea immediate rotation (para testing o emergency)
client.rotate_secret(
    SecretId='prod/payment-service/db-credentials',
    RotateImmediately=True,
)

3.2 Lambda Rotation Function

import boto3
import json
import psycopg2

def lambda_handler(event, context):
    secret_arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']

    client = boto3.client('secretsmanager')

    if step == 'createSecret':
        # Genera new secret version
        current = json.loads(client.get_secret_value(SecretId=secret_arn)['SecretString'])
        new_password = generate_secure_password(32)
        new_secret = {**current, 'password': new_password}
        client.put_secret_value(
            SecretId=secret_arn,
            ClientRequestToken=token,
            SecretString=json.dumps(new_secret),
        )

    elif step == 'setSecret':
        # Updatea database con new password
        pending = client.get_secret_value(
            SecretId=secret_arn,
            VersionStage='AWSPENDING',
            VersionId=token,
        )
        pending_creds = json.loads(pending['SecretString'])
        current = json.loads(client.get_secret_value(SecretId=secret_arn)['SecretString'])

        conn = psycopg2.connect(
            host=current['host'],
            user=current['username'],
            password=current['password'],
        )
        with conn.cursor() as cur:
            cur.execute(f"ALTER USER {pending_creds['username']} WITH PASSWORD %s",
                       (pending_creds['password'],))
        conn.commit()
        conn.close()

    elif step == 'testSecret':
        # Testea new credentials
        pending = client.get_secret_value(
            SecretId=secret_arn,
            VersionStage='AWSPENDING',
            VersionId=token,
        )
        creds = json.loads(pending['SecretString'])
        conn = psycopg2.connect(
            host=creds['host'],
            user=creds['username'],
            password=creds['password'],
        )
        conn.close()

    elif step == 'finishSecret':
        # Promotea pending a current
        client.update_secret_version_stage(
            SecretId=secret_arn,
            VersionStage='AWSCURRENT',
            MoveToVersionId=token,
            RemoveFromVersionId=event['CurrentVersionId'],
        )

---

4. HashiCorp Vault Rotation

4.1 Database Secrets Engine

# Enablea database secrets engine
vault secrets enable database

# Configura PostgreSQL connection
vault write database/config/payment-postgres \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@db-prod:5432/payment?sslmode=require" \
    allowed_roles="payment-service" \
    username="vault-admin" \
    password="admin-password"

# Crea role con rotation
vault write database/roles/payment-service \
    db_name=payment-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# Application requestea credentials (auto-rotated)
vault read database/creds/payment-service
# Returns: username=v-token-payment-123, password=auto-generated, lease_duration=1h

4.2 Transit Key Rotation

# Crea encryption key
vault write -f transit/keys/payments

# Rotea el key (crea new version, old version still decrypta)
vault write -f transit/keys/payments/rotate

# Lista key versions
vault read transit/keys/payments
# latest_version: 2

# Encrypta con latest version
vault write transit/encrypt/payments plaintext=$(base64 <<< "sensitive data")

# Decrypta (automaticamente usa correct version)
vault write transit/decrypt/payments ciphertext=vault:v2:...

---

5. JWT Signing Key Rotation

5.1 Dual-Key Strategy

import jwt
from datetime import datetime, timedelta

# Current y new signing keys
SIGNING_KEYS = {
    'v1': load_private_key('jwt-signing-v1'),
    'v2': load_private_key('jwt-signing-v2'),
}
CURRENT_VERSION = 'v2'

# Issue tokens con current key
def issue_token(user_id: str) -> str:
    payload = {
        'sub': user_id,
        'exp': datetime.utcnow() + timedelta(hours=1),
        'iat': datetime.utcnow(),
        'kid': CURRENT_VERSION,  # Key ID header
    }
    return jwt.encode(payload, SIGNING_KEYS[CURRENT_VERSION], algorithm='RS256', headers={'kid': CURRENT_VERSION})

# Verify tokens — acceptea both v1 y v2 durante transition
def verify_token(token: str) -> dict:
    # Try decodear sin verification para get kid
    unverified = jwt.decode(token, options={'verify_signature': False})
    kid = unverified.get('kid', 'v1')

    if kid not in SIGNING_KEYS:
        raise ValueError(f"Unknown key ID: {kid}")

    public_key = get_public_key(SIGNING_KEYS[kid])
    return jwt.decode(token, public_key, algorithms=['RS256'])

---

6. Emergency Rotation

6.1 Emergency Triggers

Trigger                              | Action                              | Timeline
─────────────────────────────────────┼─────────────────────────────────────┼──────────
Secret committed to git              | Rotea immediately, scan git history | < 1 hour
Secret exposed en logs               | Rotea immediately, purge logs       | < 1 hour
Secret shared via email/chat         | Rotea immediately                   | < 1 hour
Suspected breach                     | Rotea all secrets en blast radius    | < 4 hours
Employee departure con access        | Rotea dentro de 24 hours            | < 24 hours
Third-party breach notification      | Rotea affected API keys             | < 24 hours

6.2 Emergency Rotation Procedure

1. Identifica compromised secret(s) y blast radius
2. Genera new secret(s) immediately
3. Updatea secrets manager con new value(s)
4. Triggerea application reload / restart para pick up new secrets
5. Verifica all services estan usando new secret(s)
6. Revokea / invalida old secret(s) en el source (DB, API provider)
7. Audita access logs para misuse de old secret(s)
8. Documenta incident en security incident log
9. Notifica affected teams y stakeholders
10. Schedulea post-incident review dentro de 48 hours

See Also

• Complete Guide to Secrets Management
• Secrets Rotation Template
• Security Best Practices Guide
• Docker Secrets Management Without Hardcoding Credentials
• Live Database Credentials with HashiCorp Vault