Secrets Rotation Template

Use this template to document who rotates which secrets, how often, and with what procedure — including the zero-downtime path for credentials that production can’t live without. For storing secrets safely in the first place, see Docker Secrets Management.

Overview

A leaked API key is only a disaster if it’s still valid. Rotation limits the blast radius by making sure credentials expire before attackers can use them — and it doubles as an audit, because rotating a secret forces you to find out who actually uses it. I’ve watched a “simple” password rotation take down dashboards for two days because nobody knew a reporting tool shared the same credential. The inventory is what prevents that.

Rotation without process is chaos: services break mid-deploy, someone shares the new root password in Slack, and the next engineer guesses at the procedure. This template coordinates the lifecycle — inventory, cadence, procedure, emergency path, validation — so rotation is routine instead of a fire drill.

When to Use

• You’re setting up a secrets management policy for the first time and need an inventory to start from.
• An audit or compliance framework (SOC 2, PCI DSS, ISO 27001) requires a documented rotation cadence.
• A secret was exposed and you need to rotate it without breaking production.
• You’re migrating from manually managed secrets to a vault and need to know what exists before you move it.

Don’t use it for one-off credential resets — a single leaked key needs the emergency rotation row, not the whole schedule.

Solution

# Secrets Rotation Schedule

## 1. Secret Inventory

| Secret | Type | Environment | Storage Location | Current Version | Last Rotated | Next Rotation | Owner |
|--------|------|-------------|------------------|-----------------|--------------|---------------|-------|
| `DB_MASTER_PASSWORD` | Password | Production | HashiCorp Vault | `v3` | `YYYY-MM-DD` | `YYYY-MM-DD` | @db-owner |
| `AWS_ACCESS_KEY_ID` | API key | Production | AWS Secrets Manager | `v2` | `YYYY-MM-DD` | `YYYY-MM-DD` | @infra-owner |
| `JWT_SIGNING_KEY` | Signing key | All | Vault / HSM | `v4` | `YYYY-MM-DD` | `YYYY-MM-DD` | @security-owner |
| `TLS_CERT_API` | Certificate | Production | Cert-manager + Vault | `v2` | `YYYY-MM-DD` | `YYYY-MM-DD` | @infra-owner |
| `GITHUB_DEPLOY_TOKEN` | Token | CI/CD | GitHub Secrets | `v1` | `YYYY-MM-DD` | `YYYY-MM-DD` | @devops-owner |

## 2. Rotation Cadence Policy

| Secret Type | Rotation Interval | Trigger Events | Grace Period |
|-------------|-------------------|----------------|--------------|
| Database passwords | 90 days | Employee departure, suspected breach | 7 days |
| Cloud API keys | 90 days | Key exposed in logs, IAM policy change | 7 days |
| TLS certificates | Auto-renew (30 days before expiry) | Certificate expiry alert | 14 days |
| Signing / encryption keys | 180 days | Key compromise, algorithm deprecation | 14 days |
| CI/CD tokens | 90 days | Token scope change, repository access change | 3 days |
| Service account passwords | 90 days | Role change, service decommission | 7 days |
| Emergency break-glass | Per-use | After each use | Immediate |

Alert at 30, 14, and 7 days before each due date — last-minute rotation is where mistakes happen.

## 3. Rotation Procedure

### Zero-Downtime Key Rotation (Application Secrets)

1. **Generate new secret** in Vault / secrets manager
2. **Deploy application with dual-read:** app accepts both old and new secret
3. **Validate:** confirm new secret works in staging
4. **Update consumers:** rotate downstream services one by one
5. **Remove old secret:** after validation period (24–48 hours)
6. **Update inventory:** mark new version and date

### Certificate Rotation (TLS)

1. **Provision new certificate** from CA
2. **Deploy to load balancer / ingress** alongside existing certificate
3. **Validate:** SSL Labs scan + synthetic monitor
4. **Wait for propagation:** ensure all edge caches refreshed
5. **Remove old certificate** from all endpoints
6. **Update inventory:** mark new expiry date

## 4. Emergency Rotation Playbook

| Trigger | Action | Owner | SLA |
|---------|--------|-------|-----|
| Secret exposed in public repo | Rotate immediately; revoke old | @security-owner | 1 hour |
| Employee with secret access terminated | Rotate all secrets they had access to | @security-owner + @owner | 4 hours |
| Suspected breach | Rotate all critical secrets; incident response | @security-owner | 24 hours |
| Certificate expires in < 7 days | Emergency reissue + deploy | @infra-owner | 24 hours |

## 5. Validation Checklist

- [ ] New secret is readable by all expected services
- [ ] Old secret is revoked / deleted (not just unused)
- [ ] Application logs show no authentication failures post-rotation
- [ ] CI/CD pipelines pass with new secret
- [ ] Monitoring alerts are green for 24 hours
- [ ] Inventory updated with new version and rotation date
- [ ] Runbook updated if procedure changed

## 6. Exception Log

| Secret | Reason for Extended Interval | Risk Accepted By | New Date | Review |
|--------|------------------------------|------------------|----------|--------|
| | | | | |

## 7. Rotation Metrics

| Month | Secrets Scheduled | Rotated On Time | Late | Compliance Rate |
|-------|------------------:|----------------:|-----:|----------------:|
| | | | | |

Explanation

The template treats rotation as a lifecycle, not a one-time event. The inventory answers “what do we have?” The cadence answers “how often?” The procedure answers “how?” The emergency playbook answers “what if?” And the metrics answer “is the program working?”

Zero-downtime rotation is the hardest part, and dual-read mode is what makes it possible: the application accepts both the old and new secret during a transition window, so consumers can migrate one by one instead of in a coordinated outage. The validation checklist exists because the most common rotation failure isn’t the rotation itself — it’s a cron job or a cached connection string still using the old value three days later.

The exception log matters more than it looks. Every program ends up with secrets that can’t rotate on schedule (a vendor token with a 12-month contract, a legacy system that can’t do dual-read). Writing them down with a named risk-acceptor is the difference between a documented exception and a forgotten secret.

Dual-Read vs Maintenance Window

Not every secret can do dual-read. Use this to pick the procedure:

Approach	When it fits	Cost
Dual-read	App can read two secrets, or the system supports two valid credentials (new DB user)	Zero downtime, more coordination
Maintenance window	Legacy systems, single-slot credentials, secrets baked into config	Short planned outage, simpler procedure
Canary rotation	Unknown or third-party consumers	Low blast radius, slower

If a consumer is a third party you don’t control, the emergency playbook’s canary approach is safer: rotate for a subset, watch for failures, then complete the rotation on their timeline.

Secrets Rotation Automation Example

#!/bin/bash
# Rotate a database password with zero downtime using the dual-read
# pattern: the old and new credentials stay valid during the transition.

set -euo pipefail

DB_HOST="db.internal"
DB_NAME="appdb"
OLD_USER="app_user_old"
NEW_USER="app_user_new"
NEW_PASSWORD=$(openssl rand -base64 32)

# 1. Create the new user with identical permissions
psql -h "$DB_HOST" -U admin -d "$DB_NAME" -c "
  CREATE USER $NEW_USER WITH PASSWORD '$NEW_PASSWORD';
  GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO $NEW_USER;
  GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO $NEW_USER;
"

# 2. Push the new credentials to the secret manager
aws secretsmanager update-secret \
  --secret-id prod/db/app-user \
  --secret-string "{\"username\":\"$NEW_USER\",\"password\":\"$NEW_PASSWORD\"}"

# 3. Rolling restart so pods pick up the new secret
kubectl rollout restart deployment/app -n production
kubectl rollout status deployment/app -n production --timeout=300s

# 4. Verify connections now use the new user
psql -h "$DB_HOST" -U admin -d "$DB_NAME" -c "
  SELECT usename, count(*) FROM pg_stat_activity
  WHERE datname='appdb' GROUP BY usename;
"

# 5. After the validation window, drop the old user
psql -h "$DB_HOST" -U admin -d "$DB_NAME" -c "DROP USER $OLD_USER;"

echo "Rotation complete. Old user dropped."

Example: A Filled Rotation Entry

| Secret | Type | Environment | Storage Location | Current Version | Last Rotated | Next Rotation | Owner |
|--------|------|-------------|------------------|-----------------|--------------|---------------|-------|
| `DB_MASTER_PASSWORD` | Password | Production | Vault | `v4` | `2026-10-15` | `2027-01-13` | @db-owner |

Rotation log — 2026-10-15 02:00 UTC (low-traffic window):
- New user `app_user_2026q4` created with identical grants; staged at 01:30.
- Secret updated in Vault; rolling restart finished 02:12.
- pg_stat_activity: 0 connections on `app_user_2026q3` by 02:20.
- Reporting dashboard confirmed reading via analytics replica (separate
  credential — not affected).
- Old user dropped 2026-10-17 after 48h clean. No auth failures in logs.

That entry answers the questions the next audit will ask: what changed, when, who verified it, and what happened to the old credential. Notice what it records that a bare table row can’t — the reporting dashboard was checked because it shares the database, and the batch ETL was verified to use a separate credential. Rotations fail on the consumers nobody wrote down; the log is where you prove you checked.

Emergency secrets deserve their own row

Break-glass credentials — the admin password you hope never to use — follow different rules. They rotate per use, not on a calendar, and the procedure should be a one-command reset, not a dual-read dance. Keep them out of the normal cadence table and in the emergency playbook, where the SLA is measured in minutes.

Variants

Context	Key Difference	Tool
Kubernetes	cert-manager for TLS, External Secrets Operator for app secrets	Vault, AWS SM, Azure Key Vault
Cloud-native (AWS)	IAM role rotation via STS, Secrets Manager auto-rotation	AWS Secrets Manager, Lambda rotation functions
Enterprise on-prem	HSM for key storage, manual change control	Thales Luna, SafeNet
Startup / small team	Shared 1Password / Bitwarden, quarterly manual rotation	1Password, Bitwarden
High-frequency microservices	Short-lived mTLS certs (SPIFFE/SPIRE)	SPIRE, Istio

What Works

• Automate certificates and cloud keys first — humans forget schedules; cert-manager and Secrets Manager don’t.
• Use dual-read during transition — single-secret rotation causes outages; old+new during the window causes none.
• Rotate after incidents, not just on schedule — a breach doesn’t wait for your 90-day cadence.
• Alert 30/14/7 days before each due date — rotations done at the last minute are the ones that break things.
• Rotate during low-traffic windows — even zero-downtime procedures fail occasionally; fail when it costs least.
• Audit every rotation event — failed rotations are early indicators of configuration drift.
• Never hardcode secrets — rotation only helps if the secret isn’t also sitting in a repo from 2024.

Common Mistakes

• Rotating without identifying consumers first — the fastest rotation is useless if it leaves a service broken. Query the secret manager’s access logs before touching anything.
• Forgetting downstream secrets — updating the database password but not the reporting tool breaks dashboards.
• Not revoking the old secret — “unused” secrets are still valid attack vectors.
• Rotating everything at once — a failed mass rotation is a total outage; batch by service.
• Treating exceptions as permanent — the exception log needs a review date, or “temporary” becomes “forever”.

How to Adopt This Template

Start with the inventory — most teams discover they have more secrets than they thought, and half have no owner. Then pick one low-risk secret (a staging token, a CI credential) and run it through the full procedure once. The pilot tells you where the procedure breaks before a production password does it for you.

The cadence that works in practice:

1. Week 1: fill the inventory from the secret manager’s access logs and cloud provider list — don’t rely on memory.
2. Week 2: assign cadences and owners; enable the 30/14/7-day alerts.
3. Week 3: pilot rotation on a low-risk secret with dual-read; time how long it takes.
4. Ongoing: review the metrics table monthly — late rotations and exception-log growth are the two signals that the program is slipping.

Troubleshooting

• A service you didn’t know about breaks after rotation: it was consuming the secret from an unknown path. Query the secret manager’s access logs for every identity that read the secret in the last 90 days — the list is usually longer than the inventory suggests. Add the consumer, then decide whether to roll back or fix forward.
• The old secret is still valid weeks after rotation: someone skipped the revocation step, or a “temporary” grace period was never closed. Set the revocation as a follow-up task with a deadline when you schedule the rotation — open grace periods are the most common audit finding.
• A cron job or batch worker fails days later: it caches the connection or reads the secret only at start. After rotating, search logs for authentication failures over the full interval between runs — not just the first hour.
• cert-manager renews the cert but clients still see the old one: the new certificate didn’t propagate to all ingress endpoints or edge caches. Check every LB, ingress, and CDN that terminates TLS — the propagation wait in the procedure exists for this reason.
• Emergency rotation during an incident stalls: nobody knows who can approve the exception. The playbook’s owner column is the fix — a rotation that needs three approvals at 3 a.m. was designed in a meeting, not tested.
• The rotation script works in staging but the secret never reaches production pods: the External Secrets Operator sync interval or the deployment’s refresh strategy is the usual cause — a kubectl rollout restart after the sync fixes it, but the fix belongs in the runbook, not in someone’s memory.
• Two services rotate the same secret on different schedules: shared credentials are a coupling problem, not a scheduling problem. Split them into per-service credentials; if the system can’t express that, log it in the exception register with a fix-by date.

Production Notes

• Keep the old secret readable during the grace period — immediate revocation removes your rollback path if a consumer surfaces late.
• Watch auth-failure metrics for 48 hours post-rotation — the failures that matter arrive when the batch job runs, not when the app reloads.
• Test the emergency playbook quarterly — a tabletop exercise is cheap; finding out the on-call can’t revoke a Vault token mid-breach isn’t.
• Log rotations to the SIEM — rotation events are audit evidence and anomaly signal at the same time.

Key Takeaways

• Rotation is a lifecycle: inventory → cadence → procedure → validation → metrics. Skipping the inventory makes everything else guesswork.
• Dual-read is the difference between routine and outage — build for old+new valid during transition.
• The emergency playbook isn’t optional — secrets leak on their own schedule, not yours.
• Measure the program: rotated-on-time rate and exception-log length tell you whether rotation is a habit or a hope.
• Start with the secrets nobody owns — unowned secrets are the ones that leak silently; assigning an owner is half the rotation.

Further Reading

• NIST SP 800-57 — Key Management — cryptoperiod and key lifecycle guidance behind the cadence table
• HashiCorp Vault — Dynamic Secrets — database credentials that rotate themselves
• cert-manager documentation — automatic TLS certificate renewal in Kubernetes
• Companion resources — downloadable schedule, rotation runbook, and validation checklist

Common Production Pitfalls

• Keeping the old secret “temporarily” active for months — set the revocation date when you schedule the rotation.
• Treating the inventory as a one-time census — every new service adds secrets; stale inventories rot fast.
• Rotating the password but not the tokens derived from it (replica credentials, read-only users).
• Assuming the secret manager’s auto-rotation covers everything — it rotates the value, not the consumers.
• Skipping the exception log review — exceptions without dates become permanent exposures.
• Documenting the procedure in the rotation log only — the next engineer searches the wiki, not your spreadsheet.