Secrets Management: Vault, Cloud Managers

Overview

Secrets (passwords, API keys, tokens, certificates) are the keys to your kingdom. Storing them in source code, configuration files, or environment variables is a common source of breaches. Proper secrets management makes sure credentials are encrypted, rotated, audited, and accessible only to authorized services and users. For foundational knowledge, see the Secure Coding Guide and the Cryptography Basics Guide. The following walks through the leading secret management solutions and the practices that make them useful.

When to Use

• For alternatives, see Complete Guide to Secrets Management.

• You’ve got credentials, API keys, or certificates to protect

• You need to share secrets across teams or services

• You want to audit who accessed what secret and when

• You’re building a CI/CD pipeline that needs runtime secrets

What Not to Do

Anti-Pattern	Why It Fails	What to Do Instead
Hardcode secrets in source	Commits to Git are forever; history leaks	Use secret references
Store secrets in env vars	Visible in process dumps, /proc, and debug endpoints	Use secret managers with runtime injection
Share one password across services	Blast radius is entire infrastructure	Service-specific credentials
Never rotate secrets	Compromised keys remain valid indefinitely	Automate rotation
Send secrets in Slack/email	Unencrypted, unlogged, uncontrolled	Use approved secret sharing tools

HashiCorp Vault

The open-source standard for secrets management. HashiCorp Vault gives you a unified API to store, rotate, and audit secrets across clouds. The OWASP secrets management cheat sheet recommends using a dedicated secret store rather than environment variables.

Core Concepts

Component	Purpose
Secrets Engine	Stores or generates secrets (KV, database, PKI, AWS)
Auth Method	How users/services authenticate (Kubernetes, OIDC, AppRole)
Policy	Fine-grained access control (ACL)
On-demand Secret	Short-lived, automatically revoked credentials

On-demand Database Credentials

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/my-postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app" \
  connection_url="postgresql://{{username}}:{{password}}@db:5432/mydb" \
  username="vaultadmin" \
  password="vaultpass"

# Create a role that generates 1-hour leases
vault write database/roles/app \
  db_name=my-postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';" \
  default_ttl="1h" \
  max_ttl="24h"

Reading Secrets in Applications

import hvac

client = hvac.Client(url='https://vault.example.com')
client.auth.kubernetes.login(role='my-app', jwt=service_account_token)

# Read a static secret
secret = client.secrets.kv.v2.read_secret_version(path='my-app/config')
api_key = secret['data']['data']['api_key']

# Generate on-demand database credentials
db_creds = client.secrets.database.generate_credentials(name='app')
username = db_creds['data']['username']
password = db_creds['data']['password']

Vault Policy Example

Vault uses policies to control who can access which secrets. A least-privilege policy grants read-only access to a specific path:

# Grant read-only access to payment-service secrets
path "secret/data/payment-service/*" {
  capabilities = ["read"]
}

# Grant generate access to database credentials
path "database/creds/payment-app" {
  capabilities = ["read"]
}

Avoid wildcard policies like path "secret/data/*" in production. Each service should have its own policy scoped to its secrets. Audit policy changes with vault audit enable file to log every access decision.

AWS Secrets Manager

Fully managed secret rotation for AWS workloads. The AWS Secrets Manager documentation covers automatic rotation with Lambda functions. Secrets Manager integrates natively with RDS, Redshift, and DocumentDB, and supports custom rotation for other services via Lambda.

How Rotation Works

When you configure rotation, Secrets Manager calls a Lambda function on a schedule. The Lambda function:

1. Generates a new password.
2. Updates the database or service with the new credentials.
3. Updates the secret in Secrets Manager.
4. Optionally notifies applications to refresh cached credentials.

For RDS, AWS has pre-built rotation templates. For custom services, you write a Lambda that uses the four-step rotation protocol.

# Create a secret
aws secretsmanager create-secret \
  --name prod/database/password \
  --secret-string '{"username":"admin","password":"supersecret"}'

# Retrieve a secret
aws secretsmanager get-secret-value --secret-id prod/database/password

# Configure automatic rotation
aws secretsmanager rotate-secret \
  --secret-id prod/database/password \
  --rotation-lambda-arn arn:aws:lambda:...:function:rotation \
  --automatically-after-days 30

IAM Policy for Access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:*:*:secret:prod/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceVpc": "vpc-12345"
        }
      }
    }
  ]
}

Azure Key Vault

Integrated with Azure AD and Microsoft ecosystems. Key Vault stores secrets, certificates, and keys with hardware security module (HSM) backing for premium tiers. The main advantage over other managers is native integration with Azure AD authentication: services authenticate with Managed Identity, eliminating the need to store credentials to access the vault itself.

Managed Identity vs Service Principal

Managed Identity is the recommended approach for Azure workloads. It eliminates the need to store client secrets: the identity is assigned to the compute resource (VM, App Service, AKS pod) and Azure AD handles token acquisition automatically. Service Principals require storing a client secret, which creates a chicken-and-egg problem.

# Create a Key Vault
az keyvault create --name myvault --resource-group mygroup --location eastus

# Store a secret
az keyvault secret set --vault-name myvault --name db-password --value secret123

# Retrieve a secret
az keyvault secret show --vault-name myvault --name db-password

Managed Identity Access

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

credential = DefaultAzureCredential()
client = SecretClient(vault_url="https://myvault.vault.azure.net/", credential=credential)

secret = client.get_secret("db-password")
print(secret.value)

GCP Secret Manager

Native integration with GCP IAM and Cloud Run. Secret Manager stores versions of secrets, so you can roll back to a previous version if a rotation fails. Access is controlled via IAM roles: roles/secretmanager.secretAccessor for reading, roles/secretmanager.secretAdmin for managing.

# Create a secret
echo -n "supersecret" | gcloud secrets create db-password --data-file=-

# Add a version
echo -n "newsecret" | gcloud secrets versions add db-password --data-file=-

# Access from Cloud Run (no code changes needed)
gcloud run deploy my-app --set-secrets=DB_PASSWORD=db-password:latest

IAM Access Control

Grant a service account access to a specific secret:

# Grant access to a specific secret
gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:my-app@my-project.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Use secretAccessor for application runtime, secretAdmin for CI/CD pipelines that manage secrets. Avoid granting secretAdmin to production workloads.

CI/CD Integration

GitHub Actions with OIDC

jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/GitHubActionsRole
          aws-region: us-east-1
      - run: |
          DB_PASSWORD=$(aws secretsmanager get-secret-value --secret-id db-password --query SecretString --output text)
          echo "DB_PASSWORD=$DB_PASSWORD" >> $GITHUB_ENV

Secrets Scanning in Pipelines

# Detect secrets before merge
 trufflehog filesystem --directory=.
 gitleaks detect --source .
 detect-secrets scan

Rotation Strategies

Strategy	Best For	Complexity
Manual	Ad-hoc, small teams	Low
Lambda/Function	AWS RDS, standard databases	Medium
Vault On-demand	Microservices, multi-cloud	High
Certificate Auto	TLS certificates (Let’s Encrypt, ACM)	Low

When to Use Each Strategy

Manual works for teams with fewer than 10 secrets and no compliance requirements. Document the rotation process and set calendar reminders. The risk is human error: forgotten rotations leave stale credentials.

Lambda/Function is the sweet spot for AWS workloads. AWS has pre-built rotation templates for RDS, Redshift, and DocumentDB. For custom services, you write a Lambda that uses the four-step protocol (generate, update, verify, finish). Set rotation intervals between 30-90 days.

Vault On-demand eliminates rotation entirely. Instead of rotating a static password, Vault generates short-lived credentials on each request. The credentials expire after a TTL (typically 1 hour). This is the most secure approach but requires Vault infrastructure and application changes.

Certificate Auto applies to TLS certificates. Let’s Encrypt and AWS ACM handle rotation automatically. The challenge is ensuring the renewed certificate is deployed to all endpoints before the old one expires.

Dual-Secret Rotation Flow

The dual-secret pattern lets you rotate secrets without downtime. The new secret is deployed while the old one stays active, then the old one is revoked after verification:

Common Mistakes

• Using one secret for all environments: separate prod, staging, and dev secrets
• No audit logging: you can’t investigate breaches without access logs
• Overly permissive policies: a compromised CI/CD token shouldn’t access production secrets
• Ignoring secret sprawl: old API keys in environment variables, logs, and backups
• No revocation plan: when a secret leaks, how quickly can you rotate it?
• Storing secrets in .env files committed to Git: even a .env.example with real values leaks. Use .env.example with placeholder values only
• Hardcoding secrets in Docker images: docker history exposes every layer. Use runtime injection via Vault or cloud secret managers
• Sharing secrets via Slack or email: these channels aren’t encrypted, logged, or auditable. Use a secret sharing tool like OnionShare or your secret manager’s sharing feature

Advanced Topics

Scenario: Secrets Management for Microservices

System: 10 microservices on K8s, AWS
Stack: AWS Secrets Manager + External Secrets Operator

Architecture:
  Developer -> GitHub (secret in repo: NEVER)
  Developer -> AWS Secrets Manager (manual or CLI)
  Secrets Manager -> External Secrets Operator (K8s)
  ESO -> creates Kubernetes Secret
  Pod -> mounts Secret as env var or volume

Secret rules:
  | Rule | Reason |
  |------|--------|
  | Never in code | Git history is permanent |
  | Never in .env in prod | Plaintext file on disk |
  | Never in logs | Logs are accessible |
  | Never in error messages | Exposes to client |
  | Automatic rotation | Minimizes leak impact |
  | Least privilege | Each service only accesses its secrets |
  | Audit log | Who accessed which secret and when |

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-service-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: payment-service-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: production/payment-service/database-url
    - secretKey: STRIPE_SECRET_KEY
      remoteRef:
        key: production/payment-service/stripe-key
    - secretKey: JWT_PRIVATE_KEY
      remoteRef:
        key: production/payment-service/jwt-private

Automatic rotation (Secrets Manager):

• Database: rotate every 30 days (Lambda function)
• Stripe: manual rotation (API does not support auto-rotation)
• JWT: rotate every 90 days (deploy new key, keep old 7 days)
• API keys: rotate every 60 days

Leak detection:

• git-secrets: pre-commit hook blocks commits with patterns
• TruffleHog: scans git history
• GitHub Secret Scanning: automatic alerts
• AWS CloudTrail: audit log of Secrets Manager access

Lessons:

• External Secrets Operator syncs secrets without manual K8s secrets
• Automatic rotation minimizes leak impact
• git-secrets in pre-commit is the first line of defense
• Each service should have its own secrets (no sharing)
• Audit log of secret access is mandatory for SOC2

How do I rotate secrets without downtime?

Use the dual-secret pattern: configure the new secret while the old one is still active. Deploy the app with the new secret. Verify it works. After confirming, invalidate the old one. For JWT, accept both keys during a transition period (7 days). For DB, rotate the password via Secrets Manager with a Lambda that updates the password and refreshes the pods.

See Also

• Zero Trust Architecture Guide — secrets management is a pillar of zero trust
• CI/CD Security Guide — securing secrets in pipelines
• HashiCorp Vault documentation
• AWS Secrets Manager documentation
• OWASP Secrets Management Cheat Sheet