Manage Application Secrets Securely

Overview

Secrets — API keys, database passwords, TLS certificates, encryption keys — are the crown jewels of any application. Hardcoding them in source code commits them to version control forever, exposed to anyone with repository access. Storing them in plaintext .env files on servers leaves them readable by any process running as the same user.

Secure secret management means storing secrets in dedicated vaults with encryption at rest, access control, audit logging, and automatic rotation. Applications fetch secrets at runtime through authenticated API calls, never persisting them to disk. The solution below covers cloud-native secret managers (AWS, GCP, Azure), HashiCorp Vault, and Kubernetes Secrets.

When to Use

Use this recipe when:

• Moving from development .env files to production secret storage. See Environment Variables for local config patterns.
• Rotating compromised credentials or complying with security audit requirements. See JWT Authentication for token rotation strategies.
• Sharing secrets across microservices, CI/CD pipelines, and team members. See Docker Basics for container secret injection.
• Managing TLS certificates, SSH keys, or database connection strings. See Parse Config Files for config-driven secret references.
• Auditing who accessed which secret and when. See Structured Logging for audit logging.

Solution

AWS Secrets Manager (Python)

import boto3
import json

client = boto3.client('secretsmanager')

def get_secret(secret_name):
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

# Usage
db_creds = get_secret('prod/db/postgres')
conn = psycopg2.connect(
    host=db_creds['host'],
    user=db_creds['username'],
    password=db_creds['password'],
)

HashiCorp Vault (Go)

import "github.com/hashicorp/vault/api"

client, _ := api.NewClient(api.DefaultConfig())
client.SetToken("s.xxx")

secret, _ := client.KVv2("secret").Get(context.Background(), "database/creds")
username := secret.Data["username"].(string)
password := secret.Data["password"].(string)

Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  username: admin
  password: "{{ .Values.dbPassword }}"

# Deployment referencing the secret
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username

Explanation

• Encryption at rest: Secrets are encrypted before being written to disk. AWS uses KMS, Vault uses its own encryption engine, and Kubernetes stores base64-encoded secrets (always enable etcd encryption for K8s).
• Live secrets: Vault and AWS can generate short-lived credentials on demand. A PostgreSQL role might be valid for 1 hour and then automatically revoked, minimizing blast radius if leaked.
• Access control: IAM policies, Vault policies, and Kubernetes RBAC restrict which services or users can read which secrets. Never grant blanket read access to all secrets.
• Audit logging: every secret read, write, and rotation is logged. Forward these logs to SIEM tools for anomaly detection.

Variants

Tool	Platform	Live Secrets	Auto-Rotation	Best For
AWS Secrets Manager	AWS	Yes	Yes	AWS-native workloads
HashiCorp Vault	Multi	Yes	Yes	Multi-cloud, on-prem
Azure Key Vault	Azure	Partial	Yes	Azure ecosystems
GCP Secret Manager	GCP	No	No	GCP-native workloads
Kubernetes Secrets	K8s	No	No	In-cluster injection

What Works

• Never commit secrets to Git: use .gitignore for .env files and pre-commit hooks (like git-secrets or truffleHog) to scan for accidental commits.
• Rotate secrets regularly: set automatic rotation policies (30-90 days) and rotate immediately if a secret is exposed or an employee leaves.
• Use least-privilege access: grant each service exactly the secrets it needs. A web server does not need the backup encryption key.
• Cache secrets briefly, not forever: fetch secrets at startup and refresh them periodically. Do not call the secret manager on every request.
• Separate secrets by environment: prod/db/password, staging/db/password, and dev/db/password should be different values in different vault paths.

Common Mistakes

• Storing secrets in environment variables on shared hosts: environment variables are visible to all processes on the same machine. Use file-based injection or dedicated secret sidecars instead.
• Forgetting to rotate after breaches: changing the application password is not enough. Rotate API keys, certificates, and session secrets thoroughly.
• Logging secrets: never log the full value of a secret. If you must log access, log the secret name and timestamp, never the password itself.
• Using Kubernetes Secrets without etcd encryption: by default, Kubernetes Secrets are base64-encoded, not encrypted. Enable etcd encryption at rest.

Performance Tips

1. Cache secrets in memory. Fetch once at startup, refresh periodically:

import time

class SecretCache:
    def __init__(self, ttl=300):
        self._cache = {}
        self._ttl = ttl
        self._timestamps = {}

    def get(self, name, fetch_func):
        if name not in self._cache or time.time() - self._timestamps[name] > self._ttl:
            self._cache[name] = fetch_func(name)
            self._timestamps[name] = time.time()
        return self._cache[name]

2. Use bulk secret reads. Fetch all secrets for a service in one API call:

# AWS: store all service secrets as a single JSON secret
response = client.get_secret_value(SecretId='prod/api/all-secrets')
all_secrets = json.loads(response['SecretString'])
# all_secrets = {'db_url': '...', 'stripe_key': '...', 'sendgrid_key': '...'}

3. Use connection pooling for Vault. Reuse HTTP connections to Vault:

client, _ := api.NewClient(api.DefaultConfig())
// Client reuses connections internally
// Adjust transport settings for high-throughput:
transport := &http.Transport{
    MaxIdleConns:        10,
    IdleConnTimeout:     30 * time.Second,
}

4. Preload secrets in container init. Fetch secrets during container startup, not on first request:

# Kubernetes init container
initContainers:
- name: secret-loader
  image: secret-loader:latest
  command: ["/bin/sh", "-c"]
  args:
    - |
      vault kv get -field=password secret/db > /secrets/db_password
      vault kv get -field=apikey secret/api > /secrets/api_key
  volumeMounts:
  - name: secrets
    mountPath: /secrets

5. Use sidecar for secret rotation. A sidecar can watch for secret changes and send signals to the main container:

# Vault Agent sidecar sends SIGHUP when secrets change
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-inject-command-db-creds: "kill -HUP 1"