Logging Standards Document Template
A document template for defining structured logging conventions, log levels, retention, and observability requirements across services.
Overview
A Logging Standards Document is the contract that says how your services, applications, and infrastructure should emit logs. When every service logs the same way, debugging, monitoring, security investigation, and compliance all get easier. This template covers log levels, structured formats, required fields, retention, sampling, and security rules.
Fill in each section, assign an owner and a review cadence, and keep the document next to your runbooks — a standard nobody can find is a standard nobody follows.
When to Use
Reach for this document when you’re onboarding a new service or development team, consolidating logs from several systems into a central observability platform, or getting ready for a security audit or compliance review. It’s also the right artifact after a production incident where the logs turned out incomplete or inconsistent, and when you’re setting the logging strategy for microservices or serverless environments.
For a broader observability strategy, see the Complete Guide to Observability with the Grafana Stack.
Prerequisites
- Somewhere for the logs to land: ELK, Splunk, Datadog, Grafana Loki, CloudWatch, or similar.
- One timestamp standard and one timezone policy, agreed across the org.
- A shared definition of the events that can’t afford to go missing — the ones that always have to reach the log stream.
- Agreement on sensitive data classification and log redaction rules.
Solution
Document
1. Log Levels
| Level | Use | Example |
|---|---|---|
| DEBUG | Detailed diagnostic information during development | cache miss for key user:1234 |
| INFO | Normal application events | user logged in, order completed |
| WARN | Unexpected but recoverable situations | connection timeout, retrying |
| ERROR | Failures that affect operation | payment gateway returned 500 |
| FATAL | Critical failures requiring immediate attention | database unavailable, service shutdown |
Guidelines:
- DEBUG must be off in production by default.
- INFO is the default production level for most services.
- ERROR must trigger an alert or ticket.
- FATAL must page the on-call team.
2. Structured Log Format
Emit every log entry as JSON carrying these required fields:
| Field | Type | Description | Example |
|---|---|---|---|
timestamp | ISO 8601 | Event time in UTC | 2026-06-27T14:30Z |
level | string | Log level | INFO |
service | string | Service or application name | payment-service |
environment | string | Environment | production |
message | string | Human-readable summary | Order 12345 completed |
correlation_id | string | Request trace ID | abc-123-def |
span_id | string | OpenTelemetry span ID | span-xyz-789 |
Optional fields:
user_id: Identity of the user associated with the event.tenant_id: Identifier for multi-tenant isolation.duration_ms: Time taken to complete an operation.error_code: Stable error code for programmatic handling.source_file: File and line where the log was emitted.
Naming matters more than it looks. Pick one convention — this template uses snake_case — and apply it everywhere, including fields nested inside context objects. A service logging correlation_id and another logging correlationId can’t be queried together, and the failure is silent.
3. Required Event Categories
| Category | Events to Log | Level |
|---|---|---|
| Authentication | Login, logout, failed login, MFA challenge | INFO / WARN |
| Authorization | Access denied, permission escalation | WARN |
| Data changes | Create, update, delete on sensitive records | INFO |
| Errors | Exceptions, external failures, retries | ERROR |
| Performance | Slow queries, high latency, timeouts | WARN |
| Security | Suspicious activity, rate limit hits, blocked requests | WARN |
| Operational | Startup, shutdown, configuration changes | INFO |
| Business | Order placed, payment received, workflow completed | INFO |
4. Sensitive Data and Redaction
| Data Type | Logged | Redaction |
|---|---|---|
| Passwords | Never | Redact or exclude |
| Credit card numbers | Never | Tokenize or exclude |
| API keys | Never | Redact or exclude |
| Personal names | With approval | Mask if not required |
| Email addresses | Allowed | Partial mask for non-admins |
| IP addresses | Allowed | Allowed for security logs |
| User IDs | Allowed | Allowed |
Rules:
- Never log secrets or credentials.
- Use allowlists for personal data fields.
- Redact or tokenize values before logging.
- Encrypt logs if they contain sensitive data.
5. Retention and Sampling
| Log Type | Retention | Sampling | Notes |
|---|---|---|---|
| Application logs | 30 days | 100% | Keep all for debugging |
| Security logs | 1 year | 100% | Compliance requirement |
| Audit logs | 7 years | 100% | Legal and regulatory |
| Debug logs | 7 days | 100% | Only when enabled |
| Trace logs | 14 days | 1% | High volume, cost control |
6. Log Aggregation and Transport
| Requirement | Rule |
|---|---|
| Transport | Send logs to the central platform with backpressure handling. |
| Ordering | Use timestamps for ordering; tolerate minor clock skew. |
| Buffering | Buffer locally if the collector is unavailable. |
| Encoding | Use UTF-8 JSON. |
| Backups | Replicate critical logs to a secondary storage. |
| Alerting | Route ERROR and FATAL logs to the alerting system. |
Explanation
Consistent logging turns noisy text files into searchable, structured data. By pinning down levels, fields, and retention up front, teams can correlate events across services, investigate incidents faster, and answer compliance questions without archaeology. Structured logs also plug straight into tracing and metrics — a correlation_id is the join key between all three signals.
The document only works if it’s enforced, not just written — see Enforcing the Standard below.
Structured Log Format Example
{
"timestamp": "2026-07-11T10:55:32.123Z",
"level": "ERROR",
"service": "auth-service",
"environment": "production",
"correlation_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"user_id": "usr_8f7e6d5c",
"message": "JWT validation failed",
"error_code": "AUTH_JWT_INVALID",
"context": {
"endpoint": "/api/v1/auth/verify",
"method": "POST",
"status_code": 401,
"duration_ms": 12,
"ip_address": "10.0.1.42",
"user_agent": "MobileApp/2.3.1"
},
"error": {
"type": "TokenExpiredError",
"message": "jwt expired",
"stack": "TokenExpiredError: jwt expired\n at ..."
}
}
Log Level Decision Matrix
=== When to Use Each Log Level ===
FATAL - System cannot continue. Process will exit.
Examples: config load failure, port binding error, OOM
ERROR - Operation failed but system continues.
Examples: request failed, DB query error, external API timeout
WARN - Unexpected but recoverable condition.
Examples: retry succeeded, cache miss, deprecated API usage
INFO - Significant business or operational event.
Examples: user login, order placed, deployment started
DEBUG - Diagnostic detail for troubleshooting.
Examples: variable state, query params, cache contents
TRACE - Finest-grained execution flow.
Examples: function entry/exit, loop iteration count
RULES:
- Production: INFO and above (DEBUG/TRACE off)
- Staging: DEBUG and above
- Development: TRACE and above
- Never log at DEBUG in production unless actively debugging
- ERROR must be actionable — if it isn't, it's INFO
Enforcing the Standard
A written standard without enforcement drifts within a quarter. Three cheap mechanisms keep it alive:
- Shared logging wrapper. Ship a library per language that hardcodes the required fields, applies the naming convention, and exposes the redaction hooks. Nobody should have to keep the schema in their head.
- CI schema check. A test that asserts emitted logs are valid JSON and contain
timestamp,level,service,environment,message, andcorrelation_idcatches format regressions before merge. Add a regex pass over common secret patterns (Bearer,password=, AWS key formats) while you’re at it. - Versioning and ownership. Give the document an owner, a changelog section, and a review cadence (quarterly works). Field additions are backwards-compatible; renames are a breaking change that deserves a migration note.
The failure mode to watch for is the write-once-forget document: six months in, half the services log correlation_id, the other half settled on trace_id, and nobody remembers which name their dashboards still query. That is why enforcement matters more than wording — put the standard where code review happens, not in a wiki page nobody opens. If a reviewer can block a PR for logging a password or dropping correlation_id, the standard survives contact with reality. If it can’t, the document is decoration.
Variants
- Cloud logging standards: adjusted to AWS CloudWatch, Azure Monitor, or Google Cloud Logging.
- Container and Kubernetes logging: Covers sidecar log shippers, Fluentd, and pod log conventions.
- Security-focused logging: puts audit events, integrity, and tamper detection first.
- Serverless logging: deals with short-lived functions, cold starts, and centralized collection.
- Mobile or client logging: Focuses on privacy, batching, and offline buffering.
What works
The teams that get value from a logging standard share a few habits. They pick one structured format and use it in every service, carry a correlation ID through every request so distributed tracing actually traces, and log outcomes at business boundaries instead of narrating each internal step. Messages stay short — context lives in the structured fields, not in the sentence.
They also treat log levels as a contract: an ERROR means someone should act, which is what keeps alerts worth reading. Sensitive data never reaches the stream by default, retention gets reviewed against real storage bills and compliance needs, and log parsing plus alert rules get exercised during deployments like any other part of the release.
Common Mistakes
The classic failure is logging everything at INFO: real problems drown in routine noise and alerts stop meaning anything. Close behind come unparsable plain-text logs, missing or inconsistent timestamps, and services that each invent their own field names — correlation_id in one, correlationId in the next — so cross-service queries silently match nothing.
The other recurring sins are about data and retention: passwords or tokens ending up in the stream, not enough context to reproduce a failure after the fact, and keeping every log forever while the storage bill quietly climbs.
Troubleshooting
- No logs for a failing request: verify log shipping, retention, and that the request reached the service.
- Fields missing or renamed after a deploy: check the shared wrapper version — a library bump often changes the schema silently.
- Duplicate log lines in the index: look for two shippers tailing the same file, or a direct-shipping logger running alongside the agent.
- Alert fires but the service is healthy: tune thresholds and use multi-signal alerts.
- Log volume exploded after a release: check for a new INFO log inside a hot loop or a retry path; enforce sampling on the noisy category.
- Trace is incomplete across services: ensure all services propagate trace context. Instrument async and background jobs.
Further Reading
- Structured logging guide: the Complete Guide to Structured Logging goes deeper on formats and libraries.
- Pipeline setup: the Log Aggregation Guide covers shipping, retention tiers, and platform selection end-to-end.
- Runnable configs: the log aggregation recipe has a Fluentd → Elasticsearch Compose stack to test against.
- Public postmortems: worth reading — real incidents from teams that hit similar problems in production.
Production Notes
Roll the standard out gradually: start with one service, fix the schema surprises it surfaces, then scale to the rest. Keep the document versioned with a visible changelog, since silent edits are exactly how services drift apart. Revisit the redaction rules every quarter against new endpoints and fields, and check the Troubleshooting section before filing pipeline bugs — most log gaps have a boring cause.
Key Takeaways
- A logging standard is only as good as its enforcement: shared wrapper, CI schema check, and a named owner.
- One naming convention across every field — snake_case here — or cross-service queries fall apart.
- ERROR must be actionable and FATAL must page; anything less belongs at INFO or WARN.
- Never log secrets, and redact personal data before it reaches the log stream.
Common Production Pitfalls
- Leaving required fields empty, or filling them with vague one-word answers.
- Filling the document once and never updating it after scope or decisions change.
- Storing the document where the team doesn’t look during incidents or reviews.
- Not assigning an owner, due date, or review cadence.
- Copying boilerplate without removing sections that don’t apply.
- Skipping version control, which makes rollback and accountability impossible.
- Failing to link the document to related decisions or follow-up actions.
- Avoiding quarterly reviews that would retire stale or unused sections.
Frequently Asked Questions
Should we log in production at DEBUG level?
No, DEBUG should be off by default. Enable it temporarily for targeted troubleshooting, and disable it when the issue is resolved.
What is a correlation ID?
A correlation ID is a unique identifier passed through every service that handles a single request. Without one, stitching together everything that request touched across a distributed system is guesswork.
How do we handle sensitive data in logs?
Use an allowlist approach: only log fields that are explicitly approved, and redact or tokenize sensitive values before they reach the log stream.
How do we implement correlation IDs in a microservices architecture?
Generate a correlation ID at the API gateway for each incoming request. Propagate it via HTTP headers (e.g., X-Correlation-Id). Each downstream service reads the header, includes it in all log entries, and passes it to further downstream calls. For asynchronous messages, include the correlation ID in message metadata. For background jobs, store it in job context. Use middleware or interceptors to automate propagation so developers don't need to handle it manually. Ensure the correlation ID appears in error reports and monitoring alerts.
What is log sampling and when should we use it?
Log sampling means logging only a percentage of events to reduce volume and cost. Use sampling for high-volume, low-value logs (e.g., health check responses, static asset requests). Never sample ERROR or security logs. Common strategies: random sampling (log 1 in 100), rate-based (log first N per second), or tail-based (log all errors, sample successes). Fluentd or Logstash can do the sampling at the collector level. Document the sampling rate and ensure it doesn't hide important patterns.
How do we handle log storage costs?
Control costs through: tiered retention (hot storage for 7-30 days, cold archive for longer), sampling high-volume logs, compressing log files, excluding noisy endpoints from logging, and using structured logging to enable efficient querying. Set up billing alerts for log storage. Review log volume monthly and identify top contributors. Consider using a dedicated log archival solution (S3 Glacier, Azure Archive Storage) for long-term retention instead of expensive hot log platforms.
What's the difference between logs, metrics, and traces?
Logs are discrete events with timestamps — what happened at a specific moment. Metrics are aggregated measurements over time — CPU usage, request count, error rate. Traces follow a single request across service boundaries — the path and timing of a request through the system. The three of them are the classic pillars of observability: logs answer "what happened," metrics answer "how much," and traces answer "where did the time go." Use all three for complete observability.
How do we test logging in CI/CD?
Cover the basics in tests: emitted logs parse as valid JSON, the required fields are there (timestamp, level, service, correlation_id), no sensitive data leaks in, levels are used correctly, and volume stays under thresholds. Capture log output in the test suite and validate it against the schema. Run log parsing tests to ensure the aggregation pipeline can ingest the logs. Include logging tests in the deployment checklist.
Related Resources
Monitoring and Alerting Policy Template
A policy template that defines how alerts are configured, routed, escalated, and reviewed across services and infrastructure.
DocRunbook Template
A reusable template for operational runbooks: incident response, deployment procedures, and routine tasks.
GuideStructured Logging: JSON Logs, Correlation IDs, Aggregation
Structured logging with JSON output, correlation IDs, and log levels. Python structlog, Node.js pino, Java SLF4J, plus ELK and Loki aggregation stacks.
GuideComplete Guide to Log Aggregation
A practical guide to log aggregation: structured logging, shipping strategies, retention policies, and building searchable log pipelines with ELK, Loki, and cloud-native solutions.
RecipeLog Aggregation with ELK, Fluentd, and Loki
Centralize logs from distributed services with ELK, Fluentd, and Loki for search, alerting, and troubleshooting in production.
PatternStructured Logging: Emit JSON Logs with Consistent Fields
How to emit structured JSON logs with consistent fields for searchability. Covers Python structlog, Winston, pino, Serilog, log levels, and log aggregation.