StackPractices
beginner By Mathias Paulenko

Logging Standards Document Template

A document template for defining structured logging conventions, log levels, retention, and observability requirements across services.

Overview

A Logging Standards Document is the contract that says how your services, applications, and infrastructure should emit logs. When every service logs the same way, debugging, monitoring, security investigation, and compliance all get easier. This template covers log levels, structured formats, required fields, retention, sampling, and security rules.

Fill in each section, assign an owner and a review cadence, and keep the document next to your runbooks — a standard nobody can find is a standard nobody follows.

When to Use

Reach for this document when you’re onboarding a new service or development team, consolidating logs from several systems into a central observability platform, or getting ready for a security audit or compliance review. It’s also the right artifact after a production incident where the logs turned out incomplete or inconsistent, and when you’re setting the logging strategy for microservices or serverless environments.

For a broader observability strategy, see the Complete Guide to Observability with the Grafana Stack.

Prerequisites

  • Somewhere for the logs to land: ELK, Splunk, Datadog, Grafana Loki, CloudWatch, or similar.
  • One timestamp standard and one timezone policy, agreed across the org.
  • A shared definition of the events that can’t afford to go missing — the ones that always have to reach the log stream.
  • Agreement on sensitive data classification and log redaction rules.

Solution

Document

1. Log Levels

LevelUseExample
DEBUGDetailed diagnostic information during developmentcache miss for key user:1234
INFONormal application eventsuser logged in, order completed
WARNUnexpected but recoverable situationsconnection timeout, retrying
ERRORFailures that affect operationpayment gateway returned 500
FATALCritical failures requiring immediate attentiondatabase unavailable, service shutdown

Guidelines:

  • DEBUG must be off in production by default.
  • INFO is the default production level for most services.
  • ERROR must trigger an alert or ticket.
  • FATAL must page the on-call team.

2. Structured Log Format

Emit every log entry as JSON carrying these required fields:

FieldTypeDescriptionExample
timestampISO 8601Event time in UTC2026-06-27T14:30Z
levelstringLog levelINFO
servicestringService or application namepayment-service
environmentstringEnvironmentproduction
messagestringHuman-readable summaryOrder 12345 completed
correlation_idstringRequest trace IDabc-123-def
span_idstringOpenTelemetry span IDspan-xyz-789

Optional fields:

  • user_id: Identity of the user associated with the event.
  • tenant_id: Identifier for multi-tenant isolation.
  • duration_ms: Time taken to complete an operation.
  • error_code: Stable error code for programmatic handling.
  • source_file: File and line where the log was emitted.

Naming matters more than it looks. Pick one convention — this template uses snake_case — and apply it everywhere, including fields nested inside context objects. A service logging correlation_id and another logging correlationId can’t be queried together, and the failure is silent.

3. Required Event Categories

CategoryEvents to LogLevel
AuthenticationLogin, logout, failed login, MFA challengeINFO / WARN
AuthorizationAccess denied, permission escalationWARN
Data changesCreate, update, delete on sensitive recordsINFO
ErrorsExceptions, external failures, retriesERROR
PerformanceSlow queries, high latency, timeoutsWARN
SecuritySuspicious activity, rate limit hits, blocked requestsWARN
OperationalStartup, shutdown, configuration changesINFO
BusinessOrder placed, payment received, workflow completedINFO

4. Sensitive Data and Redaction

Data TypeLoggedRedaction
PasswordsNeverRedact or exclude
Credit card numbersNeverTokenize or exclude
API keysNeverRedact or exclude
Personal namesWith approvalMask if not required
Email addressesAllowedPartial mask for non-admins
IP addressesAllowedAllowed for security logs
User IDsAllowedAllowed

Rules:

  • Never log secrets or credentials.
  • Use allowlists for personal data fields.
  • Redact or tokenize values before logging.
  • Encrypt logs if they contain sensitive data.

5. Retention and Sampling

Log TypeRetentionSamplingNotes
Application logs30 days100%Keep all for debugging
Security logs1 year100%Compliance requirement
Audit logs7 years100%Legal and regulatory
Debug logs7 days100%Only when enabled
Trace logs14 days1%High volume, cost control

6. Log Aggregation and Transport

RequirementRule
TransportSend logs to the central platform with backpressure handling.
OrderingUse timestamps for ordering; tolerate minor clock skew.
BufferingBuffer locally if the collector is unavailable.
EncodingUse UTF-8 JSON.
BackupsReplicate critical logs to a secondary storage.
AlertingRoute ERROR and FATAL logs to the alerting system.

Explanation

Consistent logging turns noisy text files into searchable, structured data. By pinning down levels, fields, and retention up front, teams can correlate events across services, investigate incidents faster, and answer compliance questions without archaeology. Structured logs also plug straight into tracing and metrics — a correlation_id is the join key between all three signals.

The document only works if it’s enforced, not just written — see Enforcing the Standard below.

Structured Log Format Example

{
  "timestamp": "2026-07-11T10:55:32.123Z",
  "level": "ERROR",
  "service": "auth-service",
  "environment": "production",
  "correlation_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "user_id": "usr_8f7e6d5c",
  "message": "JWT validation failed",
  "error_code": "AUTH_JWT_INVALID",
  "context": {
    "endpoint": "/api/v1/auth/verify",
    "method": "POST",
    "status_code": 401,
    "duration_ms": 12,
    "ip_address": "10.0.1.42",
    "user_agent": "MobileApp/2.3.1"
  },
  "error": {
    "type": "TokenExpiredError",
    "message": "jwt expired",
    "stack": "TokenExpiredError: jwt expired\n    at ..."
  }
}

Log Level Decision Matrix

=== When to Use Each Log Level ===

FATAL   - System cannot continue. Process will exit.
          Examples: config load failure, port binding error, OOM

ERROR   - Operation failed but system continues.
          Examples: request failed, DB query error, external API timeout

WARN    - Unexpected but recoverable condition.
          Examples: retry succeeded, cache miss, deprecated API usage

INFO    - Significant business or operational event.
          Examples: user login, order placed, deployment started

DEBUG   - Diagnostic detail for troubleshooting.
          Examples: variable state, query params, cache contents

TRACE   - Finest-grained execution flow.
          Examples: function entry/exit, loop iteration count

RULES:
  - Production: INFO and above (DEBUG/TRACE off)
  - Staging: DEBUG and above
  - Development: TRACE and above
  - Never log at DEBUG in production unless actively debugging
  - ERROR must be actionable — if it isn't, it's INFO

Enforcing the Standard

A written standard without enforcement drifts within a quarter. Three cheap mechanisms keep it alive:

  • Shared logging wrapper. Ship a library per language that hardcodes the required fields, applies the naming convention, and exposes the redaction hooks. Nobody should have to keep the schema in their head.
  • CI schema check. A test that asserts emitted logs are valid JSON and contain timestamp, level, service, environment, message, and correlation_id catches format regressions before merge. Add a regex pass over common secret patterns (Bearer, password=, AWS key formats) while you’re at it.
  • Versioning and ownership. Give the document an owner, a changelog section, and a review cadence (quarterly works). Field additions are backwards-compatible; renames are a breaking change that deserves a migration note.

The failure mode to watch for is the write-once-forget document: six months in, half the services log correlation_id, the other half settled on trace_id, and nobody remembers which name their dashboards still query. That is why enforcement matters more than wording — put the standard where code review happens, not in a wiki page nobody opens. If a reviewer can block a PR for logging a password or dropping correlation_id, the standard survives contact with reality. If it can’t, the document is decoration.

Variants

  • Cloud logging standards: adjusted to AWS CloudWatch, Azure Monitor, or Google Cloud Logging.
  • Container and Kubernetes logging: Covers sidecar log shippers, Fluentd, and pod log conventions.
  • Security-focused logging: puts audit events, integrity, and tamper detection first.
  • Serverless logging: deals with short-lived functions, cold starts, and centralized collection.
  • Mobile or client logging: Focuses on privacy, batching, and offline buffering.

What works

The teams that get value from a logging standard share a few habits. They pick one structured format and use it in every service, carry a correlation ID through every request so distributed tracing actually traces, and log outcomes at business boundaries instead of narrating each internal step. Messages stay short — context lives in the structured fields, not in the sentence.

They also treat log levels as a contract: an ERROR means someone should act, which is what keeps alerts worth reading. Sensitive data never reaches the stream by default, retention gets reviewed against real storage bills and compliance needs, and log parsing plus alert rules get exercised during deployments like any other part of the release.

Common Mistakes

The classic failure is logging everything at INFO: real problems drown in routine noise and alerts stop meaning anything. Close behind come unparsable plain-text logs, missing or inconsistent timestamps, and services that each invent their own field names — correlation_id in one, correlationId in the next — so cross-service queries silently match nothing.

The other recurring sins are about data and retention: passwords or tokens ending up in the stream, not enough context to reproduce a failure after the fact, and keeping every log forever while the storage bill quietly climbs.

Troubleshooting

  • No logs for a failing request: verify log shipping, retention, and that the request reached the service.
  • Fields missing or renamed after a deploy: check the shared wrapper version — a library bump often changes the schema silently.
  • Duplicate log lines in the index: look for two shippers tailing the same file, or a direct-shipping logger running alongside the agent.
  • Alert fires but the service is healthy: tune thresholds and use multi-signal alerts.
  • Log volume exploded after a release: check for a new INFO log inside a hot loop or a retry path; enforce sampling on the noisy category.
  • Trace is incomplete across services: ensure all services propagate trace context. Instrument async and background jobs.

Further Reading

  • Structured logging guide: the Complete Guide to Structured Logging goes deeper on formats and libraries.
  • Pipeline setup: the Log Aggregation Guide covers shipping, retention tiers, and platform selection end-to-end.
  • Runnable configs: the log aggregation recipe has a Fluentd → Elasticsearch Compose stack to test against.
  • Public postmortems: worth reading — real incidents from teams that hit similar problems in production.

Production Notes

Roll the standard out gradually: start with one service, fix the schema surprises it surfaces, then scale to the rest. Keep the document versioned with a visible changelog, since silent edits are exactly how services drift apart. Revisit the redaction rules every quarter against new endpoints and fields, and check the Troubleshooting section before filing pipeline bugs — most log gaps have a boring cause.

Key Takeaways

  • A logging standard is only as good as its enforcement: shared wrapper, CI schema check, and a named owner.
  • One naming convention across every field — snake_case here — or cross-service queries fall apart.
  • ERROR must be actionable and FATAL must page; anything less belongs at INFO or WARN.
  • Never log secrets, and redact personal data before it reaches the log stream.

Common Production Pitfalls

  • Leaving required fields empty, or filling them with vague one-word answers.
  • Filling the document once and never updating it after scope or decisions change.
  • Storing the document where the team doesn’t look during incidents or reviews.
  • Not assigning an owner, due date, or review cadence.
  • Copying boilerplate without removing sections that don’t apply.
  • Skipping version control, which makes rollback and accountability impossible.
  • Failing to link the document to related decisions or follow-up actions.
  • Avoiding quarterly reviews that would retire stale or unused sections.

Frequently Asked Questions

Should we log in production at DEBUG level?

No, DEBUG should be off by default. Enable it temporarily for targeted troubleshooting, and disable it when the issue is resolved.

What is a correlation ID?

A correlation ID is a unique identifier passed through every service that handles a single request. Without one, stitching together everything that request touched across a distributed system is guesswork.

How do we handle sensitive data in logs?

Use an allowlist approach: only log fields that are explicitly approved, and redact or tokenize sensitive values before they reach the log stream.

How do we implement correlation IDs in a microservices architecture?

Generate a correlation ID at the API gateway for each incoming request. Propagate it via HTTP headers (e.g., X-Correlation-Id). Each downstream service reads the header, includes it in all log entries, and passes it to further downstream calls. For asynchronous messages, include the correlation ID in message metadata. For background jobs, store it in job context. Use middleware or interceptors to automate propagation so developers don't need to handle it manually. Ensure the correlation ID appears in error reports and monitoring alerts.

What is log sampling and when should we use it?

Log sampling means logging only a percentage of events to reduce volume and cost. Use sampling for high-volume, low-value logs (e.g., health check responses, static asset requests). Never sample ERROR or security logs. Common strategies: random sampling (log 1 in 100), rate-based (log first N per second), or tail-based (log all errors, sample successes). Fluentd or Logstash can do the sampling at the collector level. Document the sampling rate and ensure it doesn't hide important patterns.

How do we handle log storage costs?

Control costs through: tiered retention (hot storage for 7-30 days, cold archive for longer), sampling high-volume logs, compressing log files, excluding noisy endpoints from logging, and using structured logging to enable efficient querying. Set up billing alerts for log storage. Review log volume monthly and identify top contributors. Consider using a dedicated log archival solution (S3 Glacier, Azure Archive Storage) for long-term retention instead of expensive hot log platforms.

What's the difference between logs, metrics, and traces?

Logs are discrete events with timestamps — what happened at a specific moment. Metrics are aggregated measurements over time — CPU usage, request count, error rate. Traces follow a single request across service boundaries — the path and timing of a request through the system. The three of them are the classic pillars of observability: logs answer "what happened," metrics answer "how much," and traces answer "where did the time go." Use all three for complete observability.

How do we test logging in CI/CD?

Cover the basics in tests: emitted logs parse as valid JSON, the required fields are there (timestamp, level, service, correlation_id), no sensitive data leaks in, levels are used correctly, and volume stays under thresholds. Capture log output in the test suite and validate it against the schema. Run log parsing tests to ensure the aggregation pipeline can ingest the logs. Include logging tests in the deployment checklist.