Logging, Monitoring & Observability Guide

Introduction

When something breaks in production, the first question is always the same: why? Without observability, answering it means SSH-ing into a box, grepping text logs, and guessing. With it, you open a dashboard, paste a trace ID, and see exactly where the request slowed down or failed.

Observability is the ability to understand a system’s internal state by examining its outputs. It isn’t a tool you install — it’s a property you build into the system through three complementary signals: logs (what happened), metrics (how it’s trending), and traces (where the time went). Each answers a different question, and the value comes from correlating them: a spike in an error metric leads you to a trace, and the trace’s ID leads you to the logs that explain the failure.

This guide covers the practical side: how to structure logs, which metrics to collect, how to instrument traces with OpenTelemetry, how to alert on SLOs instead of CPU, and how the pieces fit together in a real microservices setup. A solid runbook complements all of this — see the Runbook Template for the operational side.

Who is this for? Engineers running a production service who want observability that actually shortens incidents — not a wall of dashboards nobody reads. You don’t need a specific stack: everything here uses OpenTelemetry, Prometheus and open formats that work with any backend, self-hosted or managed.

The Three Pillars

Pillar	Question	Granularity	Retention
Logs	What happened?	High (individual events)	Days to weeks
Metrics	How is it trending?	Low (aggregated)	Months to years
Traces	Where did time go?	Medium (request paths)	Days to weeks

No pillar replaces the others. Metrics are cheap to store and fast to query, but they’re pre-aggregated — you can’t drill into a single bad request. Logs keep every detail but are expensive to store and slow to search at scale. Traces sit in between: one record per request, with timing for each step.

The glue is the correlation ID (usually the trace ID). Stamp it on every log line, include it in error responses, and expose it as a metric label where cardinality allows. With that single ID you can jump from an alert to a trace to the exact log lines of the failed request.

Structured Logging

Replace free-form text with machine-parseable JSON. "Payment failed for user 456" forces you to write regexes forever; {"level": "error", "user_id": "user_456", "error": "Card declined"} is a query. See Structured Logging for the standalone recipe.

Format

Every log entry should carry a consistent set of fields:

{
  "timestamp": "2026-06-11T14:32:01Z",
  "level": "ERROR",
  "message": "Payment failed",
  "service": "billing-api",
  "trace_id": "abc123",
  "user_id": "user_456",
  "amount": 99.99,
  "error": "Card declined",
  "duration_ms": 245
}

The non-negotiables: timestamp (ISO-8601, UTC), level, service, and trace_id. Everything else is context — add what you’d want when debugging at 3 AM.

Implementation (Python, structlog)

import structlog
import logging

structlog.configure(
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.JSONRenderer()
    ],
    context_class=dict,
    logger_factory=structlog.stdlib.LoggerFactory(),
)

logger = structlog.get_logger()
logger.info("payment_processed", user_id="123", amount=49.99)

Log Levels

Level	Use Case	Example
DEBUG	Development detail	Variable values, loop iterations
INFO	Normal operations	Request completed, job started
WARN	Unexpected but handled	Retry attempted, deprecated API used
ERROR	Failed operation	Request failed, exception caught
FATAL	System unavailability	Database connection lost

Two rules of thumb: everything at INFO is as useless as nothing logged, and ERROR should mean “a human probably needs to look at this.” If ERROR fires constantly and nobody acts, your level assignment is wrong.

Sampling for high-volume services

A service doing 50K requests/minute can’t afford to log every request at full detail. Sample by outcome: keep 100% of errors and slow requests, 1–5% of successful ones. Most backends (Loki, Datadog, CloudWatch) support head-based sampling rules; the Parse Log Files recipe shows the other end — querying what’s already stored.

Metrics

Metrics are numeric data points collected over time. Unlike logs, they aggregate beautifully: rate(http_requests_total{status="500"}[5m]) works the same whether you serve 10 or 10 million requests per minute.

Metric Types

Type	Description	Example
Counter	Only increases	Requests served, errors occurred
Gauge	Can go up or down	Current queue size, memory usage
Histogram	Distribution of values	Request duration, payload size
Summary	Calculated percentiles	p95 latency, p99 latency

Prefer histograms over summaries when you’ll aggregate across instances — Prometheus can compute percentiles from histogram buckets across many pods, while summary percentiles can’t be merged.

Implementation (Prometheus, Python)

from prometheus_client import Counter, Histogram, start_http_server

requests_total = Counter('http_requests_total', 'Total requests', ['method', 'status'])
request_duration = Histogram('http_request_duration_seconds', 'Request duration')

@request_duration.time()
def handle_request():
    requests_total.labels(method='GET', status='200').inc()
    # ... process request

start_http_server(8000)  # Exposes /metrics

The cardinality trap

Labels multiply your series count. user_id as a label on a counter creates one time series per user — with a million users, your metrics backend falls over. Keep label cardinality low (dozens, not thousands): status, method, endpoint are fine; user_id, email, request_id are not. High-cardinality identifiers belong in logs and traces.

Which metrics to collect first

Two frameworks cover most cases. For services, use the RED method: Rate (requests/second), Errors (failed requests/second), Duration (latency distribution). Three metrics per endpoint give you a working dashboard and cover most alert conditions. For resources, use the USE method: Utilization, Saturation, Errors — how busy the resource is, how much work is queued, and whether it’s failing.

See the Prometheus Monitoring & Alerts recipe for scrape config and rule examples.

Distributed Tracing

Traces follow a request across multiple services. A trace is a tree of spans — one per unit of work — sharing a common trace ID and linked by parent span IDs.

Trace ID: abc123
├── Service A: 5ms  (HTTP request received)
├── Service B: 12ms (Auth check)
├── Service C: 45ms (Database query)
│   ├── Connection acquire: 2ms
│   ├── Query execution: 30ms
│   └── Result mapping: 13ms
└── Service D: 8ms  (Response formatting)

That single view already tells you the 45ms is in Service C’s query — not “something is slow somewhere.” The Distributed Tracing guide goes deeper into propagation and backend choice.

Implementation (OpenTelemetry, Python)

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

span_processor = BatchSpanProcessor(OTLPSpanExporter())
trace.get_tracer_provider().add_span_processor(span_processor)

with tracer.start_as_current_span("process_payment") as span:
    span.set_attribute("payment.amount", 99.99)
    span.set_attribute("payment.currency", "USD")
    # ... business logic

OpenTelemetry is the standard to bet on: one instrumentation layer that can export to Jaeger, Tempo, Zipkin, or a vendor backend without code changes. Use BatchSpanProcessor in production (it buffers and sends spans asynchronously); the simple processor exports synchronously and adds latency to the request path.

For trace context to propagate between services, the outgoing HTTP/gRPC call must inject the trace headers (traceparent, tracestate) and the receiving service must extract them. OTel’s auto-instrumentation handles this for common frameworks and HTTP clients — that’s the main reason to prefer it over hand-rolled middleware.

Alerting

Alert on symptoms, not causes. “p99 latency above 1s for 10 minutes” pages someone; “CPU at 80%” probably doesn’t — the box might be fine doing its job.

Alerting Rules

# Prometheus alerting rule
groups:
  - name: api_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.service }}"

The for: 2m clause matters more than it looks: it filters out momentary blips and is the cheapest defense against alert fatigue.

Alert Severity Levels

Severity	Response Time	Example
Critical	Immediate (page)	Service down, data loss risk, SLO burning fast
Warning	Within 1 hour	Elevated error rate, high latency
Info	Next business day	Capacity approaching limit, cert expiring in 30d

Every alert should answer three questions: is a user affected right now, can a human do something about it, and is the runbook entry linked in the annotation? If any answer is no, it should be a ticket or a dashboard panel, not a page.

Finally, give every alert an owner. A “critical” alert routed to a channel nobody watches is worse than no alert — it trains the team to ignore the channel.

Cost, Retention, and Sampling

Observability bills scale with volume, and volume grows faster than traffic as you add services. A few numbers worth keeping in mind:

• Logs dominate cost. Full-fidelity logging at 500K requests/minute means terabytes per week. Combine short retention for DEBUG/INFO (7 days) with longer retention for ERROR and audit logs (30–90 days), and sample successful requests rather than storing all of them.
• Metrics are cheap per data point but explode with cardinality. Ten labels with unbounded values can produce more series than a year of normal traffic. Cap label values and drop unused metrics.
• Traces are the middle ground. Head-based sampling (keep N% of all traces) is simple but throws away the interesting requests; tail-based sampling in the OTel Collector keeps all errors plus a fraction of successes, usually the right default.

Budget rule of thumb: if observability costs more than 5–10% of your compute bill, something is misconfigured — usually log volume or metric cardinality.

SLOs and Error Budgets

An SLO is a measurable target for reliability: “99.9% of requests succeed in under 500ms over a rolling 30 days.” The gap between your SLO and 100% is the error budget — 0.1% of requests, or 43.2 minutes of downtime per month.

The budget turns reliability into a management tool. Budget left? Ship faster, take risks, run experiments. Budget burned? Freeze features and fix reliability. It’s the most honest answer to “can we deploy this week?”

Burn rate alerting

Burn rate measures how fast you’re consuming the budget. If the SLO allows 0.1% errors and you’re currently at 1.4%, you’re burning at 14x — the whole month’s budget in about 2 hours. Google’s SRE workbook recommends two-tier alerting: page on a fast burn (14x over 1 hour, budget gone soon) and ticket on a slow burn (2x over 3 days, chronic problem).

- alert: SloBurnRateFast
  expr: rate(http_requests_total{status=~"5.."}[1h]) > 14.4 * 0.001
  labels:
    severity: critical
- alert: SloBurnRateSlow
  expr: rate(http_requests_total{status=~"5.."}[6h]) > 2 * 0.001
  labels:
    severity: warning

Detailed Scenario: Observability for E-commerce Microservices

Pulling it together: a 15-service e-commerce platform at 500K requests/minute. Payments go through checkout → payment-service → gateway, and each hop is a span. Every log line carries the traceId, so when the payments.errors metric spikes, on-call can search that ID in Loki for the log context and in Jaeger for the full request path — three queries, one incident, minutes instead of hours.

System: 15 microservices, 500K requests/min
Stack: OpenTelemetry -> Jaeger (traces), Prometheus (metrics), Loki (logs)

Instrumentation (Node.js):
  const { trace, metrics } = require("@opentelemetry/api");
  const tracer = trace.getTracer("payment-service");
  const meter = metrics.getMeter("payment-service");
  const paymentsTotal = meter.createCounter("payments.total");
  const paymentsErrors = meter.createCounter("payments.errors");

  async function processPayment(payment) {
    const span = tracer.startSpan("processPayment");
    span.setAttribute("payment.amount", payment.amount);
    span.setAttribute("payment.currency", payment.currency);
    try {
      const result = await gateway.charge(payment);
      span.setAttribute("payment.status", result.status);
      paymentsTotal.add(1, { status: result.status, gateway: "stripe" });
      return result;
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: 2, message: error.message });
      paymentsErrors.add(1, { type: error.constructor.name });
      throw error;
    } finally {
      span.end();
    }
  }

Structured logs (JSON):
  {
    "timestamp": "2026-01-15T10:30:00Z",
    "level": "error",
    "service": "payment-service",
    "traceId": "abc123",
    "spanId": "def456",
    "message": "Payment failed",
    "paymentId": "pay_789",
    "amount": 99.99,
    "currency": "USD",
    "error": "InsufficientFunds"
  }

  // Correlation: traceId connects logs, metrics, and traces
  // Search traceId in Loki -> see all request logs
  // Search traceId in Jaeger -> see full trace

SLO dashboard:
  | SLO | Target | Metric |
  |-----|--------|--------|
  | Availability | 99.9% | http_requests_total{status!~5..} / total |
  | Latency p99 | < 500ms | histogram_quantile(0.99, http_duration_bucket) |
  | Error rate | < 0.1% | http_requests_total{status=~5..} / total |
  | Throughput | > 10K/s | rate(http_requests_total[5m]) |

Alerts (user-facing symptoms):
  - Error rate > 1% for 5 min -> page on-call
  - p99 latency > 1s for 10 min -> page on-call
  - SLO burn rate > 14x in 1h -> page on-call
  - Throughput < 5K/s for 5 min -> ticket (no page)

Lessons:
  - OpenTelemetry unifies traces, metrics, and logs
  - traceId is the way to correlate everything
  - Structured JSON logs > plain text
  - Alert on SLOs, not on infrastructure metrics
  - The collector decouples app from observability backend

The collector sits between your app and the backends, which is what makes swapping Jaeger for Tempo (or adding a second backend) a config change instead of a redeploy:

The runnable version of this setup (instrumentation + collector config + docker-compose lab) lives in the companion repo — linked in Further Reading below.

What Works

• Use correlation IDs: pass trace_id through every service call, log line, and error response
• Log at boundaries: entry/exit of requests, jobs, and transactions. The handoffs are where things fail
• Avoid logging sensitive data: no passwords, tokens, or PII. Scrub at the processor level, not by discipline
• Set SLOs and error budgets: define what “good” means and measure against it; see the monitoring & alerting guide
• Alert fatigue is real: page only for issues that are urgent and actionable; everything else is a ticket
• Instrument once, export anywhere: standardize on OpenTelemetry and let the collector route to backends. Swapping Jaeger for Tempo later is a config change, not a rewrite
• Adopt in order: start with structured logs, add RED metrics next, then tracing. Each layer builds on the previous one’s correlation IDs

Common Mistakes

• Logging everything at INFO level: the signal drowns in noise and nobody trusts ERROR anymore
• Metrics without labels (no dimensions to slice by) or with unbounded labels (user_id) that blow up cardinality
• Alerting on CPU usage instead of user-facing symptoms: the box being busy is not the user being hurt
• Storing logs indefinitely without a retention policy: storage costs grow linearly, value decays fast
• Instrumenting after the first incident: retrofitting correlation IDs across 15 services is weeks of work
• Keeping dashboards nobody opens: if a panel hasn’t been looked at in a quarter, delete it

Further Reading

• Runnable companion — OTel instrumentation, collector config, and a docker-compose lab for the e-commerce scenario
• OpenTelemetry docs: instrumentation, collector, and semantic conventions
• Prometheus documentation: metric types, PromQL, and alerting rules
• Google SRE Workbook — Alerting on SLOs: the burn-rate model used in this guide
• structlog documentation: structured logging for Python