StackPractices
advanced By Mathias Paulenko

Complete Guide to Observability with the Grafana Stack

Set up the full Grafana observability stack: Prometheus, Loki, Tempo, Alloy and Grafana with a single docker-compose, correlation between signals, and SLO alerting.

Introduction

3 a.m., a deploy just broke checkout, and the evidence you need is split across three places: a latency spike in Prometheus, an error in Loki, a trace in Tempo. If those signals don’t share labels and IDs, you’re copying trace IDs into a search bar by hand — while the error budget burns.

This guide is for backend and SRE engineers standing up the full Grafana stack — the LGTM stack (Loki, Grafana, Tempo, Metrics) — self-hosted with Docker Compose. You’ll get one unified docker-compose.yml with all six services, working instrumentation with OpenTelemetry, provisioned datasources and dashboards, and signal correlation that actually resolves. Prerequisites: Docker Compose and at least one HTTP service you want to instrument. If you’re deploying on Kubernetes the configs still apply, but the provisioning layer changes to Helm or the Grafana Operator.

When to Use (and When Not To)

Your situationRecommendation
Microservices (>3) with on-call responsibilitySelf-hosted LGTM — full control, data stays in your infra
Small team, no ops capacityGrafana Cloud free tier (10k metrics, 50GB logs) — skip self-hosting
Quick local experimentationgrafana/otel-lgtm all-in-one image — the stack in one container
Monolith, low trafficPrometheus + Grafana only — Loki/Tempo add real memory/storage overhead
Production scale / long retentionThis guide’s compose is a starting point — plan S3 object storage, Mimir, and multi-node Loki/Tempo before calling it production

The rest of this guide builds the middle option: full self-hosted LGTM for teams who own their observability.

The Stack at a Glance

flowchart diagram: APP[

Each signal answers a different question:

PillarToolQuestion it answers
MetricsPrometheus”Is it broken?” — aggregated numbers over time
LogsLoki”What happened?” — event detail, searchable
TracesTempo”Where is it broken?” — per-request causality
VisualizationGrafanaUnifies all three in dashboards and correlation
CollectionAlloyShips logs to Loki (replaces EOL Promtail)
AlertingAlertmanagerRoutes Prometheus alerts to humans

One thing that bites every team: unify label names from day one (service, environment, version). Correlation between signals only works when service="checkout" means the same thing in metrics, logs, and traces.

Unified Docker Compose

The three services below are one file — this is the part most tutorials split apart. A single compose with all services, shared volumes, and the ports you’ll actually touch:

# docker-compose.yml — full LGTM stack in one file
services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml
      - prometheus-data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=15d"

  loki:
    image: grafana/loki:latest
    ports: ["3100:3100"]
    volumes:
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  alloy:
    image: grafana/alloy:latest
    volumes:
      - ./alloy/config.alloy:/etc/alloy/config.alloy
      - /var/run/docker.sock:/var/run/docker.sock:ro   # required for docker discovery
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: ["run", "--server.http.listen-addr=0.0.0.0:12345", "/etc/alloy/config.alloy"]

  tempo:
    image: grafana/tempo:latest
    ports: ["3200:3200", "4318:4318"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/tmp/tempo

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true   # dev only — remove in real setups
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
      - grafana-data:/var/lib/grafana

  alertmanager:
    image: prom/alertmanager:latest
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

volumes:
  prometheus-data:
  loki-data:
  tempo-data:
  grafana-data:

For production you’d swap latest for pinned versions and disable anonymous auth, but this boots a complete signal pipeline end to end.

Prometheus: Metrics

Prometheus is the metrics store — it pulls (scrapes) /metrics endpoints on an interval, stores them as time series, and evaluates alert rules. Everything is driven by prometheus.yml:

# prometheus.yml — scrape config
global:
  scrape_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - alert_rules.yml

scrape_configs:
  - job_name: "services"
    metrics_path: "/metrics"
    static_configs:
      - targets: ["app:8080"]           # your instrumented services
        labels:
          environment: "production"
  - job_name: "node-exporter"           # host-level CPU/disk for USE method
    static_configs:
      - targets: ["node-exporter:9100"]

Instrumenting an app means exposing /metrics with a client library — prometheus_client for Python, prom-client for Node, Micrometer’s Prometheus registry for Java. The exporter guide (python-prometheus-metrics-exporter) covers the code; the key habit is counters with labels, not counters per endpoint string — a label on a counter is one series; a new metric name per endpoint is a cardinality bomb.

Two PromQL forms worth memorizing — the second one is the one tutorials get wrong:

# Error rate per service over 5m
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

# p95 latency per endpoint — the by-clause goes on the sum, not on histogram_quantile
histogram_quantile(0.95, sum by (endpoint, le) (rate(http_request_duration_seconds_bucket[5m])))

histogram_quantile is a function, not an aggregator: you must sum by (…, le) inside it, keeping the le label, or the percentile math is wrong.

Loki + Alloy: Logs

Loki is the log store — it indexes only the labels (not the log content), which is why it’s cheap to run: log volume costs storage, not index. Promtail was its collector, but Promtail reached end-of-life on March 2, 2026; its replacement is Grafana Alloy — same OTel-based collector that also ships metrics and traces.

The Alloy config that replaces promtail.yml — discover Docker containers and push to Loki:

// alloy/config.alloy — discover docker containers, forward logs to Loki
discovery.docker "linux" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "logs" {
  targets = discovery.docker.linux.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "service"          // strip leading / and keep the label name consistent with metrics
  }
}

loki.source.docker "default" {
  host             = "unix:///var/run/docker.sock"
  targets          = discovery.relabel.logs.targets
  relabel_rules    = discovery.relabel.logs.rules
  forward_to       = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint { url = "http://loki:3100/loki/api/v1/push" }
}

Two details that silently break Docker log collection: the Alloy container must mount /var/run/docker.sock (it’s in the compose above), and __meta_docker_container_name includes a leading / — strip it with the regex: "/(.*)" rule or your service label won’t match the metrics one.

Query Loki with LogQL from Grafana’s Explore:

# error logs for one service
{service="checkout"} |= "error"

# error log rate per minute
sum(rate({service="checkout"} |= "error" [5m]))

# JSON logs — parse fields, filter on level, extract trace_id
{service="checkout"} | json | level="error" | line_format "{{.msg}} trace={{.trace_id}}"

Write JSON logs with trace_id — that’s the field that makes trace→log correlation work later.

Tempo: Traces

Tempo is the trace store — it receives OTLP spans, indexes by trace ID only, and stores them cheap. tempo.yaml declares the receivers (OTLP + legacy Jaeger for migrations):

# tempo.yaml — receivers and local storage
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:
    jaeger:
      protocols:
        thrift_http:

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
    wal:
      path: /tmp/tempo/wal

Your apps push spans via OpenTelemetry to http://tempo:4318/v1/traces. A Flask service looks like this:

# otel_tracing.py — Flask + OTLP HTTP exporter
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.flask import FlaskInstrumentor

resource = Resource.create({"service.name": "checkout", "service.version": "1.4.2"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://tempo:4318/v1/traces"))
)
trace.set_tracer_provider(provider)

FlaskInstrumentor().instrument_app(app)  # service.name must match the metrics/log label

The service.name resource attribute is your correlation key — same value as the service label in Prometheus and Alloy.

Grafana: Unification Layer

Grafana is where the three stores meet. Provision everything as code — datasources and dashboards load at startup, versioned in Git:

# grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    uid: prometheus             # explicit uid — correlation config references it
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo
    jsonData:
      tracesToLogsV2:           # legacy tracesToLogs needs datasourceUid present
        datasourceUid: loki
        spanStartTimeShift: "-1h"
        spanEndTimeShift: "1h"
        filterByTraceID: true
      tracesToMetrics:
        datasourceUid: prometheus
        tags: [{ key: "service.name", value: "service" }]

Without explicit uid fields, correlation silently breaks — Grafana generates random UIDs and tracesToLogsV2.datasourceUid: loki resolves to nothing.

# grafana/provisioning/dashboards/dashboards.yml — load dashboards from disk
apiVersion: 1
providers:
  - name: "default"
    folder: ""
    type: file
    options:
      path: /var/lib/grafana/dashboards

A provisionable dashboard JSON needs uid, schemaVersion, and gridPos per panel — the minimal shape Grafana accepts on import. Build it in the UI, export, then check it into grafana/dashboards/.

Alerting

Prometheus evaluates rules and fires to Alertmanager, which groups, dedupes, and routes. Alert on symptoms users feel (error rate, latency SLO), not on causes (CPU, disk) — causes go to dashboards.

# alert_rules.yml — error-rate + burn-rate alerts
groups:
  - name: api
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
              / sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "Error rate >5% on {{ $labels.service }}"

      - alert: FastErrorBudgetBurn        # 2% of budget burned in 1h at 99.9% SLO
        expr: sum(rate(http_requests_total{status=~"5.."}[1h])) by (service)
              / sum(rate(http_requests_total[1h])) by (service) > 0.05 * (1 - 0.999) * 14.4 / 0.0005
        for: 2m
        labels: {severity: critical}
        annotations:
          summary: "Fast burn of error budget on {{ $labels.service }}"
# alertmanager.yml — route warnings to Slack, criticals to on-call
route:
  receiver: slack
  group_by: ["service", "severity"]
  group_wait: 30s
  repeat_interval: 4h
  routes:
    - match: {severity: critical}
      receiver: pagerduty
receivers:
  - name: slack
    slack_configs:
      - api_url: "https://hooks.slack.com/services/XXX"   # your webhook
        channel: "#alerts"
  - name: pagerduty
    pagerduty_configs:
      - service_key: "XXX"

Test rules before they page you — promtool test rules alert_rules.yml runs them against fixtures locally.

Scaling to Production

The compose above is a solid dev/small-prod setup, but three seams separate it from real production:

  • Object storage: Loki and Tempo are designed for S3/GCS/Azure Blob, not local disk. Moving storage.trace.backend and Loki’s storage_config to object storage removes your biggest retention constraint — local volumes fill and force deletion.
  • Metrics long-term: Prometheus’s local TSDB is 15-day storage by design. For longer retention or multi-cluster querying, the same ecosystem has Mimir (Grafana’s drop-in Prometheus long-term store) or Thanos — PromQL stays identical, the backend changes.
  • Ingestion layer: putting an OpenTelemetry Collector between your apps and Tempo (apps → Collector → Tempo) gives you tail sampling — “keep only traces with errors or p99 latency” instead of keeping everything. That’s what makes trace storage affordable at volume; sampling at the app level keeps 100% of noise.

On Kubernetes, the same mental model applies — you’d swap the compose file for the lgtm-distributed Helm chart or the individual charts per component, and docker_sd for kubernetes_sd_configs.

A note on sampling strategy while you’re scaling: head sampling (decide at the app, before the span finishes) is cheap but throws away exactly the traces you want — the slow, errored ones. Tail sampling (decide at the Collector, after the whole trace completes) costs more memory because it buffers traces, but lets you keep 100% of errors and a small percentage of the boring traffic. Most teams start with “always keep error traces + keep ~10% of the rest” and tune from there — watch the Collector’s own memory metric while you tune, since buffered traces are where it burns.

The Collector config that does that — apps point their OTLP exporter at it instead of at Tempo directly:

# otel-collector-config.yaml — tail sampling: errors + 10% of the rest
receivers:
  otlp:
    protocols:
      http:
      grpc:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: keep-10-percent
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlphttp:
    endpoint: http://tempo:4318/v1/traces

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlphttp]

Policies are OR-ed — a trace kept by any policy is forwarded. That’s why errors and slow traces always survive while only a slice of healthy traffic makes it to storage.

SLO-Driven Observability

The stack’s real payoff: alerts tied to objectives, not guesses. Three terms that get conflated:

TermDefinitionExample
SLIService Level Indicator — the metric itselfp99 latency < 300ms
SLOService Level Objective — target over a window99.9% of requests < 300ms over 30 days
SLAThe contractual commitment (usually looser than SLO)99.5% with penalties

The error budget is what your SLO allows you to spend: a 99.9% availability SLO is 0.1% failures = ~43 min/month of allowed downtime. Burn it slowly and you keep shipping; burn it fast (the FastErrorBudgetBurn alert above) and you freeze deploys and work on reliability. That trade-off is the difference between observability as decoration and observability as an operating system for deploys.

For the SLO document template and the alerting policy that goes with it, see the SLO template and the monitoring and alerting guide.

Correlating Signals

Correlation is the whole point: a user reports “checkout fails,” and you want metric → trace → log in three clicks, not three searches:

  1. Metric → trace: exemplars on histograms link p99 points to a specific trace_id (enable with tracesToMetrics + exemplar storage on Prometheus).
  2. Trace → log: tracesToLogsV2 on the Tempo datasource filters Loki for service=X + trace_id=Y — works because your JSON logs carry trace_id and the label is unified.
  3. Log → trace: derived fields on the Loki datasource turn a trace_id in a log line into a clickable link to Tempo.

That last leg needs one config nobody includes — derivedFields on the Loki datasource:

  - name: Loki
    type: loki
    url: http://loki:3100
    uid: loki
    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: '"trace_id":\\s*"([a-f0-9]+)"'   # regex with one capture group = the trace id
          url: "$${__value.raw}"                        # $$ escapes provisioning interpolation
          datasourceUid: tempo                          # where the link jumps to

Each field only extracts when it captures one group — get the regex wrong and nothing errors, the button just doesn’t appear.

That chain only works if service means the same thing everywhere — which is why label discipline is a best practice, not a nice-to-have.

Best Practices

  • Instrument with OpenTelemetry, not vendor agents — the day you move to a managed backend, the instrumentation doesn’t change.
  • Cap label cardinality deliberately: a rule of thumb is to keep each metric under ~10 label values per dimension and avoid unbounded values like user_id or request_id — those belong in logs and traces, not time series.
  • Set retention per signal, not globally: metrics 15d locally + downsample for longer, logs by what an incident actually needs, traces with sampling — full-fidelity traces on every request get expensive fast.
  • Keep dashboards as code; every dashboard a human edits in the UI and never exports is one you’ll rebuild after an upgrade.
  • Alert on SLO burn rate and user-facing symptoms; route everything else to dashboards or silent channels.

Common Mistakes

  • Treating the three signals as three separate projects — the stack’s value is correlation, and correlation fails on inconsistent labels before it fails on missing pieces.
  • Still using Promtail — it’s EOL since March 2026; migrate to Alloy rather than pinning a dead component.
  • High-cardinality labels on metrics (user_id, order_id, raw URLs) — the most common way to blow up Prometheus memory; check prometheus_tsdb_head_series.
  • Alerting on causes instead of symptoms — “CPU > 80%” trains the team to ignore alerts; “error budget burn” trains them to act.
  • No retention or sampling policy — naive setups keep everything at full fidelity until storage forces the conversation at the worst time.
  • Skipping uid fields in provisioned datasources — correlation configs silently resolve to nothing.
  • Rate() on non-counters — rate() over a gauge or a raw histogram produces nonsense; counters only.

See Also

Further Reading

Frequently Asked Questions

What's the difference between metrics, logs, and traces?

Metrics are aggregated numbers over time ("error rate is 4%"). Logs are individual events with detail ("this request threw NullPointerException at line 42"). Traces follow a single request across services, showing the causal chain and timing of each hop. Metrics detect, traces locate, logs explain.

Should I use Promtail or Grafana Alloy?

Alloy. Promtail reached end-of-life on March 2, 2026 — Alloy is its replacement and covers logs, metrics, and traces collection in one agent. Existing promtail.yml configs translate to Alloy component syntax roughly 1:1.

How do I correlate a trace with its logs?

Write JSON logs including trace_id, configure tracesToLogsV2 on the Tempo datasource with datasourceUid: loki (with an explicit uid on the Loki datasource), and keep the service label identical across all three signals. Then a span in Grafana has a "Logs for this span" button.

When is the full LGTM stack overkill?

For a monolith or a service without on-call, Prometheus + Grafana is enough — Loki and Tempo add storage and operational cost. For a team with zero ops capacity, Grafana Cloud's free tier (10k metrics, 50GB logs) beats a self-hosted stack you can't maintain. For local experiments, the grafana/otel-lgtm all-in-one image is the fastest path.

How do I sample traces without losing the important ones?

Head sampling in the app (keep every Nth trace) is cheap but drops slow and errored traces along with the boring ones — exactly the ones you'll need during an incident. The common pattern is an OTel Collector doing tail sampling: keep 100% of traces containing errors or above a latency threshold, plus a fixed percentage of the rest. That costs Collector memory but makes trace storage viable at volume.

How do I keep Prometheus memory under control?

Cardinality. Every unique label combination is a series kept in memory; user_id, request_id, or raw URL labels explode the head block. Keep label values bounded, check prometheus_tsdb_head_series, and drop high-cardinality metrics with metric_relabel_configs before they ingest.