Complete Guide to Observability with the Grafana Stack
Set up the full Grafana observability stack: Prometheus, Loki, Tempo, Alloy and Grafana with a single docker-compose, correlation between signals, and SLO alerting.
Introduction
3 a.m., a deploy just broke checkout, and the evidence you need is split across three places: a latency spike in Prometheus, an error in Loki, a trace in Tempo. If those signals don’t share labels and IDs, you’re copying trace IDs into a search bar by hand — while the error budget burns.
This guide is for backend and SRE engineers standing up the full Grafana stack — the LGTM stack (Loki, Grafana, Tempo, Metrics) — self-hosted with Docker Compose. You’ll get one unified docker-compose.yml with all six services, working instrumentation with OpenTelemetry, provisioned datasources and dashboards, and signal correlation that actually resolves. Prerequisites: Docker Compose and at least one HTTP service you want to instrument. If you’re deploying on Kubernetes the configs still apply, but the provisioning layer changes to Helm or the Grafana Operator.
When to Use (and When Not To)
| Your situation | Recommendation |
|---|---|
| Microservices (>3) with on-call responsibility | Self-hosted LGTM — full control, data stays in your infra |
| Small team, no ops capacity | Grafana Cloud free tier (10k metrics, 50GB logs) — skip self-hosting |
| Quick local experimentation | grafana/otel-lgtm all-in-one image — the stack in one container |
| Monolith, low traffic | Prometheus + Grafana only — Loki/Tempo add real memory/storage overhead |
| Production scale / long retention | This guide’s compose is a starting point — plan S3 object storage, Mimir, and multi-node Loki/Tempo before calling it production |
The rest of this guide builds the middle option: full self-hosted LGTM for teams who own their observability.
The Stack at a Glance
Each signal answers a different question:
| Pillar | Tool | Question it answers |
|---|---|---|
| Metrics | Prometheus | ”Is it broken?” — aggregated numbers over time |
| Logs | Loki | ”What happened?” — event detail, searchable |
| Traces | Tempo | ”Where is it broken?” — per-request causality |
| Visualization | Grafana | Unifies all three in dashboards and correlation |
| Collection | Alloy | Ships logs to Loki (replaces EOL Promtail) |
| Alerting | Alertmanager | Routes Prometheus alerts to humans |
One thing that bites every team: unify label names from day one (service, environment, version). Correlation between signals only works when service="checkout" means the same thing in metrics, logs, and traces.
Unified Docker Compose
The three services below are one file — this is the part most tutorials split apart. A single compose with all services, shared volumes, and the ports you’ll actually touch:
# docker-compose.yml — full LGTM stack in one file
services:
prometheus:
image: prom/prometheus:latest
ports: ["9090:9090"]
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./alert_rules.yml:/etc/prometheus/alert_rules.yml
- prometheus-data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=15d"
loki:
image: grafana/loki:latest
ports: ["3100:3100"]
volumes:
- loki-data:/loki
command: -config.file=/etc/loki/local-config.yaml
alloy:
image: grafana/alloy:latest
volumes:
- ./alloy/config.alloy:/etc/alloy/config.alloy
- /var/run/docker.sock:/var/run/docker.sock:ro # required for docker discovery
- /var/lib/docker/containers:/var/lib/docker/containers:ro
command: ["run", "--server.http.listen-addr=0.0.0.0:12345", "/etc/alloy/config.alloy"]
tempo:
image: grafana/tempo:latest
ports: ["3200:3200", "4318:4318"]
volumes:
- ./tempo.yaml:/etc/tempo.yaml
- tempo-data:/tmp/tempo
grafana:
image: grafana/grafana:latest
ports: ["3000:3000"]
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true # dev only — remove in real setups
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- ./grafana/dashboards:/var/lib/grafana/dashboards
- grafana-data:/var/lib/grafana
alertmanager:
image: prom/alertmanager:latest
ports: ["9093:9093"]
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
volumes:
prometheus-data:
loki-data:
tempo-data:
grafana-data:
For production you’d swap latest for pinned versions and disable anonymous auth, but this boots a complete signal pipeline end to end.
Prometheus: Metrics
Prometheus is the metrics store — it pulls (scrapes) /metrics endpoints on an interval, stores them as time series, and evaluates alert rules. Everything is driven by prometheus.yml:
# prometheus.yml — scrape config
global:
scrape_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
rule_files:
- alert_rules.yml
scrape_configs:
- job_name: "services"
metrics_path: "/metrics"
static_configs:
- targets: ["app:8080"] # your instrumented services
labels:
environment: "production"
- job_name: "node-exporter" # host-level CPU/disk for USE method
static_configs:
- targets: ["node-exporter:9100"]
Instrumenting an app means exposing /metrics with a client library — prometheus_client for Python, prom-client for Node, Micrometer’s Prometheus registry for Java. The exporter guide (python-prometheus-metrics-exporter) covers the code; the key habit is counters with labels, not counters per endpoint string — a label on a counter is one series; a new metric name per endpoint is a cardinality bomb.
Two PromQL forms worth memorizing — the second one is the one tutorials get wrong:
# Error rate per service over 5m
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
# p95 latency per endpoint — the by-clause goes on the sum, not on histogram_quantile
histogram_quantile(0.95, sum by (endpoint, le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile is a function, not an aggregator: you must sum by (…, le) inside it, keeping the le label, or the percentile math is wrong.
Loki + Alloy: Logs
Loki is the log store — it indexes only the labels (not the log content), which is why it’s cheap to run: log volume costs storage, not index. Promtail was its collector, but Promtail reached end-of-life on March 2, 2026; its replacement is Grafana Alloy — same OTel-based collector that also ships metrics and traces.
The Alloy config that replaces promtail.yml — discover Docker containers and push to Loki:
// alloy/config.alloy — discover docker containers, forward logs to Loki
discovery.docker "linux" {
host = "unix:///var/run/docker.sock"
}
discovery.relabel "logs" {
targets = discovery.docker.linux.targets
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/(.*)"
target_label = "service" // strip leading / and keep the label name consistent with metrics
}
}
loki.source.docker "default" {
host = "unix:///var/run/docker.sock"
targets = discovery.relabel.logs.targets
relabel_rules = discovery.relabel.logs.rules
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint { url = "http://loki:3100/loki/api/v1/push" }
}
Two details that silently break Docker log collection: the Alloy container must mount /var/run/docker.sock (it’s in the compose above), and __meta_docker_container_name includes a leading / — strip it with the regex: "/(.*)" rule or your service label won’t match the metrics one.
Query Loki with LogQL from Grafana’s Explore:
# error logs for one service
{service="checkout"} |= "error"
# error log rate per minute
sum(rate({service="checkout"} |= "error" [5m]))
# JSON logs — parse fields, filter on level, extract trace_id
{service="checkout"} | json | level="error" | line_format "{{.msg}} trace={{.trace_id}}"
Write JSON logs with trace_id — that’s the field that makes trace→log correlation work later.
Tempo: Traces
Tempo is the trace store — it receives OTLP spans, indexes by trace ID only, and stores them cheap. tempo.yaml declares the receivers (OTLP + legacy Jaeger for migrations):
# tempo.yaml — receivers and local storage
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
http:
grpc:
jaeger:
protocols:
thrift_http:
storage:
trace:
backend: local
local:
path: /tmp/tempo/blocks
wal:
path: /tmp/tempo/wal
Your apps push spans via OpenTelemetry to http://tempo:4318/v1/traces. A Flask service looks like this:
# otel_tracing.py — Flask + OTLP HTTP exporter
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.flask import FlaskInstrumentor
resource = Resource.create({"service.name": "checkout", "service.version": "1.4.2"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://tempo:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
FlaskInstrumentor().instrument_app(app) # service.name must match the metrics/log label
The service.name resource attribute is your correlation key — same value as the service label in Prometheus and Alloy.
Grafana: Unification Layer
Grafana is where the three stores meet. Provision everything as code — datasources and dashboards load at startup, versioned in Git:
# grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
uid: prometheus # explicit uid — correlation config references it
- name: Loki
type: loki
access: proxy
url: http://loki:3100
uid: loki
- name: Tempo
type: tempo
access: proxy
url: http://tempo:3200
uid: tempo
jsonData:
tracesToLogsV2: # legacy tracesToLogs needs datasourceUid present
datasourceUid: loki
spanStartTimeShift: "-1h"
spanEndTimeShift: "1h"
filterByTraceID: true
tracesToMetrics:
datasourceUid: prometheus
tags: [{ key: "service.name", value: "service" }]
Without explicit uid fields, correlation silently breaks — Grafana generates random UIDs and tracesToLogsV2.datasourceUid: loki resolves to nothing.
# grafana/provisioning/dashboards/dashboards.yml — load dashboards from disk
apiVersion: 1
providers:
- name: "default"
folder: ""
type: file
options:
path: /var/lib/grafana/dashboards
A provisionable dashboard JSON needs uid, schemaVersion, and gridPos per panel — the minimal shape Grafana accepts on import. Build it in the UI, export, then check it into grafana/dashboards/.
Alerting
Prometheus evaluates rules and fires to Alertmanager, which groups, dedupes, and routes. Alert on symptoms users feel (error rate, latency SLO), not on causes (CPU, disk) — causes go to dashboards.
# alert_rules.yml — error-rate + burn-rate alerts
groups:
- name: api
rules:
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service) > 0.05
for: 5m
labels: {severity: warning}
annotations:
summary: "Error rate >5% on {{ $labels.service }}"
- alert: FastErrorBudgetBurn # 2% of budget burned in 1h at 99.9% SLO
expr: sum(rate(http_requests_total{status=~"5.."}[1h])) by (service)
/ sum(rate(http_requests_total[1h])) by (service) > 0.05 * (1 - 0.999) * 14.4 / 0.0005
for: 2m
labels: {severity: critical}
annotations:
summary: "Fast burn of error budget on {{ $labels.service }}"
# alertmanager.yml — route warnings to Slack, criticals to on-call
route:
receiver: slack
group_by: ["service", "severity"]
group_wait: 30s
repeat_interval: 4h
routes:
- match: {severity: critical}
receiver: pagerduty
receivers:
- name: slack
slack_configs:
- api_url: "https://hooks.slack.com/services/XXX" # your webhook
channel: "#alerts"
- name: pagerduty
pagerduty_configs:
- service_key: "XXX"
Test rules before they page you — promtool test rules alert_rules.yml runs them against fixtures locally.
Scaling to Production
The compose above is a solid dev/small-prod setup, but three seams separate it from real production:
- Object storage: Loki and Tempo are designed for S3/GCS/Azure Blob, not local disk. Moving
storage.trace.backendand Loki’sstorage_configto object storage removes your biggest retention constraint — local volumes fill and force deletion. - Metrics long-term: Prometheus’s local TSDB is 15-day storage by design. For longer retention or multi-cluster querying, the same ecosystem has Mimir (Grafana’s drop-in Prometheus long-term store) or Thanos — PromQL stays identical, the backend changes.
- Ingestion layer: putting an OpenTelemetry Collector between your apps and Tempo (apps → Collector → Tempo) gives you tail sampling — “keep only traces with errors or p99 latency” instead of keeping everything. That’s what makes trace storage affordable at volume; sampling at the app level keeps 100% of noise.
On Kubernetes, the same mental model applies — you’d swap the compose file for the lgtm-distributed Helm chart or the individual charts per component, and docker_sd for kubernetes_sd_configs.
A note on sampling strategy while you’re scaling: head sampling (decide at the app, before the span finishes) is cheap but throws away exactly the traces you want — the slow, errored ones. Tail sampling (decide at the Collector, after the whole trace completes) costs more memory because it buffers traces, but lets you keep 100% of errors and a small percentage of the boring traffic. Most teams start with “always keep error traces + keep ~10% of the rest” and tune from there — watch the Collector’s own memory metric while you tune, since buffered traces are where it burns.
The Collector config that does that — apps point their OTLP exporter at it instead of at Tempo directly:
# otel-collector-config.yaml — tail sampling: errors + 10% of the rest
receivers:
otlp:
protocols:
http:
grpc:
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: keep-errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: keep-slow
type: latency
latency: { threshold_ms: 2000 }
- name: keep-10-percent
type: probabilistic
probabilistic: { sampling_percentage: 10 }
exporters:
otlphttp:
endpoint: http://tempo:4318/v1/traces
service:
pipelines:
traces:
receivers: [otlp]
processors: [tail_sampling]
exporters: [otlphttp]
Policies are OR-ed — a trace kept by any policy is forwarded. That’s why errors and slow traces always survive while only a slice of healthy traffic makes it to storage.
SLO-Driven Observability
The stack’s real payoff: alerts tied to objectives, not guesses. Three terms that get conflated:
| Term | Definition | Example |
|---|---|---|
| SLI | Service Level Indicator — the metric itself | p99 latency < 300ms |
| SLO | Service Level Objective — target over a window | 99.9% of requests < 300ms over 30 days |
| SLA | The contractual commitment (usually looser than SLO) | 99.5% with penalties |
The error budget is what your SLO allows you to spend: a 99.9% availability SLO is 0.1% failures = ~43 min/month of allowed downtime. Burn it slowly and you keep shipping; burn it fast (the FastErrorBudgetBurn alert above) and you freeze deploys and work on reliability. That trade-off is the difference between observability as decoration and observability as an operating system for deploys.
For the SLO document template and the alerting policy that goes with it, see the SLO template and the monitoring and alerting guide.
Correlating Signals
Correlation is the whole point: a user reports “checkout fails,” and you want metric → trace → log in three clicks, not three searches:
- Metric → trace: exemplars on histograms link
p99points to a specifictrace_id(enable withtracesToMetrics+ exemplar storage on Prometheus). - Trace → log:
tracesToLogsV2on the Tempo datasource filters Loki forservice=X+trace_id=Y— works because your JSON logs carrytrace_idand the label is unified. - Log → trace: derived fields on the Loki datasource turn a
trace_idin a log line into a clickable link to Tempo.
That last leg needs one config nobody includes — derivedFields on the Loki datasource:
- name: Loki
type: loki
url: http://loki:3100
uid: loki
jsonData:
derivedFields:
- name: TraceID
matcherRegex: '"trace_id":\\s*"([a-f0-9]+)"' # regex with one capture group = the trace id
url: "$${__value.raw}" # $$ escapes provisioning interpolation
datasourceUid: tempo # where the link jumps to
Each field only extracts when it captures one group — get the regex wrong and nothing errors, the button just doesn’t appear.
That chain only works if service means the same thing everywhere — which is why label discipline is a best practice, not a nice-to-have.
Best Practices
- Instrument with OpenTelemetry, not vendor agents — the day you move to a managed backend, the instrumentation doesn’t change.
- Cap label cardinality deliberately: a rule of thumb is to keep each metric under ~10 label values per dimension and avoid unbounded values like
user_idorrequest_id— those belong in logs and traces, not time series. - Set retention per signal, not globally: metrics 15d locally + downsample for longer, logs by what an incident actually needs, traces with sampling — full-fidelity traces on every request get expensive fast.
- Keep dashboards as code; every dashboard a human edits in the UI and never exports is one you’ll rebuild after an upgrade.
- Alert on SLO burn rate and user-facing symptoms; route everything else to dashboards or silent channels.
Common Mistakes
- Treating the three signals as three separate projects — the stack’s value is correlation, and correlation fails on inconsistent labels before it fails on missing pieces.
- Still using Promtail — it’s EOL since March 2026; migrate to Alloy rather than pinning a dead component.
- High-cardinality labels on metrics (
user_id,order_id, raw URLs) — the most common way to blow up Prometheus memory; checkprometheus_tsdb_head_series. - Alerting on causes instead of symptoms — “CPU > 80%” trains the team to ignore alerts; “error budget burn” trains them to act.
- No retention or sampling policy — naive setups keep everything at full fidelity until storage forces the conversation at the worst time.
- Skipping
uidfields in provisioned datasources — correlation configs silently resolve to nothing. - Rate() on non-counters —
rate()over a gauge or a raw histogram produces nonsense; counters only.
See Also
- Complete Guide to Prometheus and Grafana — the metrics pillar in depth: PromQL, exporters, dashboard design
- Complete Guide to Monitoring and Alerting — alerting strategy, SLO process, and the policy side of the stack
- Distributed Tracing Guide — deeper OpenTelemetry and trace design
- Log Aggregation — Centralize, Search — Loki alternatives and centralized log design
- Alert Management Guide — escalation, runbooks, alert fatigue
- Companion files — runnable docker-compose and all configs from this guide
Further Reading
- Grafana LGTM stack docs — the official naming and component docs (Loki, Tempo, Mimir/Prometheus, Grafana)
- Grafana Alloy documentation — the Promtail replacement;
discovery.docker,loki.source.docker,loki.write - Prometheus docs — alerting rules and PromQL —
histogram_quantileaggregation form and rule testing - OpenTelemetry docs — OTLP endpoints, BatchSpanProcessor, resource attributes that drive correlation
- Google SRE Workbook — Alerting on SLOs — the burn-rate math behind
FastErrorBudgetBurn
Frequently Asked Questions
What's the difference between metrics, logs, and traces?
Metrics are aggregated numbers over time ("error rate is 4%"). Logs are individual events with detail ("this request threw NullPointerException at line 42"). Traces follow a single request across services, showing the causal chain and timing of each hop. Metrics detect, traces locate, logs explain.
Should I use Promtail or Grafana Alloy?
Alloy. Promtail reached end-of-life on March 2, 2026 — Alloy is its replacement and covers logs, metrics, and traces collection in one agent. Existing promtail.yml configs translate to Alloy component syntax roughly 1:1.
How do I correlate a trace with its logs?
Write JSON logs including trace_id, configure tracesToLogsV2 on the Tempo datasource with datasourceUid: loki (with an explicit uid on the Loki datasource), and keep the service label identical across all three signals. Then a span in Grafana has a "Logs for this span" button.
When is the full LGTM stack overkill?
For a monolith or a service without on-call, Prometheus + Grafana is enough — Loki and Tempo add storage and operational cost. For a team with zero ops capacity, Grafana Cloud's free tier (10k metrics, 50GB logs) beats a self-hosted stack you can't maintain. For local experiments, the grafana/otel-lgtm all-in-one image is the fastest path.
How do I sample traces without losing the important ones?
Head sampling in the app (keep every Nth trace) is cheap but drops slow and errored traces along with the boring ones — exactly the ones you'll need during an incident. The common pattern is an OTel Collector doing tail sampling: keep 100% of traces containing errors or above a latency threshold, plus a fixed percentage of the rest. That costs Collector memory but makes trace storage viable at volume.
How do I keep Prometheus memory under control?
Cardinality. Every unique label combination is a series kept in memory; user_id, request_id, or raw URL labels explode the head block. Keep label values bounded, check prometheus_tsdb_head_series, and drop high-cardinality metrics with metric_relabel_configs before they ingest.
Related Resources
Prometheus and Grafana: Metrics, Dashboards, Alerting
Instrument services with Prometheus metrics, query them in PromQL, build Grafana dashboards, and ship alerting rules that don't flap.
GuideDistributed Tracing: OpenTelemetry, Jaeger, Zipkin
Master distributed tracing with OpenTelemetry, Jaeger, and Zipkin. Trace propagation across services, span context, sampling strategies, and production debugging.
PatternHealth Check Pattern: Expose Liveness and Readiness Probes
Implement liveness and readiness probes for container orchestrators. Covers Kubernetes probe config, dependency checks, and graceful degradation.
PatternCircuit Breaker with Monitoring
How to expose circuit breaker state as metrics for observability. Covers Prometheus integration, alerting rules, dashboards, and state transitions.
GuideSentry: Error Tracking, Triage, and Resolution
Master Sentry for production error tracking. Covers SDK integration in Python, Node.js, and Java, release tracking, source maps, performance monitoring, and alerting.
GuideComplete Guide to Monitoring and Alerting
Build a production monitoring stack with Prometheus, Grafana, and AlertManager — RED/USE methods, alert design, SLOs with error budgets, and runbooks.