Grafana Dashboards for Observability with Prometheus

Overview

A Grafana dashboard turns a heap of Prometheus metrics into a live, readable view of your services. This recipe walks through wiring a data source, building a dashboard with panels and variables, provisioning it from disk, and adding alerts. Examples mix YAML, JSON, PromQL, and Terraform snippets you can drop into your own stack.

If you’re already collecting metrics with Prometheus, Grafana is the visualization layer that sits on top. It doesn’t store metrics itself; it queries Prometheus (or Loki for logs) and renders them as panels, graphs, and stat tiles. Think of it as the front-end for your observability stack. I’ve been using it for years and it’s the fastest way I know to give a team visibility.

When to Use

This is useful when your team wants one screen for request rate, latency, and error rate across the fleet, with on-call engineers spotting failing services quickly and non-technical stakeholders getting uptime visibility without writing PromQL. It also works when you want dashboards stored as code and rolled out through Git.

I’ve used this setup on teams where the on-call rotation included people who didn’t know PromQL. A well-built dashboard with template variables let them filter by service and see what was breaking without touching a query. That’s the real value: you write the PromQL once, and everyone else gets a button. I’ve seen it work with teams of 5 people and teams of 50; the dynamic is the same.

Solution

The flow is straightforward: Prometheus scrapes metrics from your services, Grafana queries Prometheus through a provisioned data source, and dashboards render those queries as panels. Alerts evaluate PromQL expressions and route notifications through Grafana or Alertmanager.

1. Provision the Prometheus data source

# provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

2. Build the dashboard JSON

{
  "dashboard": {
    "title": "API Service Overview",
    "tags": ["api", "production"],
    "timezone": "utc",
    "panels": [
      {
        "title": "Request Rate",
        "type": "timeseries",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[5m])) by (route)",
            "legendFormat": "{{ route }}"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "reqps",
            "min": 0
          }
        },
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
      },
      {
        "title": "P95 Latency",
        "type": "timeseries",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))",
            "legendFormat": "{{ route }}"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "s",
            "custom": {
              "drawStyle": "line",
              "lineWidth": 2
            }
          }
        },
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }
      },
      {
        "title": "Error Rate",
        "type": "stat",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100",
            "legendFormat": "Error %"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "percent",
            "thresholds": {
              "steps": [
                { "color": "green", "value": 0 },
                { "color": "yellow", "value": 1 },
                { "color": "red", "value": 5 }
              ]
            }
          }
        },
        "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 }
      }
    ]
  }
}

3. Add template variables

{
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "query": "label_values(http_requests_total, job)",
        "multi": true,
        "includeAll": true
      },
      {
        "name": "route",
        "type": "query",
        "query": "label_values(http_requests_total{job=~\"$service\"}, route)",
        "multi": true,
        "includeAll": true
      },
      {
        "name": "interval",
        "type": "interval",
        "options": [
          { "text": "1m", "value": "1m" },
          { "text": "5m", "value": "5m" },
          { "text": "1h", "value": "1h" }
        ],
        "current": { "text": "5m", "value": "5m" }
      }
    ]
  }
}

4. Provision dashboards from disk

# provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: default
    orgId: 1
    folder: Services
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

5. Manage dashboards as code with Terraform

# terraform/grafana.tf
resource "grafana_dashboard" "api" {
  config_json = jsonencode({
    title = "API Overview"
    panels = [
      {
        title = "Request Rate"
        type  = "timeseries"
        targets = [{
          expr = "sum(rate(http_requests_total[5m]))"
        }]
      }
    ]
  })
}

6. Add Grafana alerts

# provisioning/alerting/alerts.yml
apiVersion: 1
groups:
  - orgId: 1
    name: API Health
    interval: 30s
    rules:
      - uid: api-error-rate
        title: API Error Rate > 5%
        condition: A
        data:
          - refId: A
            relativeTimeRange:
              from: 300
            datasourceUid: prometheus
            model:
              expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
              instant: true
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          summary: "Error rate above 5%"
        labels:
          severity: critical
        notification_settings:
          group_by: ['alertname']
          group_wait: 10s

7. Include Loki log panels

# provisioning/datasources/loki.yml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    isDefault: false
    jsonData:
      maxLines: 500

# Error logs for a specific service
{service="api"} |= "error" | json | line_format "{{.msg}}"

# Slow requests (>1s)
{service="api"} |= "duration" | json | duration > 1000

Explanation

Each piece of the dashboard has a specific job. Panels turn PromQL queries into tables, graphs, gauges, or stat tiles. Variables let people filter by service, route, or interval without touching the query text. Rows group related panels into collapsible sections and keep the layout tidy.

Provisioning pulls dashboards from disk whenever Grafana starts, so they live in Git and rollbacks stay simple. Terraform manages the dashboard as real infrastructure, just like the rest of your stack. Alerts run PromQL expressions and send notifications through Grafana or Alertmanager. Loki adds log context next to the metrics, so a spike on a graph leads straight to the lines that caused it.

One thing I learned the hard way: don’t mix provisioning methods. If you provision dashboards from disk AND manage them with Terraform, you’ll get conflicts on restart. Pick one and stick with it. I prefer file-based provisioning for dashboards (simpler, Git-native) and Terraform for data sources and alert rules (lifecycle management matters more there).

I also ran into a subtle issue with dashboard folders. When you provision dashboards from disk, the foldersFromFilesStructure flag expects the folder layout to match the file structure on disk. If someone creates a folder in the UI with the same name, Grafana silently merges them, and the provisioned dashboard can overwrite the manual one. I spent an afternoon debugging that one. The fix is to namespace your folder names or disable allowUiUpdates entirely.

The Grafana provisioning docs cover the full YAML schema. The Prometheus query docs are essential for writing PromQL that performs well, especially around rate functions and histogram quantiles, which trip up a lot of people. I still reference those docs every few weeks when I’m tuning a slow panel.

Variants

Node Exporter system dashboard

# CPU usage
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes

# Disk I/O
rate(node_disk_io_time_seconds_total[5m])

Recording rules for expensive queries

- record: job:http_p99:5m
  expr: histogram_quantile(0.99, sum by(job, le)(rate(http_request_duration_seconds_bucket[5m])))

Best Practices

Store dashboards in Git and provision them at startup, so you get pull-request reviews and an easy rollback path. Match the refresh interval to the use case: five seconds when you’re live-debugging, thirty seconds to one minute for overviews.

Limit variables and label cardinality. A variable that lists every pod in a large cluster can drag query performance down. Recording rules pay for themselves when the same expensive PromQL shows up on several dashboards. Use $__rate_interval rather than a fixed window so the query tracks the zoom. Cap Loki maxLines at a few hundred to avoid dumping huge result sets into the browser.

I also recommend naming conventions. Prefix dashboard titles with the service name (API: Request Rate, API: Error Budget) so they sort together in the dashboard picker. Tag dashboards consistently (production, staging, on-call) so people can filter. These seem trivial, but on a team with 50+ dashboards, they save real time. I went crazy searching for dashboards before I adopted this convention; now I find any dashboard in seconds.

For distributed tracing, consider adding a Tempo data source alongside Loki. You can link from a Grafana panel directly to a trace, which closes the loop between metrics, logs, and traces. I used this during an incident where the metric showed high latency but didn’t say why; the link to the trace took me straight to the service that was failing.

Common Mistakes

Fifty panels on one dashboard is too many, and the page gets sluggish and unreadable. Copying a dashboard per service instead of using variables makes maintenance explode. Forgetting thresholds on stat and gauge panels leaves healthy and failing values looking the same. Querying months of data on an overview dashboard is wasteful, so set a sensible default range. If you leave dashboards editable in the UI after they’re provisioned, any change disappears on the next restart.

Another mistake I see: teams forget to set editable: false on provisioned data sources. Someone tweaks the URL in the UI, it works until the next restart, then breaks silently. Lock it down. It happened to me during an incident where we lost 20 minutes looking for the problem until we discovered someone had changed the data source URL in the UI.

Don’t put business logic in dashboard alerts. If an alert rule needs a 20-line PromQL expression with nested subqueries, move that computation into a recording rule first. The alert becomes a simple threshold check on the recorded metric, which is easier to read, test, and debug. I learned this after having an alert that nobody understood how it worked; when I simplified it with a recording rule, the whole team could reason about it.

Summary

Grafana dashboards turn Prometheus metrics into actionable views. Wire a data source, build panels with template variables, provision from disk so dashboards live in Git, and add alerts for the metrics that matter. Keep dashboards focused (under 20 panels), use recording rules for expensive queries, and lock down provisioned resources so UI changes don’t survive a restart. Pair with Loki for logs and you’ve got a full observability stack.

See Also

• Prometheus Monitoring and Alerts: set up alerting rules in Prometheus before wiring Grafana alerts
• Metrics Collection: instrument your services to expose Prometheus metrics in the first place
• Structured Logging: pair Grafana with Loki for log context alongside metrics
• Grafana Documentation: official reference for provisioning, panels, and alerting
• Prometheus Query Basics: PromQL fundamentals for writing dashboard queries