Metrics and Dashboards: A Prometheus & Grafana Guide

Overview

Metrics are numerical measurements collected over time that tell you how your systems behave. Dashboards visualize those metrics so patterns become visible at a glance. Together they’re the foundation of operational awareness: without them you’re guessing, with them you can spot trends, detect anomalies, and defend capacity decisions with data.

This guide covers the full pipeline — instrumenting your code, collecting and storing time series, designing dashboards people actually use, and wiring alerts on top. It goes deep on metric types, cardinality, and PromQL, which is where most teams get into trouble.

Two adjacent topics get their own guides and aren’t repeated here: the alerting lifecycle (escalation, on-call, alert fatigue) lives in the Alert Management Guide, and the three-pillar picture (metrics + logs + traces together) lives in the Observability Guide. If you need either of those, jump there — this guide stays focused on the metrics side.

A quick reality check on what “good” looks like: a service with 20-40 well-named metrics, a scrape config that discovers targets automatically, one golden-signals dashboard per service, an SLO with a written error-budget policy, and a handful of burn-rate alerts that page humans rarely enough that they still trust the pager. That’s it. Everything else — hundreds of dashboards, fifty alert rules, a custom monitoring platform — is optional decoration on that core.

When to Use

• You need to monitor system health and performance over time
• You want to detect trends before they become incidents
• Your team needs a shared operational picture
• You’re establishing SLOs and need to measure compliance
• You want to reduce MTTR with visual, queryable data instead of log diving

When not to use this guide: if your problem is that alerts fire too often, pages reach the wrong person, or on-call is burning people out, that’s an alerting-process problem — the Alert Management Guide covers it. If you’re choosing between metrics, logs, and traces for a specific debugging workflow, start with the umbrella guide instead.

Core Concepts

Concept	Description
Counter	A cumulative metric that only increases (requests served, errors)
Gauge	A metric that can go up or down (temperature, queue depth, CPU)
Histogram	Samples observations into configurable buckets (request duration)
Summary	Similar to histogram but calculates percentiles client-side
Cardinality	Number of unique time series (high cardinality = expensive)
SLI / SLO / SLA	Service Level Indicator, Objective, and Agreement
Recording rule	Pre-computed PromQL result stored as a new time series
Exemplar	A metric sample annotated with a trace ID for jumping to the request

One model worth internalizing early: every metric is a time series identified by its name plus its label set. http_requests_total{status="200", path="/orders"} and http_requests_total{status="500", path="/orders"} are two different series. Labels are how you slice data later — and also how you blow up storage.

Metric Types and When to Use Them

Type	Use Case	Example	Don’t Use For
Counter	Counting events	http_requests_total	Values that decrease
Gauge	Point-in-time values	memory_usage_bytes, queue_size	Rates or cumulative counts
Histogram	Distribution of values	request_duration_seconds	Exact percentile calculation (use summary)
Summary	Pre-computed percentiles	request_latency_quantile	When you need histogram heatmaps

The histogram-versus-summary decision trips people up. Histograms bucket observations server-side, so you can aggregate across instances — histogram_quantile(0.95, sum(rate(...[5m])) by (le)) works across a whole fleet. Summaries compute quantiles inside each process and can’t be aggregated afterward — a p95 of ten services’ p95s means nothing. Default to histograms; reach for summaries only for batch jobs that exist once and never get aggregated.

Cardinality: The Metric That Costs You

Cardinality multiplies silently. A counter with method (5 values), status (5 classes), and path (40 endpoints) produces up to 1,000 series per instance — before you’ve noticed the label explosion. Add pod (50 replicas) and region (3 zones) and you’re at 150,000 series for one metric. Prometheus eats this in RAM; SaaS vendors charge you for it directly.

The classic mistake is labeling by unbounded values: raw URLs, user IDs, session IDs, query parameters, customer emails. One team once shipped path labels containing full request URLs — including tracking parameters — and generated 2 million series in a weekend. The fix isn’t heroic instrumentation; it’s picking labels with bounded cardinality from the start.

Audit what you’re actually storing before optimizing:

# Top metric families by series count
topk(15, count by (__name__)({__name__=~".+"}))

# Series per label name (find the offender)
topk(10, count by (label_name)({__name__=~".+"}))

Mitigation is cheaper at ingestion than at the source: metric_relabel_configs can drop a label or drop the whole series before it’s stored. And if you truly need per-request detail — a specific user’s latency, a single order’s path — that’s what traces and logs are for, not metrics. The Distributed Tracing Guide covers that side.

Step-by-Step Metrics and Dashboards

The pipeline has five stages: instrument, collect, visualize, measure against objectives, and alert. Here’s how they fit together:

1. Instrument Your Applications

Expose metrics in a format your collector understands:

# Example: Python application metrics with Prometheus client
from prometheus_client import Counter, Histogram, Gauge, start_http_server
import time

# Define metrics
http_requests_total = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'status', 'path']
)

request_duration_seconds = Histogram(
    'request_duration_seconds',
    'HTTP request duration',
    ['method', 'path'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
)

active_connections = Gauge(
    'active_connections',
    'Number of active connections'
)

# Instrument your code
@app.route("/api/orders/<order_id>")
def get_order(order_id):
    start = time.time()
    active_connections.inc()

    try:
        order = fetch_order(order_id)
        http_requests_total.labels(method='GET', status='200', path='/api/orders').inc()
        return jsonify(order)
    except OrderNotFound:
        http_requests_total.labels(method='GET', status='404', path='/api/orders').inc()
        return jsonify({"error": "Not found"}), 404
    finally:
        request_duration_seconds.labels(method='GET', path='/api/orders').observe(time.time() - start)
        active_connections.dec()

# Expose metrics endpoint
start_http_server(8000)

// Example: Spring Boot with Micrometer
@Configuration
public class MetricsConfig {

    @Bean
    public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
        return registry -> registry.config()
            .commonTags("application", "orders-service");
    }
}

@Service
public class OrderService {
    private final Counter orderCounter;
    private final Timer orderTimer;

    public OrderService(MeterRegistry registry) {
        this.orderCounter = Counter.builder("orders.processed")
            .description("Total orders processed")
            .register(registry);
        this.orderTimer = Timer.builder("orders.processing.time")
            .description("Order processing time")
            .register(registry);
    }

    public Order processOrder(OrderRequest request) {
        return orderTimer.recordCallable(() -> {
            Order result = doProcess(request);
            orderCounter.increment();
            return result;
        });
    }
}

Instrumentation checklist:

• Instrument the four golden signals: latency, traffic, errors, saturation
• Add labels for dimensions you’ll filter by (service, environment, endpoint)
• Use consistent naming: unit suffix, total for counters, seconds for duration
• Avoid high-cardinality labels (user IDs, session IDs, request IDs)
• Measure business metrics (orders placed, payments processed) alongside technical metrics

One thing the checklist doesn’t capture: pick histogram buckets for your actual latency profile. If your p99 target is 300ms and your buckets are [0.01, 0.05, 0.1, 0.5, 1.0, 5.0], you can’t tell a 50ms request from a 499ms request — both land in the le="0.5" bucket. Put buckets around the thresholds you alert on.

Two deployment details worth deciding early. First, serve /metrics on a dedicated port (the Python example uses 8000) bound to an internal interface — the endpoint leaks your service’s internal state, so don’t expose it publicly or behind the same auth as user traffic. Second, keep the endpoint cheap: it gets scraped every 15 seconds, and a metrics endpoint that allocates heavily or hits the database will show up in your own latency graphs.

2. Collect and Store Metrics

Set up a metrics pipeline:

# Example: Prometheus scrape configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # Drop high-cardinality labels at ingestion
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_user_id]
        action: drop
        regex: true

What works for collection:

• Scrape every 10-30 seconds (faster for high-frequency changes)
• Use service discovery (Kubernetes, Consul, DNS) instead of static targets
• Run collectors in each region/zone to minimize latency
• Use remote write for long-term storage (Thanos, Cortex, VictoriaMetrics)
• Federation for hierarchical aggregation (edge → regional → global)

Recording rules deserve a mention because they fix a real problem: dashboards that take 30 seconds to load. If a panel runs histogram_quantile(0.95, sum(rate(request_duration_seconds_bucket[5m])) by (le, service)) across a million series, precompute it:

# recording-rules.yml — evaluated every 30s, stored as new series
groups:
  - name: orders_sli
    interval: 30s
    rules:
      - record: job_service:request_rate:5m
        expr: sum(rate(http_requests_total[5m])) by (job, service)
      - record: job:error_ratio:5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m]))

Recorded series become just another metric — dashboards and alerts query job:error_ratio:5m instead of re-computing the division. It also gives you a stable name for alerting, which helps when the underlying query has to change.

On retention: local Prometheus storage is a rolling window, not a data lake. 15 days at 15-second scrape intervals is a reasonable default for operational use; anything older gets remote-write to Thanos, Cortex, or VictoriaMetrics with downsampling — keep 15-second resolution for a week, 5-minute resolution for a quarter, 1-hour for a year. Query the right resolution for the question: you don’t need 15-second data to spot a monthly trend.

3. Build Useful Dashboards

Design dashboards that tell a story:

Dashboard Type	Purpose	Key Panels
Service overview	Health of a single service	Error rate, latency p95, throughput, resource usage
Golden signals	Cross-service health	RED metrics (Rate, Errors, Duration) per service
Business KPI	Impact on revenue/usage	Conversions, active users, transaction volume
Infrastructure	Cluster/node health	CPU, memory, disk, network across all nodes
Incident response	Drill-down during incidents	Detailed per-endpoint latency, error breakdown, logs

// Example: Grafana dashboard JSON snippet (simplified)
{
  "dashboard": {
    "title": "Orders Service - Golden Signals",
    "panels": [
      {
        "title": "Request Rate",
        "type": "timeseries",
        "targets": [{
          "expr": "sum(rate(http_requests_total{service=\"orders\"}[5m])) by (status)"
        }]
      },
      {
        "title": "Error Rate",
        "type": "stat",
        "targets": [{
          "expr": "sum(rate(http_requests_total{service=\"orders\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{service=\"orders\"}[5m]))"
        }],
        "thresholds": [
          {"value": 0.001, "color": "green"},
          {"value": 0.01, "color": "yellow"},
          {"value": 0.05, "color": "red"}
        ]
      },
      {
        "title": "Latency p95",
        "type": "timeseries",
        "targets": [{
          "expr": "histogram_quantile(0.95, sum(rate(request_duration_seconds_bucket{service=\"orders\"}[5m])) by (le))"
        }]
      }
    ]
  }
}

Dashboard design principles:

• Put the most important panels at the top-left
• Use consistent colors: green = good, yellow = warning, red = critical
• Add links to related dashboards, logs, and traces
• Keep the number of panels per dashboard under 20
• Use template variables for service, environment, and time range

A service overview dashboard that works looks roughly like this:

┌─ Golden signals ─────────────────────────────────────┐
│ Request rate │ Error ratio (stat) │ Latency p95     │
├─ Saturation ─────────────────────────────────────────┤
│ CPU / memory / queue depth                          │
├─ Business ───────────────────────────────────────────┤
│ Orders/min │ Payment success rate │ Active carts    │
├─ Drill-down (collapsed) ─────────────────────────────┤
│ Per-endpoint latency, per-pod breakdown, slow queries│
└──────────────────────────────────────────────────────┘

Business metrics deserve first-class panels. orders_per_minute and payment_success_rate matter more to your stakeholders than p99 latency, and they’re often the earliest incident detector — a deploy that keeps latency flat but halves order conversions is a broken deploy. Emit them from the same instrumentation (orders_total, payment_attempts_total{result}) rather than maintaining a separate analytics pipeline for ops questions.

Two anti-patterns to avoid. First, the “wall of graphs”: forty panels of raw CPU and memory that nobody reads because nothing answers “is this service healthy?”. Lead with SLI-style panels (error ratio, latency quantile, request rate), then drop a drill-down section below for engineers mid-incident. Second, dashboards that only exist for demos — a dashboard nobody opens during an alert is decoration. Check which dashboards get traffic; delete the rest.

3.5 PromQL Patterns That Actually Help

A handful of query shapes covers most dashboard and alert work:

Pattern	Query	Use
Rate of events	rate(http_requests_total[5m])	Throughput over time
Instantaneous rate	irate(http_requests_total[5m])	Volatile spikes (graphs only — noisy for alerts)
Change over window	increase(http_requests_total[1h])	Totals for reporting
Percentile	histogram_quantile(0.95, rate(..._bucket[5m]))	Latency SLIs
Top offenders	topk(5, ...)	”Which service is eating CPU?”
Missing data	absent(http_requests_total{job="orders"})	”Is the exporter down?”
Capacity forecast	predict_linear(disk_free_bytes[6h], 4*24*3600) < 0	”Will the disk fill in 4 days?”

Two gotchas worth memorizing: rate() needs at least two samples in the window (use [2*scrape_interval] minimum, and 5 minutes works better than 1 for most graphs). And sum(rate()) aggregates after the rate — rate(sum(...)) is almost never what you want because it rates a sum that already mixed series.

4. Define SLIs and SLOs

Translate metrics into reliability targets:

# Example: SLI queries for common objectives

# Availability SLI: % of successful requests
(
  sum(rate(http_requests_total{status!~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

# Latency SLI: % of requests under threshold
(
  sum(rate(request_duration_seconds_bucket{le="0.5"}[5m]))
  /
  sum(rate(request_duration_seconds_bucket{le="+Inf"}[5m]))
) * 100

# Error budget: remaining acceptable errors
# SLO: 99.9% availability
# Error budget: 0.1% of total requests per month
0.001 * sum(increase(http_requests_total[30d]))

Objective	SLI	SLO	Measurement Window
Availability	Successful requests / total requests	99.9%	30 days
Latency	Requests under 200ms / total requests	99% under 200ms	7 days
Error rate	Error responses / total responses	< 0.1%	1 hour
Throughput	Requests per second	> 1000 rps	5 minutes

SLOs without an error-budget policy are just numbers on a dashboard. The budget is what makes them actionable: at 99.9% you get 0.1% of requests to spend per window, and when the burn rate spikes you’re allowed to say “stop shipping, fix reliability first” with something more than a gut feeling. The Service Level Objective Template has the structure for writing that policy down.

Burn-rate alerting is the pattern that makes SLOs pageable instead of aspirational. The idea: don’t alert on the error ratio crossing a fixed line — alert on how fast the budget is being consumed. A 99.9% SLO allows ~43 minutes of full outage per month; an error rate that would exhaust the month’s budget in 1-2 days (burn rate ~14.4×) needs a page now, while a slow burn that would take a week (2×) deserves a ticket, not a wake-up call. The SRE workbook’s multi-window pattern checks the same burn rate across a short window (1h) and a long one (6h) so a blip that already recovered doesn’t page:

# Burn rate > 14.4x sustained across 1h AND 6h → page
(
  sum(rate(http_requests_total{status=~"5.."}[1h]))
    / sum(rate(http_requests_total[1h]))
) > (14.4 * 0.001)
and
(
  sum(rate(http_requests_total{status=~"5.."}[6h]))
    / sum(rate(http_requests_total[6h]))
) > (14.4 * 0.001)

Fast burn (page): 14.4× in 1h+6h. Slow burn (ticket): 2× in 1d+3d. Tune the multipliers to your SLO and tolerance for noise — the numbers above assume the 99.9%/0.1% budget from the table.

5. Create Meaningful Alerts

Alert on symptoms, not causes:

# Example: Prometheus alerting rules
groups:
  - name: service_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate in {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: LatencyDegradation
        expr: |
          histogram_quantile(0.95,
            sum(rate(request_duration_seconds_bucket[5m])) by (le)
          ) > 1.0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Latency p95 above 1s"

Alertmanager then routes each alert: page for critical, ticket for warning, digest for informational. A minimal routing config:

# alertmanager.yml — route by severity, silence duplicates
route:
  receiver: 'team-pager'
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
    - match:
        severity: warning
      receiver: 'slack-warnings'

inhibit_rules:
  - source_match:
      alertname: 'ServiceDown'
    target_match:
      severity: 'critical'
    equal: ['service']

Inhibition is what saves you during a cascading failure: when ServiceDown fires for orders, the ten HighErrorRate children on the same service get silenced instead of generating ten more pages. group_by batching helps too — one notification per incident, not one per alert.

Alert design principles:

• Alert on user-impacting symptoms (error rate, latency), not causes (disk full)
• Use for: duration to reduce noise (require sustained threshold breach)
• Add runbook links and dashboard links to every alert
• Severity levels: page for critical (user impact), ticket for warning (trending)
• Regularly review alert frequency and tune thresholds

One number that helps tune for:: how long can the symptom exist before a user notices? For a customer-facing error rate that’s maybe 2-5 minutes; for queue depth trending upward it could be an hour. Set for: just short of user-visible impact — long enough to skip blips, short enough to still matter.

What Works

• Name metrics consistently. service_unit format: orders_service_requests_total.
• Document your metrics. Every metric needs a description and unit.
• Use histograms over averages. Averages hide outliers; histograms show distribution.
• Cardinality is cost. Every unique label combination creates a new time series.
• Dashboards are for exploration, not monitoring. Alerts notify; dashboards investigate.
• Test your dashboards. Walk through incident scenarios to verify they provide answers.
• Keep the SLI dashboard boring. If it needs a legend to explain which line is good, rebuild it.

Common Mistakes

• High-cardinality metrics. Labeling by user ID or request ID explodes storage. If you need per-user detail, that’s what logs and traces are for.
• Alerting on everything. Too many alerts create noise and reduce response quality — each one should have an action attached.
• Missing units. A metric named latency is ambiguous — latency_seconds is clear.
• Averaging percentiles. You can’t average p95s across services. Use histograms.
• No aggregation rules. Raw high-frequency metrics overwhelm dashboards; aggregate first.
• Ignoring burn rate. An alert that fires at 1% errors for 5 minutes will wake you for a 30-second deploy blip. Burn-rate alerts catch sustained budget consumption instead.

Instrumentation Frameworks

Three ecosystems cover most codebases:

• Prometheus client libraries (Python prometheus_client, Go prometheus, Rust prometheus): the native path — direct, minimal, idiomatic per-language.
• Micrometer (JVM): vendor-neutral facade that feeds Prometheus, Datadog, or CloudWatch without code changes — standard in Spring Boot.
• OpenTelemetry: the modern pick for new services. One SDK emits metrics, traces, and logs; the Collector converts to Prometheus remote-write or OTLP for your backend. Slightly heavier setup, but you stop paying the “three SDKs” tax.

There’s also StatsD/DogStatsD for push-based, short-lived jobs — batch scripts and serverless functions that won’t live long enough to be scraped. It pre-aggregates client-side, which caps cardinality at the cost of losing per-instance visibility.

Dashboard-as-code is worth mentioning in the same breath: Grafana supports provisioning dashboards from JSON files, Jsonnet/grafonnet libraries, or Terraform. Store dashboards next to the service they monitor, review them in PRs like any other code — an undocumented hand-built dashboard is an orphan waiting to happen.

Variants

• Pull-based: Prometheus scrapes exporters (standard for Kubernetes)
• Push-based: StatsD, Telegraf, or application pushes to collector (better for short-lived jobs — the Pushgateway exists for this, use it sparingly)
• Cloud-native: AWS CloudWatch, Google Cloud Monitoring, Azure Monitor (managed, but vendor-specific)
• Enterprise: Datadog, New Relic, Dynatrace (rich capabilities, per-host pricing)
• OpenTelemetry: the newer instrumentation path — OTel SDKs emit metrics alongside traces, and collectors can convert to Prometheus format. Worth adopting for new services since one SDK covers all three pillars and exemplars come free, letting you jump from a latency spike straight to the trace that caused it.

Conclusion

Metrics and dashboards turn raw system data into operational intelligence. Instrument consistently, control cardinality before it controls your bill, design dashboards around the questions people actually ask, and alert on symptoms rather than causes. Do those four things and the observability stack starts paying rent — the alerting lifecycle and the rest of the three-pillar picture are covered in the guides linked above.

Further Reading

• Observability Guide — the three-pillar picture: metrics, logs, and traces together.
• Alert Management Guide — escalation policies, on-call, and alert fatigue.
• Service Level Objective Template — write down your SLOs and error-budget policy.
• Prometheus documentation — PromQL reference, exporters, and federation.
• Grafana documentation — dashboards, alerting, and provisioning.
• Companion resources — scrape config, recording and alert rules, Alertmanager routing, the golden-signals dashboard JSON, and an instrumented Flask example as runnable files.