Prometheus and Grafana: Metrics, Dashboards, Alerting

Introduction

Something is paging you at 3 AM and your best debugging tool is kubectl logs | grep error. That’s the moment a metrics stack pays for itself: Prometheus tells you the error rate doubled at 02:47 and which endpoint caused it, before you even open a log file.

Prometheus is a time-series database that pulls metrics from your services over HTTP, stores them locally, and evaluates alerting rules against them. Grafana is the visualization layer: it queries Prometheus with PromQL and renders dashboards. Between them sits Alertmanager, which deduplicates, groups, and routes fired alerts to Slack, PagerDuty, or email. The stack is the de facto standard for cloud-native metrics — Prometheus graduated from the CNCF in 2018 — and the rest of this guide builds it end to end: metric types, instrumentation in Python, Node.js, and Java, PromQL, alerting rules, dashboard provisioning, and a full Docker Compose deployment.

This guide is for backend and infrastructure engineers who already run HTTP services and want metrics plus alerting in front of them. You need Docker and one service exposing an HTTP endpoint. If you want the wider picture first — logs and traces alongside metrics — the Grafana stack observability guide covers all three signals; this one stays deep on the metrics half. For alert strategy and SLOs rather than the stack itself, see the monitoring and alerting guide.

How the pieces fit

The pull model is the core idea to internalize: your services expose a /metrics endpoint in a plain-text exposition format, and Prometheus scrapes it on a schedule. Nothing in your app knows Prometheus exists — it just answers HTTP. That inversion is what makes the model simple to operate: the monitoring system decides the scrape rate, detects dead targets, and never depends on your app knowing where to send data.

Two components deserve one line each because the rest of the guide uses them freely. An exporter is a process that translates a third-party system’s state into Prometheus format — node-exporter exposes host CPU, memory, and disk; there are exporters for PostgreSQL, Redis, NGINX, and most infrastructure you’d deploy. Alertmanager is the companion binary that receives alerts fired by Prometheus, deduplicates them, groups related ones, and routes them to receivers such as Slack or PagerDuty. Prometheus decides that something is wrong; Alertmanager decides who hears about it.

Prometheus metric types

Prometheus defines four metric types. Choosing the right one matters because it determines which PromQL functions you can apply — rate() needs a counter, and quantiles across instances need a histogram, not a summary.

Type	Behavior	Good for	Example query
Counter	Only goes up (resets on restart)	Totals: requests, errors, orders	rate(orders_total[5m]) → orders/sec
Gauge	Goes up and down	Current state: queue depth, memory, connections	active_users → current value
Histogram	Samples into configurable buckets	Latency, response sizes, anything with percentiles	histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
Summary	Client-side quantiles, not aggregable	Rarely — prefer histograms	histogram_quantile doesn’t apply

The histogram-versus-summary distinction bites in production. A Summary computes quantiles inside your app, so a P99 from instance A can’t be combined with a P99 from instance B — averaging them gives a wrong answer. A Histogram exports cumulative bucket counters (_bucket{le="0.5"}), and histogram_quantile computes the real P99 across all instances server-side. The cost is that you choose bucket boundaries upfront; the defaults or the (0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0) set used below cover most HTTP services.

Naming follows a convention worth adopting from day one: base name plus unit suffix — _seconds, _bytes, _total — and labels for dimensions, never embedded in the name.

Instrumenting your services

Instrumentation means three things: define metrics once at startup, update them inside your business logic, and expose them on /metrics. The official client libraries — prometheus_client for Python, prom-client for Node.js, Micrometer’s Prometheus registry for Java — handle the exposition format for you. The examples below instrument the same OrderService in each language so you can compare shapes. For a focused walkthrough, the Python exporter recipe and the Micrometer recipe go deeper per language.

The pattern to copy is where counters live: increment inside the outcome branches (success, payment failure, unexpected error), not at the top of the method. Label with what happened, never with who it happened to — status and payment_method have bounded values; user_id doesn’t, and that single label would create a time series per user.

Python: prometheus_client

# metrics.py — Prometheus instrumentation for Python
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
import time

# Define metrics
ORDERS_TOTAL = Counter(
    "orders_total",
    "Total number of orders created",
    ["status", "payment_method"],
)

ORDER_DURATION = Histogram(
    "order_creation_duration_seconds",
    "Time spent creating orders",
    ["endpoint"],
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)

ACTIVE_USERS = Gauge(
    "active_users",
    "Number of currently active users",
)

QUEUE_DEPTH = Gauge(
    "message_queue_depth",
    "Number of messages in the queue",
    ["queue_name"],
)

# Usage in service code
class OrderService:
    def create_order(self, user_id: int, items: list[dict]) -> dict:
        start = time.time()
        try:
            order = self._process_order(user_id, items)
            ORDERS_TOTAL.labels(status="success", payment_method=order["payment_method"]).inc()
            return order
        except PaymentError:
            ORDERS_TOTAL.labels(status="payment_failed", payment_method="unknown").inc()
            raise
        except Exception:
            ORDERS_TOTAL.labels(status="error", payment_method="unknown").inc()
            raise
        finally:
            ORDER_DURATION.labels(endpoint="/api/orders").observe(time.time() - start)

# Metrics endpoint
from flask import Flask, Response

app = Flask(__name__)

@app.route("/metrics")
def metrics():
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)

Node.js: prom-client

// metrics.ts — Prometheus instrumentation for Node.js
import { Counter, Histogram, Gauge, register } from "prom-client";

// Define metrics
const ordersTotal = new Counter({
  name: "orders_total",
  help: "Total number of orders created",
  labelNames: ["status", "paymentMethod"] as const,
});

const orderDuration = new Histogram({
  name: "order_creation_duration_seconds",
  help: "Time spent creating orders",
  labelNames: ["endpoint"] as const,
  buckets: [0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
});

const activeUsers = new Gauge({
  name: "active_users",
  help: "Number of currently active users",
});

const queueDepth = new Gauge({
  name: "message_queue_depth",
  help: "Number of messages in the queue",
  labelNames: ["queueName"] as const,
});

// Usage in service code
class OrderService {
  async createOrder(userId: string, items: OrderItem[]): Promise<Order> {
    const start = Date.now();
    try {
      const order = await this.processOrder(userId, items);
      ordersTotal.inc({ status: "success", paymentMethod: order.paymentMethod });
      return order;
    } catch (error) {
      ordersTotal.inc({ status: "error", paymentMethod: "unknown" });
      throw error;
    } finally {
      orderDuration.observe({ endpoint: "/api/orders" }, (Date.now() - start) / 1000);
    }
  }
}

// Metrics endpoint (Express)
import express from "express";
const app = express();

app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});

Java: Micrometer

// MetricsConfig.java — Micrometer with Prometheus
import io.micrometer.core.instrument.*;
import io.micrometer.prometheus.PrometheusMeterRegistry;
import io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics;
import io.micrometer.core.instrument.binder.system.ProcessorMetrics;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class MetricsConfig {

    @Bean
    public PrometheusMeterRegistry prometheusRegistry() {
        PrometheusMeterRegistry registry = new PrometheusMeterRegistry(key -> null);

        // Bind JVM and system metrics
        new JvmMemoryMetrics().bindTo(registry);
        new ProcessorMetrics().bindTo(registry);

        return registry;
    }
}

// Service code using Micrometer
@Service
public class OrderService {

    private final Counter ordersCounter;
    private final Timer orderDurationTimer;
    private final Gauge activeUsersGauge;

    public OrderService(MeterRegistry registry) {
        this.ordersCounter = Counter.builder("orders_total")
            .description("Total number of orders created")
            .tags("status", "success")
            .register(registry);

        this.orderDurationTimer = Timer.builder("order_creation_duration")
            .description("Time spent creating orders")
            .publishPercentileHistogram()  // exposes _bucket series, aggregable in PromQL
            .register(registry);

        this.activeUsersGauge = Gauge.builder("active_users", this, OrderService::getActiveUserCount)
            .description("Number of currently active users")
            .register(registry);
    }

    public Order createOrder(Long userId, List<OrderItem> items) {
        return orderDurationTimer.record(() -> {
            Order order = processOrder(userId, items);
            ordersCounter.increment();
            return order;
        });
    }

    private int getActiveUserCount() {
        return userSessionManager.getActiveCount();
    }
}

// Metrics endpoint
@RestController
public class MetricsController {

    private final PrometheusMeterRegistry registry;

    public MetricsController(PrometheusMeterRegistry registry) {
        this.registry = registry;
    }

    @GetMapping("/metrics")
    public String metrics() {
        return registry.scrape();
    }
}

Note the .publishPercentileHistogram() call in the Java example: it publishes bucket series instead of client-side percentiles, which is the Micrometer equivalent of choosing a histogram over a summary.

PromQL essentials

PromQL — the Prometheus Query Language — is a functional language for selecting, aggregating, and computing over time series. Every Grafana panel and every alerting rule runs a PromQL expression, so the handful of patterns below covers most of what you will write day to day.

Basic queries

# Current value of a gauge
active_users

# Rate of a counter over 5 minutes
rate(orders_total[5m])

# Total increase over 1 hour
increase(orders_total[1h])

# Average over 5 minutes
avg_over_time(memory_usage_bytes[5m])

# Max over 10 minutes
max_over_time(queue_depth[10m])

Two rules of thumb. Counters are meaningless raw — always wrap them in rate() or increase(), which also correct for counter resets when a pod restarts. And for alerting prefer rate(): irate() uses only the last two samples and produces jumpy graphs, while rate() averages over the window.

Label filtering

# Filter by status label
orders_total{status="error"}

# Filter by multiple labels
http_requests_total{method="POST", status=~"5.."}

# Regex matching (5xx errors)
http_requests_total{status=~"5.."}

# Negative regex (exclude health checks)
http_requests_total{path!="/health"}

Aggregations

# Sum by service
sum by (service) (rate(http_requests_total[5m]))

# Average latency by endpoint
avg by (endpoint) (rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]))

# Top 5 endpoints by error rate
topk(5, sum by (endpoint) (rate(http_requests_total{status=~"5.."}[5m])))

# 99th percentile latency across all instances
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Arithmetic and vector matching

# Error rate as percentage
100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Memory usage as percentage of limit, matched on the instance label
100 * (memory_usage_bytes / on(instance) memory_limit_bytes)

# CPU usage above 80%
(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80

When you divide two vector expressions, Prometheus needs to know how to pair the samples: on(instance) tells it to match series that share the same instance label. Leaving on() empty — or omitting it when labels don’t align — silently returns no data or wrong pairings.

Alerting rules

An alerting rule is a PromQL expression evaluated on every evaluation_interval. When it stays true for the duration in for, Prometheus fires the alert and hands it to Alertmanager. Keep rules in a dedicated YAML file loaded via rule_files.

# alerting_rules.yml — Prometheus alerting rules
groups:
  - name: service-alerts
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          100 * sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service) > 5
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "{{ $labels.service }} has {{ $value }}% error rate for the last 5 minutes."

      # High latency (P99 > 2s)
      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 2
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "High P99 latency on {{ $labels.service }}"
          description: "P99 latency is {{ $value }}s for the last 10 minutes."

      # Service down
      - alert: ServiceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
          team: oncall
        annotations:
          summary: "Service {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been down for more than 2 minutes."

      # High memory usage
      - alert: HighMemoryUsage
        expr: |
          100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 85
        for: 10m
        labels:
          severity: warning
          team: infra
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value }}% for the last 10 minutes."

      # Queue depth growing
      - alert: QueueDepthGrowing
        expr: |
          avg_over_time(message_queue_depth[10m]) > avg_over_time(message_queue_depth[1h]) * 2
        for: 5m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "Queue {{ $labels.queue_name }} depth is growing"
          description: "Queue depth is 2x higher than the 1h average."

The alert anatomy that matters: expr is the condition, for is the hysteresis that prevents flapping on momentary spikes, labels carry routing metadata (severity, team) that Alertmanager matches on, and annotations hold the human-readable text templated with {{ $labels.x }} and {{ $value }}.

Expensive expressions that many dashboards or alerts share belong in recording rules: a pre-computed expression stored as a new time series in the same rules file.

  - name: recording-rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: service:http_errors:ratio5m
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
          / sum by (service) (rate(http_requests_total[5m]))

Dashboards then query service:http_errors:ratio5m directly instead of recomputing a five-term expression on every refresh. Validate rules before deploying with promtool check rules alerting_rules.yml, and unit-test them against fixed input with promtool test rules — it catches off-by-one thresholds before they page someone.

Grafana dashboards

Grafana reads dashboards and data sources from YAML files dropped under /etc/grafana/provisioning, which is what makes the whole stack reproducible: dashboards live in Git, not in someone’s browser session.

Two files are required, and the first one is the one most examples forget: the data source. Without it, provisioned dashboards load but show no data.

# grafana/provisioning/datasources/datasource.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

# grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: 'default'
    orgId: 1
    folder: 'Services'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

Dashboard JSON (key panels)

{
  "dashboard": {
    "title": "Order Service Overview",
    "panels": [
      {
        "title": "Request Rate",
        "type": "timeseries",
        "datasource": { "type": "prometheus", "uid": "prometheus" },
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[5m])) by (endpoint)",
            "legendFormat": "{{endpoint}}"
          }
        ]
      },
      {
        "title": "Error Rate %",
        "type": "stat",
        "datasource": { "type": "prometheus", "uid": "prometheus" },
        "targets": [
          {
            "expr": "100 * sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))"
          }
        ],
        "thresholds": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 1 },
          { "color": "red", "value": 5 }
        ]
      },
      {
        "title": "P99 Latency",
        "type": "timeseries",
        "datasource": { "type": "prometheus", "uid": "prometheus" },
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))",
            "legendFormat": "P99"
          },
          {
            "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))",
            "legendFormat": "P95"
          }
        ]
      }
    ]
  }
}

Panels use timeseries, which replaced the deprecated graph type in Grafana 8 and later. Referencing the data source by uid keeps the JSON portable across environments. For building the panels themselves — variables, repeating rows, alert annotations — the Grafana dashboards recipe walks through it.

Deployment with Docker Compose

This compose file runs the complete stack locally — Prometheus, Grafana, Alertmanager, and node-exporter — wired to the configuration files above. For production the same topology applies, but secrets come from your secret manager and in Kubernetes you’d typically deploy kube-prometheus-stack instead.

# docker-compose.yml — Full Prometheus + Grafana stack
services:
  prometheus:
    image: prom/prometheus:v3.13.4
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerting_rules.yml:/etc/prometheus/alerting_rules.yml
      - prometheus-data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"

  grafana:
    image: grafana/grafana:13.2.4
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:?set it in .env or your secret store}
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
    depends_on:
      - prometheus

  alertmanager:
    image: prom/alertmanager:v0.34.1
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

  node-exporter:
    image: prom/node-exporter:v1.12.1
    ports:
      - "9100:9100"
    pid: host

volumes:
  prometheus-data:
  grafana-data:

# prometheus.yml — Prometheus configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - alerting_rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "order-service"
    static_configs:
      - targets: ["order-service:8080"]
    metrics_path: /metrics
    scrape_interval: 10s

  - job_name: "api-gateway"
    static_configs:
      - targets: ["api-gateway:8080"]
    metrics_path: /metrics

Each job_name becomes the job label on every scraped series, and static_configs is the simplest discovery mechanism — one entry per target. In container platforms you swap it for service discovery: kubernetes_sd_configs has Prometheus ask the Kubernetes API for pods and services and scrape them as they appear — typically by annotating pods with prometheus.io/scrape: "true" — and consul_sd_configs does the same against a Consul catalog. In Kubernetes specifically, most teams skip hand-rolled config entirely and deploy kube-prometheus-stack, a Helm chart that ships Prometheus, Alertmanager, Grafana, node-exporter, and opinionated dashboards and rules as one unit.

# alertmanager.yml — Alert routing
route:
  receiver: "default"
  group_by: ["alertname", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - 'severity="critical"'
      receiver: "oncall"

receivers:
  - name: "default"
    slack_configs:
      - api_url: "https://hooks.slack.com/services/..."
        channel: "#alerts"
        send_resolved: true
        title: '{{ .CommonLabels.alertname }}'
        text: '{{ .CommonAnnotations.summary }}'

  - name: "oncall"
    pagerduty_configs:
      - service_key: "your-pagerduty-key"

The nested routes sends severity="critical" alerts to PagerDuty while everything else lands in Slack — this is the routing that the severity and team labels on your rules exist for. group_wait batches the first alerts of a group so a deploy that trips five alerts produces one notification, not five.

Verify the stack

docker compose up -d, then check in order: http://localhost:9090/targets should list every job as UP — a target stuck in DOWN usually means a wrong metrics_path or the app isn’t exposing /metrics yet. Query up in http://localhost:9090/graph to confirm. Then open http://localhost:3000 — the provisioned Prometheus data source and any dashboards under grafana/dashboards should already be there. Alertmanager sits at http://localhost:9093 showing routed alerts.

When Prometheus is not the right tool

Pull monitoring covers long-running services well, but three situations call for something else. Short-lived batch jobs finish before the next scrape — use the Pushgateway so the job pushes its final metrics before exiting, and treat it as a cache for job metrics, not a general push replacement. Long-term storage and HA exceed what a single Prometheus TSDB does well — remote-write to Thanos, Cortex, or Mimir for retention past a month and cross-replica deduplication. You don’t want to operate a TSDB at all — hosted options (Grafana Cloud, managed Prometheus from AWS/GCP) trade self-hosting control for not running the storage layer.

Best Practices

• Use Histograms over Summaries: histograms aggregate server-side across instances; summary quantiles don’t.
• Follow the unit_suffix naming convention (_seconds, _bytes, _total) — it makes metrics self-documenting in Grafana autocomplete.
• Keep label cardinality bounded: status, endpoint, method are safe; user_id, session_id, request_id create a series per user and can take a Prometheus instance down.
• Size retention deliberately — a 15s scrape interval retained 30 days costs roughly 1GB per 100k active series.
• Alert on symptoms (error rate > 5%), not causes (CPU > 80%) — cause-based alerts fire while users see nothing wrong, and miss the failures that matter.
• Every alert needs a for clause. Without it, a two-second spike pages you and auto-resolves before you open your laptop.
• Scrape at 10-15s for services. Finer intervals multiply storage cost for resolution you will rarely look at.
• Pre-compute expensive queries as recording rules instead of paying for them on every dashboard refresh.
• Monitor the monitor: Prometheus exposes its own metrics (prometheus_tsdb_*, prometheus_rule_evaluation_failures_total) — alert on scrape failures and storage pressure too.

Common Mistakes

• High-cardinality labels: one user_id label turns 5 metrics into millions of series. Cardinality problems are the single most common reason Prometheus installs die. Keep identity in logs and traces.
• Using Summaries for percentiles: they can’t be aggregated across instances. Use Histograms with chosen buckets.
• No for clause: alerts fire on momentary spikes and immediately resolve. Add for: 5m minimum for rate-based alerts.
• Scraping too frequently: a 1s interval creates 10x the series of a 10s interval with almost no benefit. Use 10-15s.
• No datasource provisioning: dashboards load but render empty because no data source was provisioned. Ship datasource.yml alongside dashboards.yml.
• Dead receivers: declaring an oncall receiver with no route that references it means critical alerts fall back to the default. Every receiver needs a matching route.

See Also

• Complete Guide to Observability with the Grafana Stack — metrics plus Loki logs and Tempo traces on the same Grafana.
• Monitoring and alerting strategy guide — SLOs, alert fatigue, and on-call design above the tooling layer.
• Distributed tracing guide — when a metric tells you something is slow, traces tell you where.

Further Reading

• Prometheus metric types and PromQL basics — official docs.
• Prometheus alerting rules and Alertmanager configuration.
• Grafana provisioning — data sources, dashboards, and alert rules as code.
• Micrometer Prometheus registry for the Java path.
• Thanos and Mimir for long-term storage and HA.