Dashboard Design Template

Overview

A dashboard is a visual interface that answers questions about system health. Good dashboards cut time-to-diagnosis during incidents; bad ones add noise and confusion. This template defines the structure for service dashboards that on-call engineers can actually rely on at 3 AM.

It pairs well with the Alert Runbook Template — the dashboard tells you something’s wrong, the runbook tells you what to do about it.

When to Use

• Creating a new service dashboard
• Redesigning an existing dashboard that nobody uses
• Standardizing dashboard layout across teams
• Setting up SLO and error budget tracking
• Preparing dashboards for on-call handoff

Solution

Copy the whole block below and fill in the placeholders. The layout encodes the scan order: status first, then service metrics, then infrastructure, then business, then context.

# Dashboard Design: `<Service Name>`

## Dashboard Metadata

| Field | Value |
|-------|-------|
| Dashboard Title | Payment Service Health |
| Dashboard URL | https://grafana.example.com/d/payment-service |
| Owner | Payment Team |
| Last Reviewed | 2026-07-05 |
| Audience | On-call engineers, developers, SRE |
| Refresh Interval | 30 seconds |
| Time Range Default | Last 1 hour |

## 1. Dashboard Layout

```
┌─────────────────────────────────────────────────────────────┐
│  ROW 1: Service Status Banner                                │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │ Health      │  │ SLO Status  │  │ Error Budget│         │
│  │ OK/WARN/ERR │  │ 99.9%       │  │ 72% remain  │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
├─────────────────────────────────────────────────────────────┤
│  ROW 2: RED Metrics (Rate, Errors, Duration)                │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │ Request Rate│  │ Error Rate  │  │ p95 Latency │         │
│  │ (req/s)     │  │ (% of total)│  │ (ms)        │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
├─────────────────────────────────────────────────────────────┤
│  ROW 3: Traffic and Status Codes                             │
│  ┌─────────────────────────┐  ┌─────────────────────────┐  │
│  │ Requests by endpoint    │  │ Status code distribution│  │
│  │ (stacked area)          │  │ (pie chart)             │  │
│  └─────────────────────────┘  └─────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│  ROW 4: Infrastructure Health                                │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │ CPU Usage   │  │ Memory      │  │ DB Conns    │         │
│  │ (%)         │  │ (MB)        │  │ (active/idle)│        │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
├─────────────────────────────────────────────────────────────┤
│  ROW 5: Business Metrics                                     │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │ Orders/min  │  │ Revenue/min │  │ Success Rate│         │
│  │             │  │ ($)         │  │ (%)         │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
├─────────────────────────────────────────────────────────────┤
│  ROW 6: Context and Links                                    │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ Runbook | Logs | Traces | Alerts | Deploy Info     │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
```

## 2. Panel Specifications

### Row 1: Service Status Banner

| Panel | Type | Query | Thresholds | Purpose |
|-------|------|-------|------------|---------|
| Health | Stat | `up{service="payment"}` | OK: 1, WARN: 0.5, ERR: 0 | Quick health check |
| SLO Status | Stat | `payment_slo_availability_ratio` | OK: > 0.999, WARN: > 0.99, ERR: < 0.99 | SLO compliance at a glance |
| Error Budget | Gauge | `payment_error_budget_remaining_pct` | OK: > 30%, WARN: > 10%, ERR: < 10% | How much error budget is left |

### Row 2: RED Metrics

| Panel | Type | Query | Thresholds | Purpose |
|-------|------|-------|------------|---------|
| Request Rate | Time series | `rate(http_requests_total{service="payment"}[5m])` | — | Traffic volume |
| Error Rate | Time series | `rate(http_requests_total{service="payment",status=~"5.."}[5m]) / rate(http_requests_total{service="payment"}[5m])` | WARN: > 1%, ERR: > 5% | Error percentage |
| p95 Latency | Time series | `histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service="payment"}[5m]))` | WARN: > 500ms, ERR: > 1s | Response time |

### Row 3: Traffic and Status Codes

| Panel | Type | Query | Purpose |
|-------|------|-------|---------|
| Requests by endpoint | Stacked area | `sum(rate(http_requests_total{service="payment"}[5m])) by (endpoint)` | See traffic distribution |
| Status code distribution | Pie chart | `sum(rate(http_requests_total{service="payment"}[5m])) by (status)` | Spot error patterns |

### Row 4: Infrastructure Health

| Panel | Type | Query | Thresholds | Purpose |
|-------|------|-------|------------|---------|
| CPU Usage | Time series | `rate(container_cpu_usage_seconds_total{pod=~"payment.*"}[5m]) * 100` | WARN: > 70%, ERR: > 90% | Resource saturation |
| Memory | Time series | `container_memory_working_set_bytes{pod=~"payment.*"}` | WARN: > 80% limit, ERR: > 95% | Memory pressure |
| DB Connections | Time series | `payment_db_connections_active` / `payment_db_connections_max` | WARN: > 70%, ERR: > 90% | Pool exhaustion |

### Row 5: Business Metrics

| Panel | Type | Query | Thresholds | Purpose |
|-------|------|-------|------------|---------|
| Orders/min | Time series | `rate(payment_orders_total[5m]) * 60` | — | Business throughput |
| Revenue/min | Time series | `rate(payment_revenue_total[5m]) * 60` | — | Revenue tracking |
| Success Rate | Stat | `rate(payment_orders_total{status="success"}[5m]) / rate(payment_orders_total[5m])` | WARN: < 99%, ERR: < 95% | Business health |

### Row 6: Context and Links

| Link | URL | Purpose |
|------|-----|---------|
| Runbook | https://runbooks.example.com/payment-service | Incident response |
| Logs | https://kibana.example.com/app/discover#/?_a=(query:service:payment) | Log search |
| Traces | https://jaeger.example.com/search?service=payment | Distributed traces |
| Alerts | https://alertmanager.example.com/#/alerts?receiver=payment | Active alerts |
| Deploy Info | https://grafana.example.com/d/deployments?service=payment | Recent deployments |

## 3. SLO Configuration

### SLO Definition

| SLO | Target | Window | Error Budget | Measurement |
|-----|--------|--------|-------------|-------------|
| Availability | 99.9% | 30 days | 0.1% = 43.2 min | `1 - (failed_requests / total_requests)` |
| Latency p95 | < 500ms | 30 days | 0.1% = 43.2 min | `histogram_quantile(0.95, ...)` |
| Latency p99 | < 2s | 30 days | 0.1% = 43.2 min | `histogram_quantile(0.99, ...)` |

### Error Budget Tracking

| Metric | Query | Purpose |
|--------|-------|---------|
| Budget remaining | `1 - (rate(errors[30d]) / 0.001)` | How much budget is left |
| Burn rate (1h) | `rate(errors[1h]) / 0.001` | Fast burn detection |
| Burn rate (6h) | `rate(errors[6h]) / 0.001` | Sustained burn detection |
| Budget reset date | `30d - elapsed_in_window` | When budget resets |

### Alerting Rules

```yaml
- alert: PaymentSLOFastBurn
  expr: |
    (
      sum(rate(http_requests_total{service="payment",status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{service="payment"}[1h]))
    ) > 0.02
  for: 5m
  labels:
    severity: critical
    service: payment
  annotations:
    summary: "Payment SLO fast burn — 2% budget consumed in 1h"
    runbook: "https://runbooks.example.com/payment-slo-burn"

# Slow burn: 5% of budget in 6 hours
- alert: PaymentSLOSlowBurn
  expr: |
    (
      sum(rate(http_requests_total{service="payment",status=~"5.."}[6h]))
      /
      sum(rate(http_requests_total{service="payment"}[6h]))
    ) > 0.005
  for: 30m
  labels:
    severity: warning
    service: payment
  annotations:
    summary: "Payment SLO slow burn — 5% budget consumed in 6h"
    runbook: "https://runbooks.example.com/payment-slo-burn"
```

## 4. Dashboard Variables

| Variable | Type | Query | Default | Purpose |
|----------|------|-------|---------|---------|
| $datasource | Data source | — | Prometheus | Switch between environments |
| $environment | Query | `label_values(up, environment)` | production | Filter by environment |
| $instance | Query | `label_values(up{service="payment"}, instance)` | All | Filter by instance |
| $endpoint | Query | `label_values(http_requests_total{service="payment"}, endpoint)` | All | Filter by endpoint |
| $timeframe | Interval | — | 1h | Quick time range switch |

## 5. Annotation Layers

| Annotation | Query | Purpose |
|------------|-------|---------|
| Deployments | `deployments{service="payment"}` | Correlate changes with metric shifts |
| Incidents | `incidents{service="payment"}` | See incident impact on metrics |
| Maintenance | `maintenance{service="payment"}` | Expected dips during maintenance |
| Alerts | `alerts{service="payment"}` | When alerts fired relative to metrics |

Explanation

The layout follows a strict top-down scan order. During an incident, an engineer reads the dashboard the same way every time:

The status banner (Row 1) answers “is everything OK?” in one glance. RED metrics (Row 2) — Rate, Errors, Duration, from the Google SRE book’s monitoring chapter — catch most service issues. Traffic breakdown (Row 3) shows which endpoints hurt. Infrastructure health (Row 4) checks whether the service has the resources to operate. Business metrics (Row 5) translate technical health into business impact. Context links (Row 6) get the engineer to logs, traces, and runbooks without hunting.

SLOs and error budgets are the quantitative backbone — the SRE workbook’s alerting chapter is the reference for the burn-rate math used above. The SLO defines “healthy” objectively; the error budget tracks how much room the team has before breaching it. Burn rate alerts catch both fast-burning incidents (2% of budget in 1 hour) and slow-burning trends (5% in 6 hours).

Dashboard variables let engineers filter without writing queries. $endpoint is the most useful one: when an alert names a specific endpoint, filtering the whole dashboard to it is one click.

Annotation layers correlate deployments and incidents with metric changes. When latency spikes and a deployment annotation sits at the same timestamp, you already know where to look.

Dashboard Review Checklist

=== Quarterly Dashboard Review ===

[ ] All panels still have valid data sources (no "No data" panels)
[ ] Thresholds reflect current SLOs and performance baselines
[ ] Variables return correct values (environments, instances, endpoints)
[ ] Annotation layers are still receiving events (deploys, incidents)
[ ] Links to runbooks, logs, and traces are not broken
[ ] No panel has more than 10 series (cardinality check)
[ ] Color coding is consistent (green/yellow/red across all panels)
[ ] Dashboard loads in under 3 seconds
[ ] Mobile view is readable (on-call engineers check from phones)
[ ] At least one other engineer can interpret the dashboard without explanation
[ ] SLO panels match current SLO definitions
[ ] Error budget panel is accurate and not showing stale data
[ ] Business metrics panels reflect current KPIs
[ ] Unused panels removed (check Grafana panel views metric)
[ ] Dashboard is tagged with service name and team owner

Variants

Context	Approach	Notes
Database dashboard	Replace RED with USE (Utilization, Saturation, Errors)	Focus on resource metrics
Queue consumer	Add lag, throughput, and processing time	Consumer-specific metrics
Multi-service overview	Aggregate RED metrics across services	Use service map for dependencies
Business dashboard	Focus on business KPIs, not infrastructure	Revenue, conversion, churn
On-call handoff	Add shift notes, open incidents, action items	Context for incoming on-call

What Works

1. Put the most important information at the top — health status and SLO should be visible without scrolling.
2. Use consistent color coding — green = OK, yellow = warning, red = error across all panels.
3. Link to runbooks, logs, and traces — cut navigation time during incidents.
4. Use variables for filtering — let engineers drill down without writing queries.
5. Add deployment annotations — correlate changes with metric shifts.
6. Set realistic thresholds — base them on historical data, not guesses.
7. Review dashboards quarterly — remove panels nobody looks at, add missing metrics.
8. Keep dashboards readable on a phone — on-call engineers check from mobile.

Companion resources: the dashboard-design-template companion repo has the template as a standalone file, an importable Grafana dashboard.json implementing this layout, and the slo-alerts.yml Prometheus rule group.

Common Mistakes

1. Too many panels — dashboards with 50 panels are walls of noise. Keep under 15.
2. No status summary — forcing engineers to interpret 10 charts to know if things are OK.
3. No thresholds — lines without thresholds require interpretation. Set visual thresholds.
4. No links to context — dashboards without links to logs, traces, and runbooks are dead ends.
5. Stale dashboards — metrics change, services evolve. Review quarterly.
6. Mixing audiences — a dashboard for executives and a dashboard for on-call engineers are different things.
7. No variables — forcing engineers to edit queries to filter by endpoint or instance.
8. Too high cardinality — panels with 100 series are unreadable. Aggregate appropriately.

Troubleshooting

• Panels show “No data”. Check the data source first, then the time range, then the query. A renamed metric or a dropped label is the usual culprit — compare the query against label_values() in the data source explorer.
• Dashboard loads slowly. Almost always cardinality. Aggregate with sum by (...) instead of by (pod), drop regex matchers that fan out to hundreds of series, and raise the scrape interval on heavy panels. Grafana’s docs on panel performance have more.
• Template variables come back empty. The label_values() query depends on the selected data source and time range — a variable scoped to up returns nothing if the job name changed. Test each variable query alone.
• Thresholds never trigger. Threshold values are compared in the panel’s unit — a 500ms threshold on a panel displaying seconds silently never fires. Match the threshold unit to the display unit.
• Annotations stopped appearing. Annotation queries have their own data source and time range. If deploys stopped showing, the annotation metric may have moved or been renamed.