Performance Regression Template

Overview

Performance regressions are invisible until they are painful. A 20% latency increase after a release goes unnoticed for weeks, then suddenly your biggest customer churns because their API integration timed out. Benchmarking is not enough—you need a structured way to compare before/after metrics, identify root causes, and decide whether to rollback or fix forward. This template creates a repeatable process for catching, analyzing, and resolving performance regressions.

When to Use

• For alternatives, see Weekly Ops Review Template.

Use this resource when:

• Your release pipeline lacks automated performance gates
• A recent deployment degraded response times or increased resource consumption
• You are establishing performance budgets and need a way to enforce them

Solution

# Performance Regression Report: `<Release / Feature>`

## 1. Regression Summary

| Field | Value |
|-------|-------|
| Release | `version / commit` |
| Date Detected | `YYYY-MM-DD` |
| Detected By | `CI benchmark / APM alert / Customer report` |
| Severity | `Critical (> 50%) / High (> 20%) / Medium (> 10%) / Low (< 10%)` |
| Status | `Investigating / Root cause identified / Fix deployed / Resolved` |
| Owner | `@name` |

## 2. Benchmark Comparison

### 2.1. Latency

| Metric | Baseline | Current | Delta | Threshold | Breached? |
|--------|----------|---------|-------|-----------|-----------|
| P50 | `X ms` | `Y ms` | `+Z%` | `+10%` | Yes / No |
| P95 | `X ms` | `Y ms` | `+Z%` | `+15%` | Yes / No |
| P99 | `X ms` | `Y ms` | `+Z%` | `+20%` | Yes / No |

### 2.2. Throughput

| Metric | Baseline | Current | Delta | Threshold | Breached? |
|--------|----------|---------|-------|-----------|-----------|
| RPS | `X` | `Y` | `+/- Z%` | `-10%` | Yes / No |

### 2.3. Resource Utilization

| Metric | Baseline | Current | Delta | Threshold | Breached? |
|--------|----------|---------|-------|-----------|-----------|
| CPU | `X%` | `Y%` | `+Z%` | `+20%` | Yes / No |
| Memory | `X%` | `Y%` | `+Z%` | `+20%` | Yes / No |
| Disk I/O | `X MB/s` | `Y MB/s` | `+Z%` | `+30%` | Yes / No |

## 3. Root Cause Analysis

| Hypothesis | Evidence | Validated? | Owner |
|------------|----------|------------|-------|
| | | | |

### Diagnostic Steps Taken

1. Correlate regression with deployment time
2. Review code changes in release diff
3. Check database query plans for new queries or schema changes
4. Profile CPU and memory (flame graphs, heap dumps)
5. Check downstream service latency
6. Verify infrastructure changes (instance type, scaling events)
7. Review cache hit rates and eviction patterns
8. Check for background jobs or batch processes coinciding with peak traffic

## 4. Action Plan

| Action | Owner | Deadline | Status |
|--------|-------|----------|--------|
| | | | |

### Decision: Rollback or Fix Forward?

| Criterion | Rollback | Fix Forward |
|-----------|----------|-------------|
| Severity | Critical or High | Medium or Low |
| Time to Fix | > 4 hours estimated | < 2 hours estimated |
| Risk of Rollback | Low (rollback tested) | High (irreversible migration) |
| Customer Impact | > 5% of users affected | < 5% of users affected |
| **Decision** | [ ] | [ ] |

## 5. Verification After Fix

- [ ] Benchmark re-run shows metrics within 5% of baseline
- [ ] APM dashboards show stable trend for 24 hours
- [ ] No new alerts triggered post-deployment
- [ ] Customer-facing synthetic tests pass
- [ ] Resource utilization returned to baseline
- [ ] Post-fix review documented in incident tracker

Explanation

The template forces a quantified decision rather than gut feeling. Many teams either panic-rollback every regression or never rollback and let performance decay. The comparison tables make the regression visible with numbers, and the rollback/fix-forward decision matrix removes ambiguity. The diagnostic steps are ordered by frequency: most regressions are caused by a bad query, a missing cache, or a downstream slowdown—not exotic infrastructure issues.

CI Performance Gate Configuration

# GitHub Actions performance gate
name: Performance Regression Check
on:
  pull_request:
    branches: [main]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup environment
        run: |
          npm ci
          docker compose up -d test-db
      - name: Run benchmark suite
        run: npm run benchmark -- --output=results.json
      - name: Compare with baseline
        run: |
          node scripts/compare-baseline.js \
            --current results.json \
            --baseline baselines/main.json \
            --threshold 5 \
            --fail-on-regression
      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: results.json
      - name: Comment on PR
        if: always()
        run: node scripts/pr-comment.js --results results.json

Benchmark Comparison Script

#!/usr/bin/env python3
"""Compare current benchmark results against baseline."""
import json
import sys
import argparse

def compare(current_path, baseline_path, threshold_pct):
    with open(current_path) as f:
        current = json.load(f)
    with open(baseline_path) as f:
        baseline = json.load(f)

    regressions = []
    improvements = []

    for metric in baseline:
        name = metric['name']
        base_value = metric['value']
        curr = next((m for m in current if m['name'] == name), None)
        if not curr:
            continue
        change_pct = ((curr['value'] - base_value) / base_value) * 100
        if change_pct > threshold_pct:
            regressions.append({
                'name': name,
                'baseline': base_value,
                'current': curr['value'],
                'change': f'+{change_pct:.1f}%'
            })
        elif change_pct < -threshold_pct:
            improvements.append({
                'name': name,
                'baseline': base_value,
                'current': curr['value'],
                'change': f'{change_pct:.1f}%'
            })

    if regressions:
        print('REGRESSIONS DETECTED:')
        for r in regressions:
            print(f"  {r['name']}: {r['baseline']} -> {r['current']} ({r['change']})")
        sys.exit(1)
    else:
        print('No regressions detected.')
        if improvements:
            print('Improvements:')
            for i in improvements:
                print(f"  {i['name']}: {i['baseline']} -> {i['current']} ({i['change']})")

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--current', required=True)
    parser.add_argument('--baseline', required=True)
    parser.add_argument('--threshold', type=float, default=5.0)
    args = parser.parse_args()
    compare(args.current, args.baseline, args.threshold)

Performance Regression Triage Decision Tree

=== Performance Regression Triage ===

1. IDENTIFY SCOPE (5 min)
   - Which endpoint/service regressed?
   - P50, P95, or P99 affected?
   - When did it start? (correlate with deploys)

2. CHECK DEPLOY HISTORY (10 min)
   - Was there a deploy in the last 24h?
   - Did the regression start at deploy time?
   - YES -> Rollback or fix-forward. Go to step 4.
   - NO -> Continue to step 3.

3. CHECK INFRASTRUCTURE (15 min)
   - Database: slow queries? replication lag? connection pool?
   - Cache: hit rate dropped? evictions? redis memory?
   - Network: increased latency between services? DNS issues?
   - Disk: IOPS saturated? disk full?

4. DIAGNOSE ROOT CAUSE (30 min)
   - Code: N+1 queries, missing index, unbounded loop
   - Config: changed pool size, timeout, cache TTL
   - Data: table grew beyond index capacity, hot partition
   - Dependency: downstream service slower, API rate limited

5. DECIDE: ROLLBACK OR FIX-FORWARD (5 min)
   - Rollback if: regression > 20%, user-facing, fix unknown
   - Fix-forward if: regression < 10%, fix is ready, low-risk
   - Feature flag if: regression tied to specific feature

6. VERIFY FIX (ongoing)
   - Re-run benchmarks
   - Monitor APM for 24h
   - Confirm metrics return to baseline

Variants

Context	Key Metrics	Special Consideration
Web / API	P50, P95, P99 latency; RPS; error rate	Focus on user-facing percentiles
Mobile backend	API latency, payload size, battery impact	Consider data transfer costs
Database	Query latency, connection pool usage, replication lag	Query plan changes after migrations
Batch / ETL	Job duration, throughput, resource cost per record	Cost per job run, not just speed
Microservices	Inter-service latency, circuit breaker trips, retry storms	Network overhead dominates
Frontend	Time to Interactive, Largest Contentful Paint, bundle size	Lighthouse scores + RUM data

What Works

1. Run benchmarks in CI for every release; block deploys on regression > threshold
2. Establish baseline metrics from a stable period, not an arbitrary target
3. Profile in production, not just locally; local benchmarks often mislead
4. Correlate regressions with deployment time, not just “sometime last week”
5. Document the fix in the runbook; the same regression pattern often repeats

Common Mistakes

1. Ignoring P99 latency and only watching averages; averages hide tail latency
2. Benchmarking in isolation without concurrent load; race conditions appear only under traffic
3. Blaming infrastructure before checking code changes; most regressions are code, not hardware
4. Not establishing thresholds in advance; ad-hoc thresholds lead to inconsistent decisions
5. Failing to verify after a fix; the first fix often only partially addresses the issue

Troubleshooting

• Pipeline fails silently: enable verbose logging and store pipeline artifacts between stages so you can inspect the exact state that failed.
• Container crashes on startup: check that environment variables, secrets, and config files are mounted correctly. Read the first 50 lines of logs before scaling replicas.
• Deployment rolls back repeatedly: verify health checks, resource limits, and startup probes. A failing readiness probe is a common cause of rolling restarts.
• Slow CI builds: cache dependencies and docker layers. Split large test suites into parallel jobs to reduce wall-clock time.
• Drift between environments: use infrastructure-as-code and immutable artifacts. Compare deployed versions with the declared source of truth before debugging behavior differences.

FAQ

How do I establish performance baselines?

Collect metrics during a known-stable period (e.g., the last 2 weeks with no incidents or major releases). Use the 95th percentile of daily peaks as your baseline, not the average. Update baselines quarterly as traffic patterns change. Store baseline data in your APM tool or a dedicated performance database so it is queryable during regressions.

Should performance regression block a release?

Yes, for Critical and High regressions. For Medium, use a warning gate that requires engineering sign-off. For Low, log and schedule a fix in the next sprint. The threshold should be defined before the release, not debated during the incident. If your CI benchmark is flaky, fix the benchmark before using it as a gate.

What if the regression only affects a small subset of users?

A subset regression can still be severe if it affects high-value customers or a critical feature. Document the affected segment in the report. If it is a niche use case, you may fix forward. If it is a high-value segment, consider a targeted rollback or feature flag disable. Never ignore a regression just because it is “only 1% of users” without understanding who that 1% is.

How do we set performance regression thresholds?

Set thresholds based on user impact, not arbitrary percentages. A 5% regression on a 50ms endpoint is negligible (2.5ms). A 5% regression on a 5s endpoint is noticeable (250ms). Use tiered thresholds: 10% for P50, 15% for P95, 20% for P99. Block releases only on P95 and P99 regressions. Log P50 regressions for tracking. Adjust thresholds per service based on business criticality.

What tools should we use for performance benchmarking?

For API benchmarks: k6, Locust, or Artillery. For database benchmarks: pgbench (PostgreSQL), sysbench (MySQL). For frontend: Lighthouse CI, WebPageTest. For load testing: k6 with distributed execution. Integrate benchmarks into CI using GitHub Actions or Jenkins. Store results in a time-series database (InfluxDB, Prometheus) for trend analysis. Use APM tools (Datadog, New Relic, Dynatrace) for production performance monitoring.

How do we handle flaky benchmarks?

Flaky benchmarks erode trust in the performance gate. Fix them by: running benchmarks in isolated environments (dedicated runners, no shared resources), warming up before measurement, running multiple iterations and taking the median, excluding outliers, and using a cooldown between test groups. If a benchmark is inherently flaky, increase the threshold or mark it as informational rather than blocking. Track flakiness rate and fix benchmarks with > 5% flakiness.

What is the difference between load testing and performance benchmarking?

Load testing measures system behavior under expected and peak load (e.g., can we handle 10,000 RPS?). Performance benchmarking measures specific metrics against a baseline (e.g., did this change make the API 10% slower?). Load testing is about capacity; benchmarking is about regression. Run load tests quarterly or before major launches. Run benchmarks in every CI pipeline. Both are needed but serve different purposes.

How do we measure frontend performance regressions?

Track Core Web Vitals: LCP (Largest Contentful Paint), INP (Interaction to Next Paint), CLS (Cumulative Layout Shift). Use Lighthouse CI in GitHub Actions to block regressions on these metrics. Collect Real User Monitoring (RUM) data from production using the web-vitals library. Set thresholds: LCP < 2.5s, INP < 200ms, CLS < 0.1. Track bundle size as a leading indicator — a 50KB bundle increase will eventually cause LCP regression. Use source maps to attribute bundle size changes to specific commits.