API Performance Budget Template

Overview

Every endpoint burns resources — CPU, memory, DB connections, bandwidth. Skip budgets and teams keep adding capabilities until the system folds under load. This template treats performance budgets as a contract between what product wants and what infra can deliver — trade-offs get settled on paper, not in an outage. Related resources: Design a Scalable API Gateway for Microservices, Build Resilient Systems with the Circuit Breaker Pattern, and Dependency Injection.

When to Use

Use this resource when:

• Designing a new API or adding a new endpoint to an existing one
• Planning capacity for a marketing campaign or seasonal traffic spike
• Investigating why an API is slow under load
• Negotiating what “fast enough” means between product and infra — the GraphQL vs REST decision is a common trigger for revisiting budgets

Skip it when: your API has a single internal consumer with no SLO — start with a latency dashboard instead and come back when there is a contract to honor. For frontend budgets (LCP, bundle size, web vitals), use the performance budget template instead.

Solution

# API Performance Budget: `<API Name>`

## 1. Endpoint Budgets

### 1.1. Latency Budget (p95)

| Endpoint | Target | Current | Buffer | Notes |
|----------|--------|---------|--------|-------|
| `GET /health` | < 10ms | 8ms | 2ms | Must not depend on DB |
| `GET /users/{id}` | < 100ms | 85ms | 15ms | Single DB query allowed |
| `POST /orders` | < 250ms | 210ms | 40ms | Multi-table transaction |
| `GET /reports/aggregate` | < 2s | 1.8s | 200ms | Read replica allowed |

### 1.2. Throughput Budget

| Endpoint | RPS Target | Max Burst | Connection Pool | Notes |
|----------|------------|-----------|-----------------|-------|
| `GET /products` | 5,000 | 10,000 | 50 | Cached in Redis |
| `POST /checkout` | 500 | 1,000 | 20 | DB write, idempotent |

### 1.3. Payload Budget

| Direction | Max Size | Typical | Compression |
|-----------|----------|---------|-------------|
| Request body | 1 MB | 10 KB | Gzip |
| Response body | 5 MB | 50 KB | Gzip + Brotli |
| Header total | 16 KB | 1 KB | None |

### 1.4. Dependency Budget

| Dependency | Max Latency Contribution | Max Concurrent Calls | Fallback |
|------------|--------------------------|----------------------|----------|
| PostgreSQL | 40ms | 20 | Read replica |
| Redis | 5ms | 100 | Skip cache (slower) |
| Payment Gateway | 200ms | 10 | Queue for async |
| Search Service | 100ms | 50 | Basic LIKE query |

## 2. Resource Budgets

| Resource | Baseline | Headroom | Max | Alert At |
|----------|----------|----------|-----|----------|
| CPU / core | 40% | 30% | 70% | 60% |
| Memory / pod | 512 MB | 256 MB | 1 GB | 800 MB |
| DB connections | 50 | 30 | 80 | 70 |
| Network egress | 100 Mbps | 50 Mbps | 200 Mbps | 150 Mbps |

## 3. Budget Enforcement

- [ ] Latency assertions in CI (fail build if p95 exceeds budget)
- [ ] Load test gates in deployment pipeline
- [ ] Payload size validation at API gateway
- [ ] Circuit breaker on dependency calls exceeding max latency
- [ ] Autoscaling triggered at 60% CPU, not 80%

## 4. Review Cadence

| Trigger | Action | Owner |
|---------|--------|-------|
| Weekly | Review dashboards for drift above budget | SRE |
| Monthly | Adjust budgets based on traffic growth | Platform |
| Quarterly | Re-negotiate budgets with product team | Engineering Lead |
| Incident | Post-incident budget audit | Incident Commander |

Explanation

A latency budget is a chain: frontend (100ms) + network (50ms) + API (200ms) + database (40ms) = 390ms total. Blow the allocation on any one link and the whole user experience sags. The buffer column exists because real-world performance is noisy. Dependency budgets keep one slow downstream service from eating the whole API budget. Throughput budgets define connection pools and cache sizes. Resource budgets prevent one heavy endpoint from starving others on the same host.

Latency Budget Chain Example

A “Place Order” click fans out across several layers. Each layer has its own budget:

User clicks button (0ms)
  └─ Browser rendering (50ms)
       └─ CDN edge (20ms)
            └─ API gateway (10ms)
                 └─ Order service (80ms)
                      ├─ PostgreSQL (30ms)
                      ├─ Payment gateway (150ms)
                      └─ Inventory service (40ms)
                 └─ Response serialization (5ms)
            └─ CDN edge return (20ms)
       └─ Browser render (50ms)
Total: ~455ms (within 500ms budget)

If the payment gateway takes 300ms instead of 150ms, the total jumps to 605ms and the budget is blown. The dependency budget for payment gateway (200ms max) would flag this as a violation before the endpoint budget does.

CI Enforcement with k6 Load Tests

Use k6 to enforce latency budgets in your CI pipeline:

import http from "k6/http";
import { check } from "k6";

export const options = {
  stages: [
    { duration: "30s", target: 50 },
    { duration: "1m", target: 100 },
    { duration: "30s", target: 0 },
  ],
  // Thresholds gate the build: tagged metrics enforce per-endpoint budgets
  thresholds: {
    'http_req_duration{endpoint:health}': ["p(95)<10"],
    'http_req_duration{endpoint:user}': ["p(95)<100"],
    'http_req_duration{endpoint:order}': ["p(95)<250"],
  },
};

export default function () {
  const healthRes = http.get(`${__ENV.BASE_URL}/health`, {
    tags: { endpoint: "health" },
  });
  check(healthRes, {
    "health p95 < 10ms": (r) => r.timings.duration < 10,
  });

  const userRes = http.get(`${__ENV.BASE_URL}/users/123`, {
    tags: { endpoint: "user" },
  });
  check(userRes, {
    "user lookup p95 < 100ms": (r) => r.timings.duration < 100,
  });

  const orderRes = http.post(
    `${__ENV.BASE_URL}/orders`,
    JSON.stringify({ productId: 1, quantity: 2 }),
    { headers: { "Content-Type": "application/json" }, tags: { endpoint: "order" } }
  );
  check(orderRes, {
    "order creation p95 < 250ms": (r) => r.timings.duration < 250,
  });
}

Run in CI with budget enforcement:

k6 run --env BASE_URL=https://staging.example.com load-test.js
# Exit code 1 if any threshold is violated, failing the CI build

Monitoring Budget Compliance with Prometheus

Track budget violations in production with PromQL:

# Percentage of requests within budget (per endpoint)
sum(rate(http_request_duration_seconds_bucket{le="0.1", endpoint="GET /users"}[5m]))
/
sum(rate(http_request_duration_seconds_count{endpoint="GET /users"}[5m]))
* 100

# Alert when more than 1% of requests exceed budget
(
  sum(rate(http_request_duration_seconds_count{endpoint="GET /users"}[5m]))
  -
  sum(rate(http_request_duration_seconds_bucket{le="0.1", endpoint="GET /users"}[5m]))
)
/
sum(rate(http_request_duration_seconds_count{endpoint="GET /users"}[5m]))
> 0.01

Budget Tracking Dashboard

Create a Grafana dashboard panel for each endpoint showing:

1. Current p95 latency as a single stat, colored green/yellow/red based on budget
2. Latency trend over the last 7 days with a horizontal line at the budget threshold
3. Budget violation rate as a percentage of requests exceeding the target
4. Throughput vs budget showing current RPS against the RPS target
5. Dependency latency breakdown showing each upstream service contribution to total latency

Dashboard JSON for a latency budget panel:

{
  "panels": [
    {
      "title": "GET /users - p95 Latency vs Budget",
      "type": "stat",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{endpoint=\"GET /users\"}[5m])) * 1000",
          "legendFormat": "p95 latency (ms)"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              { "value": null, "color": "green" },
              { "value": 80, "color": "yellow" },
              { "value": 100, "color": "red" }
            ]
          },
          "unit": "ms"
        }
      }
    }
  ]
}

The green/yellow/red thresholds map directly to the budget: green is within 80% of budget, yellow is 80-100%, and red is over budget.

Deriving Your First Budget Numbers

Do not invent targets from thin air — measure first. Pull 30 days of production metrics and take the p95 per endpoint, then set the budget at current p95 plus 20-30% headroom. If you lack production data, start from similar endpoints in the same service or from the dependency budget: the slowest dependency plus gateway and serialization overhead sets your floor. Three mistakes to avoid at this stage: measuring in short windows (5-minute p95 swings wildly), averaging percentiles across endpoints with different cost profiles, and measuring warm-cache performance when most production traffic is cold. Cold starts deserve their own line item in serverless setups — a Lambda init of 800ms will eat most of a 1s budget by itself.

Rolling Out Budgets When You Are Already Over

Most APIs are introduced to budgets after they are already violating them. Do not set aspirational numbers nobody can meet — the budget becomes decoration. Instead, ratchet down: set the budget at current p95, log violations without failing the build for two weeks, then tighten by 10-15% per sprint while the team fixes the top offenders. When an endpoint permanently exceeds budget, escalate explicitly: either the product accepts a relaxed SLO for that endpoint, or engineering gets a work item to fix it. A budget everyone ignores is worse than no budget — it trains the team to ignore every metric.

Measurement Pitfalls

• Averaging percentiles: p95 of hourly p95s is not p95 — aggregate raw latency, then compute the percentile.
• Short windows: a 5-minute p95 during low traffic hides real tail latency; use 1h+ windows for budgets.
• Missing dependency attribution: an endpoint over budget is often a downstream service over its budget — check dependency latency first.
• Health-check contamination: exclude /health and synthetic traffic from percentiles or your numbers look better than reality.
• Single-region measurement: latency budgets measured from the same region as the server miss the WAN cost real users pay.

Governance: Owners, Exceptions, and Burn Rate

A budget without an owner decays. Assign one per layer: endpoint budgets belong to the service team, dependency budgets to the platform team, and resource budgets to SRE. The owner reviews drift weekly and is the first call when a gate fails.

Exceptions need an expiry date. When a team asks for a relaxed budget — “checkout can’t meet 250ms this quarter” — approve it with a date and a remediation ticket attached. An exception without expiry silently becomes the new budget. Track open exceptions in the same document so reviewers can count how much of the API is running outside its contract.

Track burn rate, not just snapshots. A service at 96% of budget for three consecutive weeks is already violating it — the number just hasn’t crossed the line yet. Burn rate (percentage of budget consumed over time) is what turns budgets from a post-mortem artifact into an early warning. Plot it per endpoint: flat lines are healthy, rising lines get prioritized.

Adapting the Template

For a small internal API, cut the template to latency and throughput — resource budgets and regional rows add ceremony without payoff. For a public API or a revenue-critical path, extend it: add per-region budgets, a cold-start line, and a column for who signed off on each number. The template is a negotiation tool, so whatever makes the product-versus-infra conversation explicit belongs in it; everything else is weight.

A worked example helps first-time teams. Suppose /checkout has a 500ms budget: gateway 30ms + auth 40ms + order service 80ms + payment gateway 150ms + inventory 60ms + serialization 20ms = 380ms, leaving 120ms of buffer. When payment degrades to 220ms, the total hits 450ms — still inside the endpoint budget, but the payment dependency budget (200ms) fires first. That ordering is the whole point: dependency budgets catch the failing link before the endpoint budget turns red and users notice.

In organizations with multiple teams sharing a platform, treat the template as versioned contract language. One team’s endpoint budget is another team’s dependency budget — the payment service’s 200ms ceiling is simultaneously the order service’s constraint and the payment team’s promise. Keep the document in version control, reference it in the API design review checklist, and require a signed change (even a one-line PR) whenever a number moves. Unsigned drift is how “the budget says 250ms” quietly becomes “the budget says whatever production currently does.” For external-facing APIs, publish the endpoint-level numbers in developer docs and keep the internal decomposition private — consumers see the commitment, engineers see the math. Finally, retire rows that stop mattering: a dependency you removed six months ago should not still have a row confusing the next reader. The template earns trust when it describes reality, not when it records history.

Budgets also change how teams talk during incidents. “The API feels slow” is an argument; “checkout is at 118% of its latency budget for 40 minutes” is a fact. During an incident review, the budget document answers three questions fast: was the violation inside our own allocation or did a dependency blow its share, how long did the breach last, and did the ratchet plan from the last exception actually execute? If the same endpoint shows up in three consecutive reviews, the honest options are to fix it, raise its budget with sign-off, or admit the service needs a redesign — any of the three is better than a fourth round of “we’ll look into it.” Keep a short history of budget changes at the bottom of the document. Dates, the number before and after, and who approved it — three columns that turn the template from a snapshot into an audit trail.

One last adaptation point: budgets live downstream of architecture choices, so revisit them whenever the stack changes — a migration to a new runtime, a cache layer added between the API and the database, a move to a different region. Each of those shifts every number in the table, and a budget that reflects last year’s architecture stops being a contract and becomes folklore. Schedule a quick pass at every architecture review: five minutes confirming each row still maps to a real component keeps the whole document honest — stale rows are how good budgets die quietly over time.

Variants

Context	Approach	Notes
GraphQL	Query complexity budget	Limit total field cost per query, not just depth
Streaming API	Throughput per shard	Budget per partition, not global
Mobile SDK	Bundle size + call frequency	Budget includes payload size and number of parallel requests
gRPC	Per-method deadline propagation	Use context deadlines to enforce budgets across hops

What Works

1. Set budgets during design, not after launch
2. Include budgets in API design review checklists
3. Fail CI builds when latency tests exceed budget by any margin
4. Reserve 20-30% buffer in every budget for traffic spikes
5. Document who can approve budget exceptions (usually engineering lead)
6. Track budget burn rate, not just current values
7. Separate read and write budgets since they have different cost profiles

Common Mistakes

1. Basing budgets on best-case lab conditions instead of production percentiles
2. Ignoring dependency latency when calculating endpoint budgets
3. Setting one global budget for all endpoints regardless of criticality
4. Treating budgets as one-time decisions instead of living documents
5. Failing to enforce budgets in CI or deployment gates
6. Not accounting for cold starts in serverless architectures
7. Setting budgets without input from the product team, leading to unrealistic targets

Troubleshooting

• High latency between services: trace the request path. Look for synchronous chains, missing caching, and oversized payloads that cross network boundaries.
• Single point of failure: identify components without redundancy. Add replicas, failover, or circuit breakers before scaling traffic.
• Unexpected coupling between services: look at shared databases, libraries, and schemas. Bound contexts should own their data and expose stable interfaces.
• Cost spikes after scaling: right-size instances and use autoscaling with limits. Reserved capacity or spot instances can reduce steady-state spend.
• Difficult to reason about the system: maintain architecture decision records and service dependency maps. Validate the diagrams with observability data.

Common Production Pitfalls

• Writing “fast” or “low latency” in the Notes column instead of numbers — a budget without a number is a wish.
• Setting the budget once and never revisiting it when traffic doubles.
• Keeping the budget in a wiki nobody opens during an incident review.
• Budgets with no named owner — when p95 drifts, nobody is accountable for the ratchet.
• Copying the example table without deleting rows that don’t apply (not every API has a search service).

Further Reading

• Google SRE — Error Budgets — the budget philosophy
• k6 Documentation — the load test tool used for enforcement