Capacity Planning — Forecast, Scale

Overview

Capacity planning answers two questions that decide whether your next traffic spike becomes a non-event or an outage: how much load can your infrastructure handle today, and how much will it need to handle next quarter. Get it wrong on the low side and you get saturation, cascading timeouts and angry users; get it wrong on the high side and you pay for idle machines — over-provisioned infrastructure is one of the most common sources of cloud waste.

The discipline itself is simple: measure what you’ve got, forecast what you’ll need, validate under load, and keep buffer for the unexpected. What makes it hard in practice is that demand rarely grows in a straight line — product launches, seasonal patterns and compounding growth all distort the curve. This guide walks through the whole loop, from baseline metrics to a documented capacity plan you can hand to finance. For a ready-to-fill document, see the capacity planning template; for the projection worksheet, see the forecast template.

When to Use

Reach for structured capacity planning when:

• You’re preparing for a product launch, marketing campaign, or seasonal traffic spike
• Your service degrades during recurring peak hours and nobody knows exactly which resource saturates first
• You want to cut cloud costs without gambling on reliability — over-provisioning is as much a capacity problem as under-provisioning
• You need to justify an infrastructure budget with projections instead of gut feeling
• You’re migrating from on-premise to cloud and must right-size instances rather than copying hardware specs 1:1

You can skip the formal process when the system is small enough that a single metrics dashboard answers “are we near the limit” at a glance — a side project on one VPS doesn’t need a capacity plan document. The trigger to formalize is when the answer stops being obvious: several services sharing databases, or a real budget at stake. For ongoing operational practices around this, the SRE practices guide covers error budgets and alerting, and the FinOps guide covers the cost side.

Core Concepts

Concept	Description
Current Capacity	Maximum throughput your system can handle with existing resources
Headroom	Buffer above peak usage to absorb unexpected spikes (typically 20-30%)
Saturation Point	Utilization level where performance starts degrading (usually >70% CPU, >80% memory)
Scaling Lead Time	Time required to provision and deploy additional capacity
Demand Forecast	Projected future load based on historical trends and business events

Two of these deserve more attention than they usually get. Headroom isn’t wasted capacity — it’s your insurance against the spike that arrives before your autoscaler reacts. And scaling lead time is the number most teams forget: if provisioning a new node takes 10 minutes, your capacity plan has to anticipate load 10 minutes into the future, not just the present.

The Math Behind the Thresholds

The 70% CPU rule isn’t arbitrary — it comes from queueing theory. Two small formulas do most of the work in capacity planning:

• Utilization law: U = X * S where U is utilization, X is throughput (requests per second) and S is service time per request. If a request takes 8 ms of CPU and you serve 1,000 rps, you need 1000 * 0.008 = 8 fully busy cores just to keep up — at 100% utilization with zero buffer.
• Little’s Law: N = X * R where N is the number of requests in the system and R is response time. The catch: R isn’t constant — as utilization climbs toward 100%, queue time grows non-linearly. At 50% utilization a queue adds modest delay; at 90% it dominates response time entirely.

That non-linearity is why capacity plans use saturation thresholds instead of raw capacity. Serving 1,000 rps on exactly 8 cores means running at 100% — the first burst queues everything and latency spikes. At a 70% ceiling you provision 8 / 0.7 ≈ 11.4 cores, so twelve. The “wasted” 30% is what keeps p99 flat when the burst arrives.

The same math applies to connection pools, thread pools and queues: any resource used near its limit turns small traffic wobbles into large latency jumps. Model your ceiling from the bottleneck you found in step 2, not from the resource with the most slack.

Step-by-Step Capacity Planning Process

1. Measure Current Baseline

Before planning growth, capture what “normal” looks like. Collect metrics over a representative period — 2-4 weeks minimum, longer if your traffic is seasonal. Averages alone are misleading here: record percentiles and peak values, because capacity decisions are made against peaks, not means.

# Collect metrics over a representative period (2-4 weeks)
# Key metrics: CPU, memory, disk I/O, network, request latency, error rate

# Example: Prometheus query for CPU utilization
avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) * 100

Metrics to track:

• Resource metrics: CPU, memory, disk, network
• Application metrics: requests per second, latency percentiles (p50, p95, p99), error rates
• Business metrics: active users, transactions per minute, data volume growth

Prometheus and managed equivalents (CloudWatch, Azure Monitor, Datadog) all give you this data; what matters is the retention window, not the tool.

2. Identify Bottlenecks

Your effective capacity is set by whichever resource saturates first, not by the resource with the most headroom. Measure headroom per resource and the weakest one is your ceiling:

# Example: Analyze which resource hits limits first
from dataclasses import dataclass

@dataclass
class ResourceLimit:
    name: str
    current_usage: float
    max_capacity: float
    saturation_threshold: float

    def headroom(self) -> float:
        return (self.saturation_threshold - self.current_usage) / self.saturation_threshold * 100

# Evaluate headroom for each resource
resources = [
    ResourceLimit("CPU", 45, 100, 70),
    ResourceLimit("Memory", 60, 100, 80),
    ResourceLimit("Disk IOPS", 75, 100, 85),
    ResourceLimit("Network", 30, 100, 70),
]

bottleneck = min(resources, key=lambda r: r.headroom())
print(f"Bottleneck: {bottleneck.name} with {bottleneck.headroom():.1f}% headroom")

Watch for hidden bottlenecks that don’t show up in OS metrics: database connection pools, thread pools, file descriptors, API rate limits, and downstream services that scale slower than you do. These live in application metrics — a service can idle at 30% CPU while its connection pool sits at 95% and requests queue behind it until timeouts cascade upstream. If your bottleneck analysis says “plenty of headroom” but p99 keeps climbing, the saturating resource is one you aren’t measuring yet.

3. Forecast Demand

Combine historical trends with business context — pure trend extrapolation misses the launch your marketing team scheduled for next month.

Techniques:

• Trend extrapolation: extend historical growth curves
• Seasonal adjustment: account for weekly, monthly, or annual patterns
• Event-driven forecasting: factor in known traffic events (launches, campaigns)
• Business correlation: link capacity to business metrics (new customers, revenue)

# Example: Demand forecast with headroom
peak_qps_current: 5000
weekly_growth_rate: 0.05  # 5% per week
headroom_percent: 0.30    # 30% buffer

# Forecast for 3 months (13 weeks)
peak_qps_forecast: 5000 * (1.05 ** 13) ≈ 9440
required_capacity: 9440 * 1.30 ≈ 12272 QPS

Beware compounding: 5% weekly growth doesn’t sound like much until you realize it doubles demand every ~14 weeks. Always model the optimistic, expected and pessimistic scenarios — the plan should survive being wrong by a factor of two in either direction.

Seasonality deserves its own line in the model. An e-commerce service running a 5,000 rps baseline that sees a 4x multiplier during sale events needs capacity for 20,000 rps plus headroom — around 26,000 rps provisioned at a 30% buffer — not a gently rising trend line. Separate the forecast into “organic growth” and “known events” so reviewers can challenge each part independently.

4. Load Test to Validate

Forecasts are hypotheses; load tests are how you check them. Test at 2-3x your expected peak so you see failure modes before production does.

// capacity-test.js — k6 load test
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '5m', target: 100 },   // Ramp up
    { duration: '10m', target: 100 },  // Steady state
    { duration: '5m', target: 200 },   // Stress test
    { duration: '5m', target: 0 },     // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<200'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const res = http.get('https://api.example.com/health');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 200ms': (r) => r.timings.duration < 200,
  });
  sleep(1);
}

k6, Gatling and JMeter all work for this. The important part is testing the whole stack — a load test that only hits your stateless frontend proves nothing about the database behind it.

Two test shapes matter for different questions: a spike test (sudden 3-5x jump) tells you whether your scaling lead time can absorb real bursts, while a soak test (expected peak sustained for hours) surfaces slow leaks — memory growth, connection pool exhaustion, disk filling — that short tests never reach.

• Companion code: the capacity-planning-guide companion repo packages the k6 script and the bottleneck analyzer as runnable files.

5. Choose Scaling Strategy

Strategy	When to Use	Pros	Cons
Vertical scaling	Predictable, steady growth; database workloads	Simple, no code changes	Hard limit, downtime risk, expensive
Horizontal scaling	Variable, spiky traffic; stateless services	Elastic, fault-tolerant	Added complexity, data consistency
Auto-scaling	Unpredictable or cyclical demand	Cost-efficient, hands-off	Cold start latency, configuration complexity
Reserved capacity	Predictable baseline load	Major cost savings	Less flexible, upfront commitment

In practice most production systems mix strategies: reserved instances or committed-use discounts cover the baseline, auto-scaling absorbs the variable part. On Kubernetes the Horizontal Pod Autoscaler scales pods on metrics; on cloud VMs, AWS Auto Scaling groups do the equivalent for instances. Match the tool to the scaling lead time — auto-scaling that takes 10 minutes to add a node can’t absorb a spike that arrives in 30 seconds, no matter how well it’s configured.

6. Plan for Headroom

Buffer sizing is a risk decision, not a fixed rule:

• Minimum headroom: 20% above expected peak
• Critical services: 30-40% headroom
• Cost-constrained environments: 15% with faster scaling triggers
• Seasonal businesses: plan headroom around known peak seasons

The cheaper your scaling reaction time, the less headroom you need. If you can provision capacity in seconds (serverless, fast autoscaling), 15-20% is defensible; if it takes hours (hardware orders, manual provisioning), you need the upper range plus a procurement calendar.

7. Document and Review

A capacity plan that lives only in someone’s head decays fast. Write down:

• Current baseline metrics and bottlenecks
• Demand forecast with its assumptions (so reviewers can challenge them)
• Scaling strategy and triggers
• Cost projections
• Review schedule — monthly for fast-growing services, quarterly otherwise

The capacity planning template covers this structure; the review meeting is where forecasts get corrected, so keep the previous forecast visible next to actuals — that’s how your estimates get better over time. Bring SRE, the service owner and someone from finance to that review: each sees a different failure mode of the plan.

What Works

• Start with data, not guesses. Collect at least 2 weeks of production metrics before forecasting.
• Test at scale. Load test at 2-3x expected peak to understand failure modes.
• Right-size continuously. Review instance types and reserved capacity quarterly — cloud pricing and your own usage both drift.
• Correlate with business events. Link capacity to product launches, marketing, and seasonality.
• Automate monitoring. Set up alerts when utilization crosses review thresholds (e.g., 60% sustained), not just when it breaks.
• Plan for degradation. Define what sheds load first (feature flags, rate limits, reduced quality) when capacity is exceeded — the graceful degradation pattern covers the mechanics.

Common Mistakes

• Planning for averages instead of peaks. Average load hides burst behavior; capacity decisions are peak decisions.
• Ignoring scaling lead time. If it takes 10 minutes to scale, plan for traffic 10 minutes earlier.
• Over-provisioning “just in case.” Excess capacity is wasted money; use auto-scaling for variable loads and reserved capacity for the baseline.
• Forgetting downstream dependencies. Scaling the frontend without scaling the database just moves the bottleneck — and makes it fail harder under more traffic.
• Not re-testing after changes. Architecture changes invalidate previous capacity assumptions; re-run the load test after meaningful changes.
• Treating the forecast as a promise. It’s an estimate with error bars; review it against actuals on a schedule or it drifts into fiction.

Variants

• Cloud-native capacity planning: use managed auto-scaling, spot instances, and serverless for elastic workloads. Lead times shrink to seconds, so headroom can be tighter — but per-request pricing shifts the cost risk.
• On-premise capacity planning: focus on hardware procurement cycles, virtualization density, and power/cooling constraints. Lead times stretch to months, so forecasts need to reach further and headroom needs to be wider.
• Database capacity planning: monitor query performance, connection limits, storage growth, and replication lag. Databases scale vertically far longer than app servers, so their saturation points deserve their own plan.
• Serverless and managed services: capacity is measured in concurrency limits, provisioned throughput and per-request pricing rather than machines. AWS Lambda concurrency quotas and database IOPS budgets are the equivalent of the saturation thresholds above — plan against them the same way.

Worked Example: Capacity Plan for a Growing SaaS

A realistic scenario shows how the pieces fit together. SaaS product, 10K active users, 15% monthly growth, planning six months ahead:

Metric	Current	Growth	6-month projection
Active users	10K	15%/mo	~23K
Requests/min	50K	15%/mo	~115K
Avg CPU	45%	15%/mo	~85%
Memory	60%	10%/mo	~95%
Storage	500 GB	20%/mo	~1.5 TB
Bandwidth	200 GB/mo	15%/mo	~460 GB/mo

Map projections against provisioned capacity to get an action plan:

Resource	Provisioned	Current usage	Projected	Action
CPU (8 cores)	8 cores	3.6 cores	6.8 cores	Add 4 cores, month 4
Memory (32 GB)	32 GB	19.2 GB	30.4 GB	Add 16 GB, month 3
Storage (1 TB)	1 TB	500 GB	1.5 TB	Add 2 TB, month 2
Bandwidth	1 TB/mo	200 GB	460 GB	OK until month 6
DB connections	200	80	184	Grow pool, month 5

Month	Action	Estimated cost
1-2	Storage +2 TB	$100/mo
3	Memory +16 GB	$80/mo
4	CPU +4 cores	$150/mo
5	DB pool +100 connections	$0 (config)
6	Evaluate cluster upgrade	$500/mo

The lessons this scenario illustrates, and that real plans keep re-teaching:

• Plan with data, not assumptions.
• Compounding growth is deceptive — 15% monthly is 2.3x in six months.
• Storage usually grows faster than CPU and memory.
• Auto-scaling absorbs spikes, not sustained growth; sustained growth needs provisioning.
• Review the plan monthly, not semi-annually.

Monitoring thresholds that would trigger action in this scenario:

Signal	Threshold	Action
CPU utilization	> 70% for 10 min	Scale out / upgrade
Memory utilization	> 80%	Scale out / upgrade
Disk usage	> 75%	Expand volume
DB connection pool	> 80% used	Increase pool or add replicas
Response time p99	> 500 ms	Investigate bottleneck

The compound growth formula used above: F = P * (1 + r)^n where P is the current value, r the periodic growth rate, and n the number of periods. For 10K users at 15% monthly over 6 months: 10000 * 1.15^6 ≈ 23,133. Round up and add your headroom on top.

Capacity Planning Checklist

• 2-4 weeks of baseline metrics captured (peaks and percentiles, not averages)
• Bottleneck identified per resource, including pools and downstream dependencies
• Forecast covers optimistic, expected and pessimistic scenarios
• Load test run at 2-3x expected peak against the full stack
• Scaling strategy matched to scaling lead time
• Headroom sized to risk and reaction speed (15-40%)
• Plan documented with assumptions and review cadence
• Alert thresholds set below saturation point, with owners