Auto-Scaling Policy Template: HPA-Ready

Overview

Auto-scaling is the bridge between cost efficiency and availability. Scale too late and your service crashes under load; scale too early and you burn money on idle capacity. This template documents the exact rules, thresholds, and procedures your infrastructure team uses to scale workloads up and down automatically. Pair it with a capacity planning template for sizing and a cost allocation template so the spend controls in section 5 have somewhere to live.

When to Use

Use this resource when:

• Defining scaling rules for a new service deployed to the cloud
• Auditing why an auto-scaling event caused an outage or excessive cost
• Migrating from static instance sizes to live scaling
• Onboarding a workload to Kubernetes HPA or a cloud autoscaler for the first time
• Reviewing a scaling policy after a production incident or a cost spike

Don’t use it for pure serverless workloads (concurrency is provider-managed — document concurrency limits instead) or for stateful services that can’t scale horizontally.

For alternatives, see Cloud Cost Allocation Template.

Downloadable assets for this template (Terraform ASG config, HPA manifest, policy document, event-log format): companion resources.

Solution

# Auto-Scaling Policy: `<Service Name>`

## 1. Service Metadata

| Field | Value |
|-------|-------|
| Service | `name` |
| Platform | `AWS / GCP / Azure / Kubernetes` |
| Owner Team | `@team-name` |
| Last Reviewed | `YYYY-MM-DD` |

## 2. Scale-Up Policy

### 2.1. Triggers

| Metric | Threshold | Duration | Scale Action | Cooldown |
|--------|-----------|----------|--------------|----------|
| CPU utilization | > 60% | 2 minutes | Add 1 instance | 3 minutes |
| Memory utilization | > 70% | 2 minutes | Add 1 instance | 3 minutes |
| Request count | > 5,000 RPS | 1 minute | Add 2 instances | 5 minutes |
| Queue depth | > 100 messages | 3 minutes | Add 1 instance | 3 minutes |
| Latency p95 | > 500ms | 2 minutes | Add 2 instances | 5 minutes |

### 2.2. Limits

| Limit | Value | Rationale |
|-------|-------|-----------|
| Max instances | 20 | Cost ceiling, database connection limit |
| Max scale-up per event | 50% of current | Prevent thundering herd on cold start |
| Scale-up cooldown | 3 minutes | Allow metric stabilization |

## 3. Scale-Down Policy

### 3.1. Triggers

| Metric | Threshold | Duration | Scale Action | Cooldown |
|--------|-----------|----------|--------------|----------|
| CPU utilization | < 30% | 10 minutes | Remove 1 instance | 5 minutes |
| Memory utilization | < 30% | 10 minutes | Remove 1 instance | 5 minutes |
| Request count | < 1,000 RPS | 10 minutes | Remove 1 instance | 5 minutes |

### 3.2. Limits

| Limit | Value | Rationale |
|-------|-------|-----------|
| Min instances | 3 | Redundancy, rolling deployment buffer |
| Max scale-down per event | 25% of current | Avoid over-correction |
| Scale-down cooldown | 5 minutes | Allow metric stabilization |

## 4. Instance Requirements

### 4.1. Health Checks

- [ ] Load balancer health check passes before instance receives traffic
- [ ] Instance must serve traffic for minimum 5 minutes before scale-down eligibility
- [ ] Connection draining allows in-flight requests to complete (30 seconds)

### 4.2. Warm-Up

- [ ] New instances complete initialization (app start, cache warm-up) before joining pool
- [ ] Warm-up time documented: `60 seconds`
- [ ] Startup probe / readiness probe configured in orchestrator

## 5. Cost Controls

| Control | Value | Notes |
|---------|-------|-------|
| Max hourly spend | $500 | Alert if exceeded |
| Instance type | `c5.large` | CPU-optimized for API workload |
| Spot / Preemptible | 50% of instances | Use for non-critical batch processing only |
| Reserved capacity | Baseline of 3 instances | Commitment discount for minimum |

## 6. Incident Response

| Scenario | Action | Owner |
|----------|--------|-------|
| Scale-up fails (quota exceeded) | Page on-call, escalate to cloud admin | SRE |
| Scale-down causes errors | Pause auto-scaling, investigate | Platform |
| Costs spike > 2x baseline | Review policy, check for runaway jobs | Finance + SRE |
| Latency rises despite scaling | Alert: likely database bottleneck, not compute | DBA + App Team |

Explanation

The template separates scale-up (fast, aggressive) from scale-down (slow, conservative). Scale-up triggers use shorter durations because you need capacity before failures happen. Scale-down uses longer durations to avoid thrashing instances in and out during normal traffic jitter. The cooldown prevents the autoscaler from reacting to metric noise caused by the scaling event itself. Min instances cover redundancy — even at zero traffic you need enough instances to survive a rolling deployment without downtime.

AWS Auto Scaling Terraform Configuration

resource "aws_autoscaling_group" "app" {
  name                = "app-asg"
  vpc_zone_identifier = data.aws_subnets.private.ids
  min_size            = 3
  max_size            = 20
  desired_capacity    = 5

  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }

  tag {
    key                 = "Team"
    value               = "platform"
    propagate_at_launch = true
  }
}

resource "aws_autoscaling_policy" "scale_up" {
  name                   = "scale-up"
  autoscaling_group_name = aws_autoscaling_group.app.name
  adjustment_type        = "ChangeInCapacity"
  scaling_adjustment     = 2
  cooldown               = 300
}

resource "aws_autoscaling_policy" "scale_down" {
  name                   = "scale-down"
  autoscaling_group_name = aws_autoscaling_group.app.name
  adjustment_type        = "ChangeInCapacity"
  scaling_adjustment     = -1
  cooldown               = 600
}

resource "aws_cloudwatch_metric_alarm" "high_cpu" {
  alarm_name          = "high-cpu"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 60
  statistic           = "Average"
  threshold           = 70
  alarm_actions       = [aws_autoscaling_policy.scale_up.arn]
}

resource "aws_cloudwatch_metric_alarm" "low_cpu" {
  alarm_name          = "low-cpu"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 10
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 60
  statistic           = "Average"
  threshold           = 30
  alarm_actions       = [aws_autoscaling_policy.scale_down.arn]
}

Kubernetes HPA with Custom Metrics

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "1000"
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 4
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 120
      selectPolicy: Min

Scaling Event Dashboard Query

=== Scaling Event Log (Last 24h) ===

Time         Direction   Trigger              From -> To   Duration
02:15 UTC    Scale UP    CPU > 70% (2 min)    5 -> 7       45s warm-up
02:45 UTC    Scale UP    CPU > 70% (2 min)    7 -> 9       42s warm-up
04:30 UTC    Scale DOWN  CPU < 30% (10 min)   9 -> 8       15s drain
08:00 UTC    Scale UP    RPS > 1000 (1 min)   8 -> 12      38s warm-up
08:30 UTC    Scale UP    RPS > 1000 (1 min)   12 -> 16     41s warm-up
10:00 UTC    Scale DOWN  CPU < 30% (10 min)   16 -> 14     12s drain
14:00 UTC    Scale DOWN  CPU < 30% (10 min)   14 -> 10     18s drain
18:00 UTC    Scale DOWN  CPU < 30% (10 min)   10 -> 8      15s drain
22:00 UTC    Scale DOWN  CPU < 30% (10 min)   8 -> 5       20s drain

Total scaling events: 9
Scale-up events: 4 (avg warm-up: 41s)
Scale-down events: 5 (avg drain: 16s)
Thrashing detected: No (min 2h between opposing events)

Choosing Thresholds and Cooldowns

The template’s numbers are starting points, not values to copy blindly. Three relationships drive the choice:

Threshold vs. warm-up. The scale-up threshold must fire early enough that new instances finish warming up before the existing pool saturates. If boot takes 60 s and traffic grows 20% per minute, a 60% CPU trigger leaves roughly two minutes of headroom. The Terraform example above uses 70 instead of the template’s 60 — deliberate for a service with fast warm-up. For slow-boot services (JVM, model loading, cache prefill), lower the threshold or speed up warm-up first.

Duration as a noise filter. A 2-minute evaluation window ignores single-spike noise but adds latency to the response. Shorter windows react faster and thrash more: 1–3 min for user-facing scale-up, 5 min for batch consumers.

Cooldown >= warm-up + settle time. The cooldown stops the autoscaler from reacting to metric noise caused by its own action — a fresh instance reports high CPU while warming and would trigger another scale-up. Set scale-up cooldown to at least warm-up plus one metric period, and scale-down cooldown longer still, because removing capacity is riskier than adding it.

Decision	Guideline
CPU threshold	Leave headroom for one full scale-up event: warm-up x growth rate
Scale-up duration	1–3 min user-facing; 5 min batch
Scale-down duration	10 min minimum; longer for JVM-heavy services
Scale-up cooldown	>= warm-up + 1 metric period
Scale-down cooldown	>= 2x scale-up cooldown
Max scale-up per event	<= 50% of current capacity

Metrics That Matter for Auto-Scaling

Not every metric belongs in a scaling trigger. The useful ones share three traits: they saturate before users notice, they’re cheap to collect, and they don’t lie during partial failures.

• CPU utilization is the default for a reason — it correlates with capacity on compute-bound services. It fails on I/O-bound services that sit idle waiting on downstream calls while requests pile up.
• Memory catches leaks and growing working sets, but it rarely drops fast enough to drive scale-down; use it for scale-up and out-of-memory prevention only.
• Request rate (RPS) tracks user demand directly for stateless services. Pair it with per-instance concurrency so the trigger fires on real saturation rather than raw volume.
• Queue depth is the right trigger for consumers — the number of pending messages is the backlog. Scale on messages-per-worker, not on CPU.
• Latency p95 measures the outcome users feel, but it’s a lagging signal: by the time p95 breaches, some requests already failed. Use it as an alarm, not as the primary trigger.

Metrics to avoid as triggers: averages over long windows (they smooth away spikes), business KPIs like signups-per-minute (they correlate with nothing the autoscaler can fix), and any metric a partial outage can distort — a node reporting zero RPS because it’s down looks like “low demand” to a naive policy.

In practice, strong production policies almost never rely on a single metric alone. They combine two triggers with OR logic: a leading signal (queue depth, RPS) that fires early, and a confirming signal (CPU or latency) that catches what the first one misses. The template’s five-metric table models exactly that — fill it with your two best signals and delete the rows you don’t need rather than wiring all five.

Variants

Context	Approach	Notes
Kubernetes HPA	Metrics via custom metrics API	Scale on custom metrics (queue length, request latency)
AWS EC2 Auto Scaling	CloudWatch alarms	Use predictive scaling for known patterns
Serverless (Lambda)	Concurrency limits	No traditional scaling; manage max concurrency and reserved concurrency
GPU workloads	Scale on GPU utilization	Longer warm-up, higher cost; avoid spot instances

What works

1. Always set max instance limits to prevent runaway scaling from infinite loops or DDoS
2. Use predictive scaling for predictable traffic patterns (nightly batch, business hours)
3. Test scale-up and scale-down events in staging before production
4. Monitor scaling event frequency; frequent events indicate threshold misconfiguration
5. Document why each threshold was chosen so future teams can tune intelligently

Common Mistakes

1. Setting CPU threshold at 80% or higher, leaving no headroom for spikes
2. Using the same policy for all services regardless of their workload patterns
3. Forgetting connection draining, causing dropped requests during scale-down
4. Scaling only on CPU and ignoring memory, network, or custom metrics
5. Allowing scale-down to zero for stateful services that need persistent connections

Adopting This Policy

Treat the template as a document you validate, not a form you fill in:

1. Baseline first. Pull four weeks of CPU, memory, RPS and p95 data. Thresholds chosen without a baseline are guesses — set them just above the normal p99 of each metric, not at round numbers.
2. Deploy in observe-only mode. Both HPA (scaleTargetRef pointing to a shadow deployment) and cloud autoscalers support dry-run configurations; collect what the policy would have done for a week.
3. Load-test the policy. Drive traffic with k6 or Locust through the real thresholds and measure scale-up latency end to end: metric collection, evaluation window, instance launch, warm-up, health check. That total — not the threshold — is your real responsiveness.
4. Wire incident-response rows to real owners before production. A policy that pages “SRE” without a runbook entry is a monitoring gap.
5. Enable in production with conservative limits (lower max, longer cooldowns), then tighten once you’ve got real scaling-event data.

Reviewing the Policy

Scaling policies rot silently: traffic patterns shift, instance types change, and warm-up grows as the app accumulates dependencies. Review on a fixed cadence — quarterly for steady services, monthly during growth phases, and after any incident where scaling played a role. In each review check: scaling-event frequency (the thrashing signal), warm-up drift, whether the max-instance cap still matches the database connection limit, and whether the cost ceiling still reflects this quarter’s budget. Update Last Reviewed in the template and record why each threshold changed — the rationale column exists so the next reviewer doesn’t have to guess.

Troubleshooting

• Scaling flaps up and down: the scale-down threshold sits too close to the scale-up threshold, or the cooldown is shorter than warm-up. Widen the gap between triggers and set cooldown >= warm-up + one metric period.
• Scale-up never fires: confirm the metric reaches the evaluator — HPA needs metrics-server (or the custom metrics API for RPS-based scaling), and CloudWatch alarms need data points in the queried namespace. kubectl describe hpa shows “unable to fetch metrics” when the pipeline is broken.
• New instances fail health checks: warm-up is shorter than real boot time. Measure cold start in staging and set stabilizationWindowSeconds / instance warmup above it.
• Scale-down drops in-flight requests: connection draining is missing. Enable ALB deregistration delay or terminationGracePeriodSeconds, and stop accepting new work before exit.
• Costs doubled after enabling auto-scaling: no max cap or spot misuse. Set max replicas, add a budget alert, and restrict spot instances to interruptible work.
• Scale events fire but capacity never changes: check cloud quotas and instance availability in the AZ; an ASG that can’t launch sits at desired capacity with only an activity-log warning.

Further Reading

• AWS Auto Scaling — Dynamic Scaling — the policy types
• Kubernetes HPA — the reference implementation