Service Level Objective (SLO) Document Template

Use this template to define reliability targets that balance user happiness with engineering velocity. See Monitoring and Alerting Guide for metric collection and On-Call Incident Response Guide for escalation procedures.

Template

# SLO: [Service Name]

## Overview
| Field | Value |
|-------|-------|
| **Service** | [name] |
| **Owner** | [team or individual] |
| **Review date** | [quarterly] |

## SLIs (Service Level Indicators)

| SLI | Description | Measurement |
|-----|-------------|-------------|
| **Availability** | Ratio of successful requests | (total - errors) / total |
| **Latency** | Response time distribution | p95, p99 per endpoint |
| **Throughput** | Requests per second | RPS at peak |

## SLOs (Targets)

| Objective | Target | Measurement Window |
|-----------|--------|-------------------|
| Availability | 99.9% | Rolling 30 days |
| Latency p95 | < 200ms | Rolling 7 days |
| Error rate | < 0.1% | Rolling 24 hours |

## Error Budget

- **Budget:** 100% - SLO target (e.g., 0.1% for 99.9% availability)
- **Period:** 30 days
- **Policy:** When error budget is > 50% consumed in < 50% of period, freeze non-critical deploys

## Alerting Thresholds

| Severity | Threshold | Response |
|----------|-----------|----------|
| Page | Error budget 10% consumed in 1 hour | On-call responds immediately |
| Ticket | Error budget 50% consumed in 7 days | Team reviews in next sprint |

## Dependencies

| Dependency | Their SLO | Impact If They Miss |
|------------|-----------|-------------------|
| Payment API | 99.95% | Our checkout SLO drops |
| Identity Provider | 99.9% | Login failures affect availability |

Choosing the Right SLO

User Impact	Typical SLO	Rationale
Critical path (payments, login)	99.99% (4 nines)	Downtime directly blocks revenue
Important but not critical	99.9% (3 nines)	~43 min downtime/month acceptable
Internal tools	99% (2 nines)	~7 hours downtime/month acceptable

Error Budget Policy

Budget remaining | Policy
-------------------|--------
> 50%              | Normal operations
25-50%             | Deploy freeze for risky changes
< 25%              | Deploy freeze except critical fixes
< 10%              | All hands on reliability; halt feature work

What Works

• Start with user-visible metrics — “CPU usage” is not an SLI; “request success rate” is. See Monitoring and Alerting Guide for metric selection.
• Set SLOs based on current performance — if you are at 99.5% today, do not promise 99.99%
• Review quarterly — adjust targets based on user feedback and engineering capacity
• Distinguish SLI, SLO, and SLA — SLI is the metric, SLO is the target, SLA is the contractual promise to customers

Common Mistakes

• SLOs that are too loose — 99% for a payment API means 7 hours of downtime is “acceptable”
• SLOs that are too tight — 99.999% requires expensive infrastructure for marginal user benefit
• Tracking SLIs no one looks at — every SLI needs an owner and a review cadence
• Ignoring error budget burn — the budget exists to protect engineering velocity, not to be ignored. See Incident Postmortem Template for when SLOs are breached.

Troubleshooting

• Pipeline fails silently: enable verbose logging and store pipeline artifacts between stages so you can inspect the exact state that failed.
• Container crashes on startup: check that environment variables, secrets, and config files are mounted correctly. Read the first 50 lines of logs before scaling replicas.
• Deployment rolls back repeatedly: verify health checks, resource limits, and startup probes. A failing readiness probe is a common cause of rolling restarts.
• Slow CI builds: cache dependencies and docker layers. Split large test suites into parallel jobs to reduce wall-clock time.
• Drift between environments: use infrastructure-as-code and immutable artifacts.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the devops and reliability guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply service level objective (slo) document template when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Variants

Context	Approach	Notes
Startup	Basic SLOs: availability and latency	Target Level 2 first
Enterprise	Multi-window SLOs + error budget policies	Level 3-4 with quarterly reviews
Microservices	Per-service SLOs + dependency SLOs	Include upstream service SLOs
Batch processing	Freshness and throughput SLOs	Latency does not apply the same way

Complete SLO Example

=== SLO: payment-service ===

SLI: Transaction success rate
  Definition: HTTP 2xx+3xx requests / total HTTP requests
  Source: Prometheus (http_requests_total)
  Query: sum(rate(http_requests_total{service="payment",status!~"5.."}[5m])) / sum(rate(http_requests_total{service="payment"}[5m]))

SLO: 99.9% success over 30-day window
  Error budget: 0.1% = 43.2 minutes of errors in 30 days
  Fast burn alert: 2% of budget in 1 hour
  Slow burn alert: 5% of budget in 6 hours

SLI: p95 Latency
  Definition: 95th percentile of response time
  Source: Prometheus (http_request_duration_seconds)
  Query: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service="payment"}[5m]))

SLO: p95 < 500ms over 30-day window
  Error budget: 0.1% of requests may exceed 500ms

Review: Quarterly
Owner: Team Payments

Error Budget Policy Enforcement

=== Error Budget Policy ===

Budget remaining | Action
-----------------|-------
75-100%          | Normal development; feature work prioritized
50-75%           | Normal development; monitor trends
25-50%           | Deploy freeze for risky changes
< 25%            | Freeze except critical fixes
< 10%            | All hands on reliability; halt feature work

Exceptions:
- Security hotfixes: always allowed
- Fixes that restore budget: always allowed
- Disabled feature flags: allowed (do not affect budget)

How do we define SLIs that matter to users?

A good SLI measures what users experience, not what the infrastructure does. “CPU usage” is not an SLI — users do not care about CPU. “Request success rate” is an SLI — users notice when their requests fail. To define SLIs: identify the user journey (e.g., “user makes a payment”). For each step in the journey, define what “successful” means (e.g., “transaction completed in < 500ms”). Measure that. Avoid infrastructure SLIs (CPU, memory, disk) unless they are directly correlated with user experience. Use the RED model (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for infrastructure.

How do we adjust SLOs that are too aggressive?

If an SLO is too aggressive (the team consistently burns the error budget): do not lower the SLO immediately. First, investigate why the SLO is not met: are there recurring bugs? unstable dependencies? technical debt? If the cause is systemic: create a reliability improvement plan before adjusting the SLO. If after 1-2 quarters of improvement the SLO is still not met: adjust the SLO to an achievable level. Document the change and the reason. Communicate to stakeholders that the SLO is adjusted to be realistic. Never adjust an SLO after a single bad month — it may be an anomaly. Use 2-3 months of data for SLO adjustment decisions.

How do we communicate SLOs to non-technical stakeholders?

Translate SLOs to business language: “99.9% availability means the service can be down 43 minutes per month.” Instead of “p95 < 500ms”, use “95% of requests complete in under half a second.” Connect SLOs to business impact: “every minute of payment service downtime costs $10,000 in lost transactions.” Use visual dashboards with traffic lights (green/yellow/red) instead of numbers. Report SLOs monthly to leadership. Include remaining error budget and actions taken when it is exhausted. Do not use jargon (SLI, SLO, burn rate) with non-technical stakeholders — translate to impact.

What tools do we need to implement SLOs?

To implement SLOs you need: a monitoring system (Prometheus, Datadog) to collect metrics. A dashboard system (Grafana) to visualize SLOs and error budgets. An alerting system (Alertmanager, PagerDuty) for burn rate-based alerts. Optionally: an SLO tool (Sloth, Prometheus Operator, Nobl9) to automate error budget calculation and alerting. For services without traditional metrics (batch, event-driven): use tools like OpenTelemetry to generate custom metrics. Start simple — Prometheus + Grafana is sufficient for Level 2. Add specialized tools when you reach Level 3-4.

End of document. Review and update quarterly.

Common Production Pitfalls

• Leaving required fields blank or using vague one-word answers.
• Filling the document once and never updating it after scope or decisions change.
• Storing the document where the team does not look during incidents or reviews.
• Not assigning an owner, due date, or review cadence.
• Copying boilerplate without removing sections that do not apply.
• Skipping version control, which makes rollback and accountability impossible.
• Failing to link the document to related decisions or follow-up actions.
• Avoiding quarterly reviews that would retire stale or unused sections.