StackPractices
ES
intermediate By Mathias Paulenko

Change Management Template

A template for documenting CAB reviews and rollback criteria for production changes.

devops management rollback operations template
Topics: devops

Overview

Production changes are risky. A database migration, a config update, or a feature flag flip can cascade into an outage, and the postmortem usually finds the same root cause: nobody wrote down what could go wrong or how to get back. This template documents the review, approval, and rollback workflow for any change that touches production, so the right people see the risk before it ships and the rollback path is tested before you need it. If you also need a checklist for the deploy itself, pair it with the deployment checklist template.

When to Use

Use this resource when:

  • Introducing a change to production infrastructure, databases, or configuration
  • Your compliance framework (SOC 2, ISO 27001) requires documented change approval
  • A recent outage was caused by an unreviewed or untested change
  • You’re formalizing an approval process that currently lives in Slack threads

Skip it for routine, fully reversible work. A bug triage template fits day-to-day incident work better, and tiny copy or doc fixes don’t need a CAB review.

Solution

# Change Request: `<Title>`

## 1. Change Summary

| Field | Value |
|-------|-------|
| Change ID | `CHG-YYYY-NNNN` |
| Title | `description` |
| Requester | `@name` |
| Team | `team` |
| Environment | `staging / production` |
| Scheduled Date / Time | `YYYY-MM-DD HH:MM UTC` |
| Duration (expected) | `X minutes / hours` |
| Risk Level | `Low / Medium / High / Critical` |

## 2. Change Description

**What is changing?**
[Describe the technical change in one paragraph.]

**Why is it needed?**
[Link to ticket, incident, or business justification.]

**What are the expected results?**
[Measurable outcome: latency reduced by X%, feature enabled for Y% of users.]

## 3. Impact Analysis

| System / Service | Impact | Validation Method |
|------------------|--------|-------------------|
| | | |

### Dependencies
- [ ] No dependent services affected
- [ ] Dependent services notified: `list`
- [ ] External partners / customers notified: `list`

### Risk Assessment

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| | | | |

## 4. Rollback Plan

| Condition | Rollback Action | Time to Complete | Owner |
|-----------|----------------|-------------------|-------|
| Change fails during deployment | `git revert` / `terraform destroy -target` / config rollback | 10 min | `@name` |
| Performance degradation > 20% | Rollback to previous image / revert migration | 15 min | `@name` |
| Data inconsistency detected | Restore from pre-change snapshot | 30 min | `@name` |
| Unknown failure mode | Page on-call; execute emergency rollback runbook | 5 min | `@on-call` |

- [ ] Rollback has been tested in staging
- [ ] Rollback command is documented in runbook
- [ ] Database snapshot / backup taken before change
- [ ] Feature flag kill switch is ready (if applicable)

## 5. Approval

| Role | Name | Approved | Date |
|------|------|----------|------|
| Requester | | | |
| Technical Reviewer | | | |
| Product Owner (if user-facing) | | | |
| Security (if data / access change) | | | |
| CAB / Manager (High+ risk) | | | |

## 6. Execution Log

| Time (UTC) | Step | Result | Notes |
|------------|------|--------|-------|
| | | | |

## 7. Post-Change Review

- [ ] Change completed as planned
- [ ] Monitoring shows no anomalies for 1 hour post-change
- [ ] Rollback plan archived / updated based on lessons
- [ ] Ticket closed with outcome summary

Explanation

The template enforces three gates before any production change: a description that justifies the change, an impact analysis that surfaces hidden dependencies, and a rollback plan that’s ready before the change starts. The risk level decides who approves: low-risk changes may only need a peer review, while high-risk changes require a Change Advisory Board (CAB) and explicit sign-off from security. The execution log creates an audit trail that’s invaluable during postmortems and compliance audits.

flowchart diagram: Submit change request

Change Risk Assessment Matrix

=== Change Risk Assessment ===

Risk Level Criteria:

CRITICAL:
  - Changes to authentication or authorization
  - Database schema changes affecting production data
  - Network infrastructure changes (VPC, firewall, DNS)
  - Changes during freeze periods (Black Friday, product launch)
  - Changes affecting > 50% of user traffic
  Approval: CAB + Security + Engineering Director

HIGH:
  - Changes to production configuration
  - API contract changes affecting external consumers
  - Infrastructure scaling changes
  - Database index additions/removals
  - Changes affecting 10-50% of user traffic
  Approval: CAB + Team Lead

MEDIUM:
  - Feature flag toggles in production
  - Non-breaking API additions
  - Configuration parameter updates
  - Dependency version bumps
  Approval: Team Lead + Peer Review

LOW:
  - CSS/styling changes
  - Copy/text updates
  - Documentation updates
  - Non-production environment changes
  Approval: Peer Review

Change Calendar and Collision Detection

=== Change Calendar (Week of 2026-07-08) ===

Monday    2026-07-08
  10:00  [MEDIUM] Deploy API v2.3.1 (team-a)     Status: Approved
  14:00  [LOW]     Update docs site (team-b)      Status: Approved

Tuesday   2026-07-09
  09:00  [HIGH]    DB migration: add index (dba)  Status: Pending CAB
  15:00  [MEDIUM]  Feature flag: checkout-v2 (team-c)  Status: Approved

Wednesday 2026-07-10
  --     FREEZE DAY (product launch)              No changes allowed

Thursday  2026-07-11
  10:00  [CRITICAL] Auth service upgrade (sec)    Status: Pending CAB
  11:00  [MEDIUM]  Deploy frontend v3.1 (team-d)  Status: Pending
  COLLISION DETECTED: Auth upgrade + frontend deploy at same time
  RESOLUTION: Move frontend deploy to 14:00

Friday    2026-07-12
  09:00  [LOW]     Cleanup unused S3 buckets (platform)  Status: Approved
  --     No HIGH/CRITICAL changes after 12:00 (weekend safety)

Post-Change Review Template

=== Post-Change Review ===

Change ID: CHG-2026-07-11-001
Title: Deploy API v2.3.1
Risk Level: MEDIUM
Change Owner: alice@example.com
Reviewer: bob@example.com

Execution Summary:
  Planned start:  10:00 UTC
  Actual start:   10:02 UTC
  Planned end:    10:30 UTC
  Actual end:     10:35 UTC
  Duration variance: +5 minutes

Outcome:
  [x] Change completed as planned
  [x] All deployment steps succeeded
  [x] Smoke tests passed
  [x] Monitoring shows no anomalies (1h post-change)
  [x] No rollback needed
  [x] Ticket closed with outcome summary

Issues Encountered:
  - Migration took 3 min longer than expected due to table size
  - No user impact

Lessons Learned:
  - Update migration time estimate for tables > 10M rows
  - Consider pre-warming cache after migration

Follow-up Actions:
  - Update runbook with new migration time estimate (alice, by 2026-07-18)
  - Add cache warming step to deployment script (platform, by 2026-07-25)

Example: A Filled Change Request

A blank template only gets you so far. Here’s what the change request looks like filled in for a real High-risk database change:

# Change Request: Add composite index on orders table

## 1. Change Summary

| Field | Value |
|-------|-------|
| Change ID | `CHG-2026-0142` |
| Title | Add composite index `orders(user_id, created_at)` |
| Requester | `@maria.chen` |
| Team | `platform` |
| Environment | `production` |
| Scheduled Date / Time | `2026-10-22 09:00 UTC` |
| Duration (expected) | `20 minutes` |
| Risk Level | `High` |

## 2. Change Description

**What is changing?**
Adding a composite index on `orders(user_id, created_at)` to fix the 1.8s p95
on the order-history endpoint. The migration runs `CREATE INDEX CONCURRENTLY`
so the table stays writable during the build.

**Why is it needed?**
INC-3391 — order-history queries time out for accounts with > 2M orders.

**What are the expected results?**
p95 latency for `/orders/history` drops from 1.8s to under 300ms. Verified in
staging on a production-size dataset.

## 3. Impact Analysis

| System / Service | Impact | Validation Method |
|------------------|--------|-------------------|
| orders Postgres (primary) | Index build on an 8GB table, ~15 min | `pg_stat_progress_create_index` |
| order-history API | Read-only; no code change | Smoke test + Grafana latency panel |
| Reporting replica | Replication lag possible during build | `pg_stat_replication` lag metric |

### Dependencies
- [x] No dependent services affected
- [x] Dependent services notified: `#team-data` (read-replica consumers)
- [x] External partners / customers notified: n/a — no downtime expected

### Risk Assessment

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Index build locks writes | Low | High | `CONCURRENTLY` avoids the lock; tested on staging |
| Replication lag breaks dashboards | Medium | Medium | Run in the 09:00 UTC low-traffic window; watch lag |
| Disk space exhausted | Low | Medium | 40GB free; index needs ~6GB |

## 4. Rollback Plan

| Condition | Rollback Action | Time to Complete | Owner |
|-----------|----------------|-------------------|-------|
| Lock contention blocks writes > 60s | `pg_cancel_backend` on the index build, retry off-peak | 2 min | `@maria.chen` |
| Replication lag > 5 min | Pause build; resume when lag clears | 10 min | `@maria.chen` |
| Query plan regresses other queries | `DROP INDEX CONCURRENTLY orders_user_created_idx` | 5 min | `@dba-on-call` |

- [x] Rollback tested in staging (index dropped cleanly)
- [x] Rollback command documented in `runbooks/postgres-index-ops.md`
- [x] Schema snapshot taken 2026-10-21
- [x] Feature flag kill switch: n/a — no flag involved

## 5. Approval

| Role | Name | Approved | Date |
|------|------|----------|------|
| Requester | maria.chen | yes | 2026-10-20 |
| Technical Reviewer | j.okafor | yes | 2026-10-20 |
| CAB / Manager (High+ risk) | s.ramirez | yes | 2026-10-21 |

## 6. Execution Log

| Time (UTC) | Step | Result | Notes |
|------------|------|--------|-------|
| 09:02 | Migration started | OK | `CREATE INDEX CONCURRENTLY` running |
| 09:17 | Index build complete | OK | 15 min; replication lag peaked at 90s |
| 09:20 | Smoke tests | OK | p95 now 240ms |

## 7. Post-Change Review

- [x] Change completed as planned
- [x] Monitoring clean for 1 hour post-change
- [x] Follow-up logged: index builds on 8GB tables take ~15 min, not 10

Variants

ContextApproval GateFocus
Startup / small teamPeer review + team leadSpeed; lightweight documentation
Enterprise / regulatedCAB + security + complianceFull audit trail; formal sign-off
Infrastructure (Terraform)IaC review + plan diffVerify no destructive changes
Database migrationDBA review + rollback scriptData integrity is the priority
Feature flag rolloutProduct + engineeringGradual exposure; instant rollback via flag
Emergency changePost-hoc approval within 24 hoursFix first, document second—but document

How to Adopt This Template

Rolling out a heavy change process all at once usually fails — teams route around anything that slows them down. Adopt it by risk level instead:

  • Weeks 1-2: use only sections 1-4 (summary, description, impact, rollback) and require a peer reviewer for everything above Low risk. Skip the CAB entirely.
  • Weeks 3-4: add the change calendar and the approval table. Start classifying changes with the risk matrix so the team calibrates what “High” actually means.
  • Month 2: introduce CAB review for High and Critical changes only, and start tracking the metrics from the FAQ (incident rate, rollback rate). Publish freeze windows for the next quarter.
  • Ongoing: review last month’s post-change reviews in the ops meeting. Retire fields nobody fills in — a template that costs 30 minutes to complete gets ignored.

The trap to avoid is making every change follow the heaviest path. If a Low-risk CSS fix needs the same paperwork as a schema migration, engineers will classify everything as Low and the process loses meaning.

What Works

  1. Classify risk objectively; if the change touches billing, authentication, or data, it’s at least Medium
  2. Never approve your own change; require at least one independent reviewer
  3. Test the rollback in staging, not just the forward change
  4. Schedule high-risk changes during low-traffic windows with full team availability
  5. Keep a change calendar visible to all teams to avoid collisions — two risky changes at once is how incidents compound

Common Mistakes

  1. Treating “simple” changes as low-risk without an impact analysis
  2. Not notifying downstream teams who depend on the changed service
  3. Skipping rollback testing because “it will probably work”
  4. Approving changes in a Slack DM instead of a documented record
  5. Not reviewing the change after execution — the lessons are lost if you don’t close the loop

Troubleshooting

  • CAB meetings become the bottleneck: batch High and Critical reviews into one weekly slot and approve Medium changes asynchronously in the ticket. If every change waits for a meeting, people start labeling their work as emergencies.
  • Changes ship without a tested rollback: block approval until the rollback section is filled in, and spot-check quarterly that a documented rollback actually ran in staging. A rollback plan nobody tested is decoration.
  • Collisions surface during the incident, not before: keep one shared calendar with automated conflict flags for two High+ changes in the same window or any change during a freeze. Review it in the weekly ops meeting, not ad hoc.
  • “Emergency” becomes the default lane: track the ratio of emergency to planned changes monthly. If it stays above ~10%, the planning process is too slow or teams are routing around it — fix the process, not the label.
  • Approvals happen in chat and leave no record: require the approval table inside the change ticket itself. A thumbs-up on Slack isn’t auditable; a name and a date in the change request is.

Production Notes

  • Schedule High and Critical changes in low-traffic windows with the on-call and the owning team both online; never on a Friday afternoon or right before a freeze.
  • Watch the golden signals — error rate, p95/p99 latency, saturation — for at least one hour post-change before closing the ticket.
  • Drill the rollback quarterly: pick a staging change, execute its documented rollback, and update the runbook with whatever broke.
  • Announce start and end in the shared channel so downstream teams can correlate anomalies to a change window instead of guessing.

Key Takeaways

  • The template’s real job is the rollback plan — everything else exists so you never need it blind.
  • Risk level drives ceremony: Low gets a peer review, Critical gets the CAB. One-size approvals train people to bypass them.
  • Fill it in before the change, not during the incident — a change request written retroactively is a postmortem, not a control.
  • Close the loop in the post-change review; the lessons feed the next request’s risk assessment.

Common Production Pitfalls

  • Leaving required fields blank or using vague one-word answers.
  • Filling the document once and never updating it after scope or decisions change.
  • Storing the document where the team doesn’t look during incidents or reviews.
  • Not assigning an owner, due date, or review cadence.
  • Copying boilerplate without removing sections that don’t apply.
  • Skipping version control, which makes rollback and accountability impossible.
  • Failing to link the document to related decisions or follow-up actions.
  • Avoiding quarterly reviews that would retire stale or unused sections.

Further Reading

Frequently Asked Questions

When does a change need CAB approval?

High and Critical risk changes should go through a Change Advisory Board or an equivalent senior review. Criteria include: changes to production databases, modifications to authentication or authorization, infrastructure that affects more than 50% of traffic, or changes during a known sensitive period like Black Friday. Medium risk changes may only need team-lead approval, and low-risk changes like CSS fixes or copy updates can use peer review.

How do I handle emergency changes that can't wait for full approval?

Document an emergency change process: the on-call engineer executes the fix, then creates a retroactive change request within 24 hours with the full template filled in. The emergency path should require verbal approval from a manager or team lead, recorded in Slack or on a bridge call. Don't let "emergency" become an excuse for skipping the process every week.

Should infrastructure-as-code changes use this template?

Yes, but integrate it with your IaC review process. The change request should reference the pull request and the terraform plan diff. The risk assessment focuses on whether the plan shows destructive changes like resource replacement, or just additive ones. The rollback plan for IaC is often a git revert plus re-apply, but verify that the state file stays consistent.

How do we implement change freezes?

Define freeze periods around business events: product launches, holidays, sales events like Black Friday, or regulatory deadlines. During a freeze, only emergency changes are allowed, and they need VP-level approval. Communicate freeze dates at least two weeks ahead, and block deployments in CI/CD during freezes with an override for real emergencies. After the freeze ends, do a quick debrief on any emergency changes that went through.

What is a Change Advisory Board (CAB)?

A CAB is a group of stakeholders who review and approve high-risk changes. Typical members: an engineering manager, a security representative, an operations lead, and a product owner. The CAB meets on a regular cadence — weekly or twice a week — to review upcoming changes, and can convene emergency sessions for urgent ones. Document CAB decisions in the change management system so you have an audit trail.

How do we track change success rate?

Track a small set of metrics: total changes per month, the percentage that caused incidents, the percentage that required a rollback, average time to complete a change, and the percentage with complete documentation. Reasonable targets are under 5% incident rate and under 10% rollback rate. Review the numbers monthly and look for patterns in failed changes — specific services, change types, or time windows — then fix the process or the training, not just the individual change.

How do we handle rollback when the change can't be reversed?

Some changes are irreversible: migrations that drop columns, data deletions, destructive schema changes. For those, write a forward-fix plan instead of a rollback plan. It documents how to restore functionality when the original state can't be restored, and you test it in staging like any other rollback. When possible, split irreversible changes into phases — deprecate first, verify for a period, then remove in a later change.

How do we integrate change management with GitOps?

In a GitOps workflow, the pull request is the change request. Link the PR to the change management system, use labels for risk classification (low, medium, high, critical), and require approvals based on risk: one reviewer for low, two for medium, CAB sign-off for high. The merge commit is the execution step, and tools like Argo CD or Flux track the deployment status. Put the impact analysis and rollback plan in the PR description, and close the change ticket automatically when the deploy succeeds.

What do we do with out-of-hours emergency changes?

Keep a separate emergency lane from the normal flow. Require approval from the on-call lead and notify the CAB through Slack or PagerDuty, then document the change retroactively within 24 hours. If the emergency change caused or touched an incident, run a postmortem on it. Limit the emergency lane to critical bug fixes and security patches — it isn't a shortcut around planned reviews.