Dependency Upgrade Runbook

Overview

Outdated dependencies expose projects to security vulnerabilities, compatibility issues, and missing capabilities. This runbook provides a repeatable process for upgrading dependencies safely with minimal risk.

When to Use

Use this resource when:

• A critical security patch is released for a direct or transitive dependency
• Major version upgrades are required for long-term support
• Quarterly or sprint-based maintenance windows for dependency refreshes

Solution

# Dependency Upgrade Runbook

## 1. Preparation

- [ ] Identify the dependency and target version
- [ ] Review the changelog / release notes for breaking changes
- [ ] Check open issues in the dependency repository for upgrade-related bugs
- [ ] Create a dedicated branch: `deps/upgrade-<name>-<version>`
- [ ] Ensure CI is green on the current main branch

## 2. Upgrade

- [ ] Update the version in `package.json`, `requirements.txt`, `pom.xml`, etc.
- [ ] Run the dependency installation command
- [ ] Check for peer dependency warnings or conflicts
- [ ] Run automated tests (unit, integration, lint)
- [ ] Run smoke tests against a local or staging environment

## 3. Validation

- [ ] Review test coverage reports for regressions
- [ ] Check application logs for new warnings or errors
- [ ] Verify critical user paths manually if behavior changed
- [ ] Run security scan (`npm audit`, `safety check`, OWASP dependency check)

## 4. Rollback Plan

- [ ] Tag the last known good commit before merge
- [ ] Document any manual data or config changes required
- [ ] Confirm rollback can be executed within 15 minutes

## 5. Merge & Monitor

- [ ] Open a pull request with changelog summary
- [ ] Deploy to staging and let it soak for 24 hours
- [ ] Deploy to production during low-traffic window
- [ ] Monitor error rates and latency for 48 hours post-deploy

Explanation

The runbook breaks upgrades into five phases to reduce risk. Preparation prevents surprises by reviewing changelogs. The Upgrade phase isolates changes in a branch. Validation uses automated and manual checks. The Rollback Plan ensures fast recovery. Merge & Monitor completes the cycle with production observation.

Variants

Context	Approach	Notes
Security patch	Fast-track branch	Skip soak time only for CVEs with active exploits
Major version	Feature-flag rollout	Isolate new behavior behind flags during transition
Monorepo	Batch upgrades	Upgrade shared libs first, then consumers

What works

1. Upgrade one major dependency at a time to simplify debugging
2. Pin exact versions in lock files (package-lock.json, poetry.lock) and commit them
3. Use automated tools like Dependabot or Renovate for patch and minor upgrades
4. Maintain a deprecation calendar for end-of-life dependencies
5. Document all breaking changes and migration steps in the pull request

Common Mistakes

1. Upgrading multiple major dependencies simultaneously, making failures hard to attribute
2. Ignoring peer dependency warnings that cause runtime errors
3. Skipping the rollback plan, extending downtime when issues surface
4. Not reviewing transitive dependency changes in lock files
5. Deploying during peak traffic without a soak period

Troubleshooting

• Pipeline fails silently: enable verbose logging and store pipeline artifacts between stages so you can inspect the exact state that failed.
• Container crashes on startup: check that environment variables, secrets, and config files are mounted correctly. Read the first 50 lines of logs before scaling replicas.
• Deployment rolls back repeatedly: verify health checks, resource limits, and startup probes. A failing readiness probe is a common cause of rolling restarts.
• Slow CI builds: cache dependencies and docker layers. Split large test suites into parallel jobs to reduce wall-clock time.
• Drift between environments: use infrastructure-as-code and immutable artifacts.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the devops and dependencies guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply dependency upgrade runbook when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Advanced Solutions

Automated dependency upgrade pipeline with Renovate

Configure Renovate to automate patch and minor upgrades with auto-merge rules:

{
  "extends": ["config:base"],
  "schedule": ["before 6am on Monday"],
  "automerge": true,
  "automergeType": "pr",
  "packageRules": [
    {
      "updateTypes": ["patch", "minor"],
      "automerge": true,
      "groupName": "patch and minor updates"
    },
    {
      "updateTypes": ["major"],
      "automerge": false,
      "labels": ["major-upgrade", "needs-review"],
      "dependencyDashboardApproval": true
    },
    {
      "depTypeList": ["devDependencies"],
      "automerge": true,
      "schedule": ["at any time"]
    }
  ],
  "vulnerabilityAlerts": {
    "enabled": true,
    "labels": ["security"],
    "schedule": ["at any time"]
  }
}

Rollback script for npm upgrades

A bash script to quickly revert a failed dependency upgrade:

#!/bin/bash
set -euo pipefail

BRANCH=$(git rev-parse --abbrev-ref HEAD)
TAG="pre-upgrade-$(date +%Y%m%d-%H%M%S)"

# Create a safety tag before proceeding
git tag "$TAG"
echo "Created safety tag: $TAG"

# If upgrade fails, rollback:
# git checkout "$TAG" -- package.json package-lock.json
# npm ci
# git checkout main
# git branch -D "$BRANCH"
# git tag -d "$TAG"

echo "To rollback: git checkout $TAG -- package.json package-lock.json && npm ci"

Python dependency upgrade with pip-tools

Use pip-tools to manage pinned requirements with separate source and locked files:

#!/bin/bash
set -euo pipefail

# requirements.in contains unpinned or loosely pinned deps
# requirements.txt is the locked, fully resolved output

# Upgrade a single package to a specific version
echo "package-name==2.0.0" >> requirements.in

# Recompile locked requirements
pip-compile --upgrade-package package-name --output-file requirements.txt requirements.in

# Verify no conflicting transitive deps
pip install -r requirements.txt --dry-run

# Run tests
pytest tests/ -x

# If all passes, commit both files
git add requirements.in requirements.txt
git commit -m "deps: upgrade package-name to 2.0.0"

Dependency audit dashboard with npm audit + cyclonedx

Generate an SBOM (Software Bill of Materials) and audit report for compliance:

#!/bin/bash
set -euo pipefail

# Generate CycloneDX SBOM
npx @cyclonedx/cyclonedx-npm --output-file sbom.json

# Run audit and export JSON
npm audit --json > audit-report.json

# Extract high and critical vulnerabilities
node -e "
const audit = require('./audit-report.json');
const vulns = audit.vulnerabilities || {};
const high = Object.entries(vulns).filter(([k,v]) => v.severity === 'high' || v.severity === 'critical');
if (high.length > 0) {
  console.log('HIGH/CRITICAL vulnerabilities:');
  high.forEach(([name, info]) => console.log('  ' + name + ': ' + info.severity));
  process.exit(1);
} else {
  console.log('No high or critical vulnerabilities found.');
}
"

Additional Best Practices

• For a deeper guide, see On-Call Runbook Template.

1. Use npm ci instead of npm install in CI. The ci command deletes node_modules and installs exactly from the lock file. It fails if lock file is out of sync with package.json, catching incomplete upgrades:

# GitHub Actions example
- name: Install dependencies
  run: npm ci

2. Set up Dependabot security alerts as required checks. Configure branch protection rules so that security PRs from Dependabot bypass review requirements but still need CI to pass:

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 10
    groups:
      patch-and-minor:
        update-types: ["patch", "minor"]

Additional Common Mistakes

1. Upgrading devDependencies without testing the build pipeline. Dev dependencies like webpack, babel, or eslint can break the build output even if tests pass. Always run a full production build after upgrading devDependencies:

npm run build && npm run test

2. Ignoring deprecation warnings during upgrades. Deprecation warnings in one minor version often become errors in the next major version. Track them in your issue tracker:

# Capture deprecation warnings during test runs
npm test 2>&1 | grep -i "deprecat" > deprecation-warnings.txt

Common Production Pitfalls

• Leaving required fields blank or using vague one-word answers.
• Filling the document once and never updating it after scope or decisions change.
• Storing the document where the team does not look during incidents or reviews.
• Not assigning an owner, due date, or review cadence.
• Copying boilerplate without removing sections that do not apply.
• Skipping version control, which makes rollback and accountability impossible.
• Failing to link the document to related decisions or follow-up actions.
• Avoiding quarterly reviews that would retire stale or unused sections.