Golden Master Testing: Characterize Legacy Code Safely

Overview

You inherited a 10-year-old Python service with 40,000 lines of code and zero tests. Your boss wants you to refactor it to use async/await, but how do you know you didn’t break anything? You can’t write unit tests for code you don’t fully understand.

Golden master testing (also called characterization testing, a term coined by Michael Feathers in Working Effectively with Legacy Code) captures the current behavior of a system as a baseline, then verifies that future changes don’t alter that behavior. You don’t assert what the code should do. You assert what it currently does. The “golden master” is a reference output that serves as truth. When you refactor, any deviation from the golden master signals a regression.

When to Use

• Legacy code with no tests that you need to refactor safely
• Code where the business rules are unclear or undocumented
• Systems where the output is deterministic but complex (reports, calculations, transformations)
• Before migrating a system to a new language or framework
• When you need confidence to change code you don’t fully understand

When NOT to Use

• New code with clear requirements: write tests first (TDD) instead of characterizing behavior you already understand.
• When output is non-deterministic (random values, timestamps, network calls) unless you normalize them first.
• For simple functions where explicit assertions are clearer and cheaper to maintain.
• When you understand the code well enough to write targeted tests against specific units.
• For code that’s about to be replaced entirely: don’t characterize what you’ll delete.

Solution

Basic golden master approach

# Python — capture output as golden master
import json
import subprocess
import hashlib

def run_legacy_system(inputs):
    """Run the legacy system with given inputs and capture output."""
    result = subprocess.run(
        ['python', 'legacy_calculator.py'],
        input=json.dumps(inputs),
        capture_output=True,
        text=True,
    )
    return result.stdout

def generate_golden_master(input_file):
    """Generate golden master from input file."""
    with open(input_file) as f:
        inputs = json.load(f)

    outputs = []
    for input_data in inputs:
        output = run_legacy_system(input_data)
        outputs.append({
            'input': input_data,
            'output': output,
            'hash': hashlib.sha256(output.encode()).hexdigest(),
        })

    with open('golden_master.json', 'w') as f:
        json.dump(outputs, f, indent=2)

    print(f"Golden master generated with {len(outputs)} cases")

# Step 1: Generate golden master
generate_golden_master('test_inputs.json')

Comparing against golden master

# Python — verify refactored code matches golden master
import json
import subprocess
import hashlib
import pytest

def load_golden_master():
    with open('golden_master.json') as f:
        return json.load(f)

def run_refactored_system(input_data):
    """Run the refactored system."""
    result = subprocess.run(
        ['python', 'refactored_calculator.py'],
        input=json.dumps(input_data),
        capture_output=True,
        text=True,
    )
    return result.stdout

@pytest.mark.parametrize("case", load_golden_master())
def test_matches_golden_master(case):
    input_data = case['input']
    expected_output = case['output']
    expected_hash = case['hash']

    actual_output = run_refactored_system(input_data)
    actual_hash = hashlib.sha256(actual_output.encode()).hexdigest()

    assert actual_hash == expected_hash, (
        f"Output mismatch for input: {input_data}\n"
        f"Expected: {expected_output[:200]}\n"
        f"Actual:   {actual_output[:200]}"
    )

Golden master with file-based output

# Python — compare file output (e.g., generated reports)
import os
import filecmp
import subprocess
import pytest

def generate_report(input_file, output_dir, system='legacy'):
    """Generate a report using either legacy or refactored system."""
    script = f'{system}_report_generator.py'
    subprocess.run([
        'python', script,
        '--input', input_file,
        '--output', output_dir,
    ], check=True)

@pytest.fixture(scope="session")
def golden_master_reports(tmp_path_factory):
    """Generate golden master reports once per session."""
    golden_dir = tmp_path_factory.mktemp("golden")
    test_inputs = [
        'tests/data/report_input_1.json',
        'tests/data/report_input_2.json',
        'tests/data/report_input_3.json',
    ]

    for i, input_file in enumerate(test_inputs):
        output_file = golden_dir / f'report_{i}.html'
        generate_report(input_file, str(output_file), system='legacy')

    return golden_dir, test_inputs

def test_refactored_matches_golden(golden_master_reports, tmp_path):
    golden_dir, test_inputs = golden_master_reports

    for i, input_file in enumerate(test_inputs):
        # Generate with refactored system
        refactored_output = tmp_path / f'report_{i}.html'
        generate_report(input_file, str(refactored_output), system='refactored')

        # Compare
        golden_file = golden_dir / f'report_{i}.html'
        assert filecmp.cmp(golden_file, refactored_output, shallow=False), (
            f"Report mismatch for {input_file}"
        )

Golden master with large input sets

# Python — generate large input sets for thorough characterization
import random
import json

def generate_test_inputs(count=1000):
    """Generate diverse inputs to exercise the legacy system."""
    inputs = []

    # Normal cases
    for i in range(count // 2):
        inputs.append({
            'operation': random.choice(['add', 'subtract', 'multiply', 'divide']),
            'a': random.uniform(-1000, 1000),
            'b': random.uniform(-1000, 1000),
        })

    # Edge cases
    inputs.extend([
        {'operation': 'add', 'a': 0, 'b': 0},
        {'operation': 'divide', 'a': 1, 'b': 0},  # Division by zero
        {'operation': 'multiply', 'a': -1, 'b': -1},
        {'operation': 'add', 'a': 1e10, 'b': -1e10},
        {'operation': 'subtract', 'a': 0.1, 'b': 0.3},
    ])

    # Boundary values
    inputs.extend([
        {'operation': 'add', 'a': float('inf'), 'b': 1},
        {'operation': 'add', 'a': float('-inf'), 'b': 1},
        {'operation': 'add', 'a': float('nan'), 'b': 1},
    ])

    with open('test_inputs.json', 'w') as f:
        json.dump(inputs, f, indent=2)

    print(f"Generated {len(inputs)} test inputs")

Approval testing approach

# Python — approval testing with approvaltests library
from approvaltests import Approvals
from approvaltests.reporters import PythonNativeReporter

def test_legacy_formatter_output():
    # Run the legacy formatter with test input
    input_data = [
        {"name": "Alice", "age": 30, "role": "admin"},
        {"name": "Bob", "age": 25, "role": "member"},
        {"name": "Charlie", "age": 35, "role": "guest"},
    ]

    output = legacy_formatter.format_table(input_data)

    # First run: creates approved file (golden master)
    # Subsequent runs: compares against approved file
    # If mismatch: opens diff tool for review
    Approvals.verify(output, reporter=PythonNativeReporter())

def test_legacy_calculator_with_multiple_inputs():
    results = []
    for a, b, op in [(1, 2, 'add'), (10, 5, 'subtract'), (3, 4, 'multiply')]:
        result = legacy_calculator.calculate(a, b, op)
        results.append(f"{a} {op} {b} = {result}")

    output = '\n'.join(results)
    Approvals.verify(output)

JavaScript golden master

// JavaScript — golden master with Jest snapshot
const { execSync } = require('child_process');

function runSystem(input, system = 'legacy') {
  const result = execSync(`node ${system}_calculator.js`, {
    input: JSON.stringify(input),
    encoding: 'utf-8',
  });
  return result;
}

describe('Golden Master: legacy vs refactored', () => {
  const testCases = require('./test_inputs.json');

  test.each(testCases)('matches golden master for %j', (input) => {
    const legacyOutput = runSystem(input, 'legacy');
    const refactoredOutput = runSystem(input, 'refactored');

    expect(refactoredOutput).toBe(legacyOutput);
  });
});

// Alternative: use Jest snapshots as golden master
describe('Golden Master with snapshots', () => {
  const testCases = require('./test_inputs.json');

  test.each(testCases)('output matches snapshot for %j', (input) => {
    const output = runSystem(input, 'legacy');
    expect(output).toMatchSnapshot();
  });
});

Incremental refactoring with golden master

# Python — step-by-step refactoring with golden master safety net

# Step 1: Generate golden master (before any changes)
# python generate_golden_master.py

# Step 2: Write characterization tests
class TestCharacterization:
    @pytest.mark.parametrize("case", load_golden_master())
    def test_legacy_behavior_preserved(self, case):
        output = run_current_system(case['input'])
        assert output == case['output']

# Step 3: Refactor in small steps, running tests after each change
# - Extract method
# - Rename variable
# - Replace conditional with polymorphism
# - Each step: run golden master tests

# Step 4: Once refactored, write proper unit tests for the new structure
# - Delete golden master tests once unit tests provide equal confidence

Handling non-deterministic output

# Python — normalize non-deterministic values before comparison
import re
from datetime import datetime

def normalize_output(output):
    """Remove non-deterministic elements from output."""
    # Replace timestamps
    output = re.sub(
        r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+Z?',
        '<TIMESTAMP>',
        output,
    )
    # Replace UUIDs
    output = re.sub(
        r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}',
        '<UUID>',
        output,
    )
    # Replace random IDs
    output = re.sub(r'"id": \d+', '"id": <ID>', output)
    # Replace file paths
    output = re.sub(r'/tmp/[^\s"]+', '<TMP_PATH>', output)
    return output

def test_with_normalization():
    input_data = {'action': 'create', 'name': 'Alice'}
    legacy_output = normalize_output(run_legacy_system(input_data))
    refactored_output = normalize_output(run_refactored_system(input_data))

    assert refactored_output == legacy_output

Explanation

Think of the legacy system as a black box. You feed it inputs, capture what comes out, and store that as the baseline. After refactoring, run the same inputs through the new code and compare. Matching outputs mean the refactoring preserved behavior. A mismatch means you caught a regression before it reached production.

Trade-offs

Setup time is the first cost. Generating a golden master for a complex system can take days: you need to identify all entry points, handle non-deterministic output (timestamps, UUIDs, random IDs), and generate enough inputs to cover edge cases. For a 40k-line service, expect 2-3 days of setup before you write a single refactoring commit. The baseline also grows linearly with input count — 10,000 inputs produce a 2MB JSON file. For large baselines, store SHA-256 hashes instead of full output: a hash comparison is O(1) while string comparison is O(n) in output size.

These tests aren’t permanent. Golden master tests are a temporary safety net. Once you’ve refactored the code and written proper unit tests (using the Test Pyramid Pattern as a guide), the golden master becomes redundant. Keep it only as long as you need it.

When the golden master itself has a bug

The golden master captures current behavior, bugs included. If you discover a bug in the legacy code, fix it and regenerate the golden master. Document that the behavior changed intentionally. Never update the golden master silently: a diff might be a regression, not an improvement. Review every change as carefully as you would review a pull request.

Comparison with Snapshot Testing

Snapshot testing is one specific implementation of golden master testing. Golden master is the concept; snapshots are one tool. Jest 29 snapshots serialize React component output to a file and compare on each run. Other tools include file comparison (filecmp), hash comparison (SHA-256), and approval testing with the approvaltests Python library or ApprovalTests.Net for .NET.

Parallel run in production

# Python — run legacy and refactored in parallel, compare results
import logging

def parallel_run(input_data):
    """Run both systems and log differences."""
    try:
        legacy_result = run_legacy_system(input_data)
        refactored_result = run_refactored_system(input_data)

        if legacy_result != refactored_result:
            logging.warning(
                f"Output mismatch detected!\n"
                f"Input: {input_data}\n"
                f"Legacy: {legacy_result[:200]}\n"
                f"Refactored: {refactored_result[:200]}"
            )
            return False
        return True
    except Exception as e:
        logging.error(f"Error in parallel run: {e}")
        return False

# Run in production with shadow traffic
# Log mismatches but serve from legacy system

Golden master with database state

# Python — capture database state as golden master
import json
import hashlib

def capture_db_state(db_connection):
    """Capture current database state as a hash."""
    tables = ['users', 'orders', 'products']
    state = {}

    for table in tables:
        rows = db_connection.execute(f'SELECT * FROM {table} ORDER BY id')
        state[table] = [dict(row) for row in rows]

    state_json = json.dumps(state, sort_keys=True, default=str)
    return hashlib.sha256(state_json.encode()).hexdigest()

def test_database_state_unchanged_after_refactor():
    # Setup: populate database with test data
    setup_test_data(db)

    # Capture golden master state
    golden_hash = capture_db_state(db)

    # Run the refactored operation
    refactored_service.process_orders()

    # Verify database state matches
    current_hash = capture_db_state(db)
    assert current_hash == golden_hash, "Database state changed unexpectedly"

Golden master with API responses

// JavaScript — capture API responses as golden master
const axios = require('axios');

async function captureApiResponses(baseUrl) {
  const endpoints = [
    { method: 'GET', path: '/api/users' },
    { method: 'GET', path: '/api/users/1' },
    { method: 'GET', path: '/api/orders' },
    { method: 'POST', path: '/api/orders', body: { userId: 1, total: 100 } },
  ];

  const responses = [];
  for (const endpoint of endpoints) {
    const res = await axios({
      method: endpoint.method,
      url: `${baseUrl}${endpoint.path}`,
      data: endpoint.body,
    });
    responses.push({
      endpoint,
      status: res.status,
      body: res.data,
    });
  }

  return responses;
}

test('refactored API matches golden master', async () => {
  const goldenMaster = require('./api_golden_master.json');
  const refactoredResponses = await captureApiResponses('http://localhost:3001');

  expect(refactoredResponses).toEqual(goldenMaster);
});

Best Practices

• For a deeper guide, see Contract Testing: Verify Consumer-Producer API Contracts.

• Generate diverse inputs: normal, edge, boundary, and invalid cases. Use Hypothesis 6.100+ for property-based input generation when the input space is large.

• Normalize non-deterministic values (timestamps, UUIDs, random IDs) before comparison. A single unnormalized timestamp causes a false failure that erodes trust in the test suite.

• Use SHA-256 hashes for outputs larger than 1KB. Hash comparison is O(1) and keeps the golden master file small. Store full output only for the first 50 cases for debugging.

• Run golden master tests in CI with pytest 8.3 and --json-report to get machine-readable output for diff analysis when a regression fires.

• Keep the golden master under version control. Tag the commit that generated it so you can trace which behavior you characterized and when.

• Refactor in small steps (extract method, rename, replace conditional) and run the golden master after each change. If you batch 5 changes and the test fails, you won’t know which one broke it.

• Delete golden master tests once you’ve written proper unit tests that provide equal confidence. The golden master is a safety net, not a permanent testing strategy.

• Document what the golden master covers in a GOLDEN_MASTER.md file: which entry points, which inputs, which normalizations. Future developers need this context to trust the baseline.

Companion code: The golden-master-testing-pattern companion repo has a runnable Python example with 258 passing tests, a legacy calculator, a refactored version, and the golden master generation script.

Common Mistakes

• Insufficient test inputs: generating only happy-path inputs misses edge cases. Include boundary and invalid inputs.
• Not normalizing output: timestamps and random values cause false failures. Always normalize before comparison.
• Updating golden master blindly: when the golden master changes, review the diff carefully. A change might be a regression, not an improvement.
• Keeping golden master forever: golden master tests are a safety net for refactoring. Once you’ve written proper unit tests, delete them.
• Testing too much output: comparing entire HTML pages or large JSON makes diffs unreadable. Focus on the relevant parts.
• Skipping the documentation: if you don’t write down which entry points, inputs, and normalizations the golden master covers, the next developer won’t trust the baseline.