AI Prompt Version Control Template

Overview

This template gives you a structure for versioning LLM prompts. Each prompt version has metadata, a changelog, evaluation scores, and a rollback path. Use this to manage prompt changes with the same rigor as code changes. The approach mirrors OpenAI’s prompt engineering guide and integrates with evaluation platforms like LangSmith.

---

When to Use This Template

Use this template when your team manages prompts that affect production behavior. You don’t need it for one-off scripts or exploratory work, but you do need it when:

• A prompt drives a user-facing feature (support classification, content moderation, extraction).
• Two or more people edit the same prompt over time.
• You need to compare prompt versions with metrics, not gut feeling.
• A regression in prompt behavior would hurt users or cost money.
• You want to ship prompt changes through the same review flow as code.

If your prompt is a single system message that never changes, skip this template. If you ship prompt updates weekly or daily, this template pays for itself within the first rollback.

---

1. Prompt Metadata Schema

# prompt_metadata.yaml
prompt_id: "customer-support-classifier"
name: "Customer Support Ticket Classifier"
description: "Classifies incoming support tickets into categories"
current_version: "3.2.0"
status: "production"
owner: "ai-team@company.com"

versions:
  - version: "3.2.0"
    date: "2026-07-04"
    author: "jane@company.com"
    status: "production"
    model: "gpt-4o-mini"
    temperature: 0
    change_type: "patch"
    changes:
      - "Reduced system prompt from 450 to 280 tokens"
      - "Removed redundant category descriptions"
    eval_scores:
      accuracy: 0.92
      precision: 0.91
      recall: 0.89
      f1: 0.90
      latency_p95: 280
      cost_per_1k: 0.53
    rollback_to: "3.1.0"
    
  - version: "3.1.0"
    date: "2026-06-20"
    author: "jane@company.com"
    status: "archived"
    model: "gpt-4o-mini"
    temperature: 0
    change_type: "minor"
    changes:
      - "Added 'BILLING' category"
      - "Updated category descriptions for clarity"
    eval_scores:
      accuracy: 0.90
      precision: 0.89
      recall: 0.88
      f1: 0.88
      latency_p95: 320
      cost_per_1k: 0.55
    rollback_to: "3.0.0"
    
  - version: "3.0.0"
    date: "2026-06-01"
    author: "john@company.com"
    status: "archived"
    model: "gpt-4o-mini"
    temperature: 0
    change_type: "major"
    changes:
      - "Migrated from GPT-4o to GPT-4o-mini (15x cost reduction)"
      - "Rewrote system prompt for mini model"
      - "Added UNCLEAR category for ambiguous tickets"
    eval_scores:
      accuracy: 0.88
      precision: 0.87
      recall: 0.86
      f1: 0.86
      latency_p95: 300
      cost_per_1k: 0.53
    rollback_to: "2.1.0"

---

2. Prompt File Structure

prompts/
├── customer-support-classifier/
│   ├── prompt.md                 # Current production prompt
│   ├── metadata.yaml             # Version metadata and eval scores
│   ├── versions/
│   │   ├── 3.2.0.md
│   │   ├── 3.1.0.md
│   │   ├── 3.0.0.md
│   │   └── 2.1.0.md
│   ├── eval/
│   │   ├── test_set.jsonl        # 200 labeled test cases
│   │   ├── eval_results_3.2.0.json
│   │   ├── eval_results_3.1.0.json
│   │   └── eval_results_3.0.0.json
│   └── ab_tests/
│       ├── 3.1.0_vs_3.2.0.json
│       └── 3.0.0_vs_3.1.0.json

---

3. Version Numbering

MAJOR.MINOR.PATCH

MAJOR: Breaking changes
  - Model change (e.g., GPT-4o → Claude 3.5)
  - Output schema change
  - Category set change (added/removed categories)
  - Requires re-evaluation and stakeholder approval

MINOR: Feature additions
  - New category added
  - New few-shot example added
  - System prompt restructured
  - Requires evaluation but not stakeholder approval

PATCH: Optimizations
  - Token reduction
  - Wording tweaks
  - Formatting changes
  - Requires evaluation, fast approval

---

4. Changelog Format

## [3.2.0] — 2026-07-04

### Changed
- Reduced system prompt from 450 to 280 tokens (37% reduction)
- Removed redundant category descriptions for COMPLAINT and FEEDBACK
- Consolidated overlapping instructions into a single rules section

### Impact
- Cost per 1000 queries: $0.55 → $0.53 (4% savings)
- Latency p95: 320ms → 280ms (12% faster)
- Accuracy: 0.90 → 0.92 (+2%)
- F1: 0.88 → 0.90 (+2%)

### Rollback
- Revert to 3.1.0: `cp versions/3.1.0.md prompt.md`
- No data migration needed

---

5. Evaluation Protocol

Pre-Deployment Evaluation

The evaluation protocol follows the OpenAI Evals framework pattern: run each prompt version against a fixed test set and record accuracy, precision, recall, and F1.

import json
from openai import OpenAI

client = OpenAI()

def evaluate_prompt_version(prompt_path: str, test_set_path: str, model: str):
    with open(prompt_path) as f:
        prompt_template = f.read()
    
    with open(test_set_path) as f:
        test_cases = [json.loads(line) for line in f]
    
    correct = 0
    total = len(test_cases)
    
    for case in test_cases:
        prompt = prompt_template.replace("{{input_text}}", case["input"])
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
        )
        prediction = response.choices[0].message.content.strip()
        
        if prediction == case["expected"]:
            correct += 1
    
    accuracy = correct / total
    
    return {
        "version": prompt_path.split("/")[-1].replace(".md", ""),
        "accuracy": accuracy,
        "test_cases": total,
        "correct": correct,
        "incorrect": total - correct,
    }

# Run evaluation on new version
results = evaluate_prompt_version(
    "prompts/classifier/versions/3.2.0.md",
    "prompts/classifier/eval/test_set.jsonl",
    "gpt-4o-mini",
)

print(json.dumps(results, indent=2))
# {
#   "version": "3.2.0",
#   "accuracy": 0.92,
#   "test_cases": 200,
#   "correct": 184,
#   "incorrect": 16
# }

Regression Check

def compare_versions(old_results: dict, new_results: dict) -> dict:
    accuracy_delta = new_results["accuracy"] - old_results["accuracy"]
    
    verdict = "PASS"
    if accuracy_delta < -0.02:
        verdict = "FAIL — accuracy dropped more than 2%"
    elif accuracy_delta < 0:
        verdict = "WARN — accuracy decreased slightly"
    
    return {
        "old_version": old_results["version"],
        "new_version": new_results["version"],
        "old_accuracy": old_results["accuracy"],
        "new_accuracy": new_results["accuracy"],
        "delta": accuracy_delta,
        "verdict": verdict,
    }

---

6. A/B Testing

For statistical significance, use a chi-square test or Welch’s t-test before declaring a winner. The class below routes traffic and records results.

import random
import json
from collections import defaultdict

class PromptABTest:
    def __init__(self, prompt_a: str, prompt_b: str, traffic_split: float = 0.5):
        self.prompt_a = prompt_a
        self.prompt_b = prompt_b
        self.traffic_split = traffic_split
        self.results = defaultdict(lambda: {"correct": 0, "total": 0})
    
    def route(self) -> str:
        if random.random() < self.traffic_split:
            return self.prompt_a, "A"
        return self.prompt_b, "B"
    
    def record(self, variant: str, correct: bool):
        self.results[variant]["total"] += 1
        if correct:
            self.results[variant]["correct"] += 1
    
    def report(self) -> dict:
        a = self.results["A"]
        b = self.results["B"]
        
        return {
            "variant_a": {
                "accuracy": a["correct"] / a["total"] if a["total"] > 0 else 0,
                "samples": a["total"],
            },
            "variant_b": {
                "accuracy": b["correct"] / b["total"] if b["total"] > 0 else 0,
                "samples": b["total"],
            },
            "winner": "A" if a["correct"] / max(a["total"], 1) > b["correct"] / max(b["total"], 1) else "B",
        }

# Usage
ab_test = PromptABTest(
    prompt_a="prompts/classifier/versions/3.1.0.md",
    prompt_b="prompts/classifier/versions/3.2.0.md",
)

# After 1000 samples
report = ab_test.report()
print(json.dumps(report, indent=2))

---

7. CI/CD Integration

# .github/workflows/prompt-eval.yml
name: Prompt Evaluation

on:
  pull_request:
    paths:
      - "prompts/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install dependencies
        run: pip install openai
      
      - name: Run prompt evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python scripts/eval_prompt.py \
            --prompt prompts/classifier/prompt.md \
            --test-set prompts/classifier/eval/test_set.jsonl \
            --model gpt-4o-mini \
            --threshold 0.88
      
      - name: Check regression
        run: |
          python scripts/check_regression.py \
            --old-results prompts/classifier/eval/eval_results_3.1.0.json \
            --new-results eval_output.json \
            --max-drop 0.02

---

8. Approval Process

1. Developer creates a new prompt version in prompts/[name]/versions/
2. Developer runs evaluation: python scripts/eval_prompt.py
3. Developer creates PR with:
   - New prompt version file
   - Updated metadata.yaml
   - Evaluation results
4. CI runs automated evaluation and regression check
5. Reviewer checks:
   - Eval scores meet threshold
   - No regression beyond max-drop
   - Changelog is clear and accurate
6. Reviewer approves PR
7. PR merged → new version becomes production
8. Old version archived in versions/ directory
9. If new version causes issues in production → rollback to previous version

Common Mistakes

Skipping the test set

Teams often ship a prompt change without a labeled test set, then discover a regression days later. Build the test set before the first version, not after the first incident. Even 50 labeled cases catch most regressions; 200 gives you confidence for 2-point accuracy deltas.

Using accuracy as the only metric

Accuracy hides class imbalance. A classifier that predicts “OTHER” for 90% of inputs can hit 90% accuracy while being useless. Track precision, recall, and F1 per class, plus latency and cost. A prompt that’s 2% more accurate but 3x slower is usually a net loss.

Treating prompts as code without the review step

Versioning prompts in Git is a good start, but merging without review defeats the purpose. The PR review is where you catch missing edge cases, unclear changelogs, and eval results that don’t match the claim. Make the review mandatory, even for patch changes.

Rolling back without documenting why

When you roll back, the changelog should explain what broke. A rollback reason like “accuracy dropped” is useless six months later. Write the specific failure: “3.2.0 misclassified BILLING tickets as COMPLAINT when the ticket mentioned a refund — edge case not in test set.”

A/B testing without statistical significance

Declaring a winner after 100 samples is noise, not signal. Use the sample sizes from the FAQ (3000 per variant for a 2% delta at 95% confidence). If you can’t reach that sample size, run the test longer or accept that the result is inconclusive.

See Also

• Complete Guide to LLM Prompt Engineering
• AI LLM Prompt Template Library
• Complete Guide to LLM Application Architecture
• Complete Guide to LLM Evaluation
• AI Agent Design Document Template