AI LLM Incident Response Runbook

Overview

This runbook covers the five LLM production incidents that wake on-call teams up most often: model API outages, hallucination events, cost spikes, safety failures, and quality degradation. Each section gives you the symptoms to confirm and the recovery actions in priority order.

Unlike a classic infrastructure runbook, LLM incidents rarely mean “the server is down.” The failure modes are fuzzier: the model answers but lies, the bill triples overnight, a prompt injection slips past your guardrails. That’s why each incident type here pairs technical diagnostics with product-level decisions like disabling a feature or rolling back a prompt version.

Copy it, replace the placeholders with your own contacts and thresholds, and keep it somewhere your on-call rotation can actually find it at 3 a.m.

Adopting This Runbook

Before this runbook can do its job during an incident, adapt it to your setup:

• Fill in the contacts table: replace the placeholder emails and channels with your real on-call rotation, and test that each one actually reaches a human.
• Tune the severity thresholds: the defaults (20% error-rate increase, 3x/10x cost multipliers, 15-minute escalation) assume a mid-traffic product. A hobby project can relax them; a paid API should tighten them.
• List your AI functionality by criticality: the recovery steps assume you know which parts are safe to disable. Write it down now, not mid-outage.
• Pick your fallback model: section 3 assumes a second provider exists. If you only use one, decide whether cached or static responses are your degraded mode.
• Wire the diagnostics to real dashboards: every “check monitoring” step should link to a concrete dashboard URL in your copy.

The severity model borrows from classic incident management: see Google’s SRE book chapter on managing incidents and PagerDuty’s public incident response guide for the framework this adapts to LLM-specific failures.

Companion files: the ai-llm-incident-response-runbook companion repo has this runbook as a standalone file plus a check-llm-status.sh script that polls provider status pages and tests API connectivity.

---

1. Severity Levels

Severity is what decides how fast you move and who gets woken up. Classify by blast radius, not by how scary the symptom looks: a hallucination in an internal tool is SEV-3, while the same one in medical advice is SEV-1.

SEV-1 (Critical):
  - LLM service completely down
  - Safety failure causing harm to users
  - Cost spike > 10x normal daily spend
  - Data leak via prompt injection
  Response: Immediate, all hands, page on-call + management

SEV-2 (High):
  - Significant quality degradation (> 20% error rate increase)
  - Model latency > 5x normal
  - Single feature broken (e.g., RAG not returning results)
  - Cost spike > 3x normal daily spend
  Response: Page on-call within 15 minutes

SEV-3 (Medium):
  - Minor quality degradation (< 20% error rate increase)
  - Intermittent errors (< 5% of requests)
  - Non-critical feature degraded
  Response: Notify on-call during business hours

SEV-4 (Low):
  - Cosmetic issues in output formatting
  - Occasional hallucination on edge cases
  - Monitoring alert without user impact
  Response: Create ticket, address in next sprint

When you adopt this, give two thresholds a second look: the cost multipliers depend on your budget headroom, and the “20% error rate” line should be set against your own baseline variance, not copied blindly.

---

2. Escalation Path

Escalation exists so a stuck responder never burns the whole incident alone. Each rung gets a time limit: no progress means escalate, even when you’re “close to figuring it out.”

On-call engineer
  ↓ (cannot resolve in 15 min for SEV-1/2)
AI team lead
  ↓ (cannot resolve in 30 min)
Engineering manager
  ↓ (SEV-1 only)
CTO / VP Engineering
  ↓ (if user-facing harm or data leak)
Legal / PR / Security team

Contacts

Role                | Primary              | Secondary
────────────────────┼──────────────────────┼──────────────────
On-call engineer    | #ai-on-call          | PagerDuty: ai-team
AI team lead        | [lead-email]         | [lead-backup]
Eng manager         | [manager-email]      | #engineering
Security team       | [security-email]     | #security-urgent

---

3. Incident: Model API Outage

The provider is down or unreachable. Of the five, this is the only incident where the fix is mostly waiting; your job is keeping the product alive while they recover.

Symptoms

• All LLM API calls returning 5xx errors
• Timeout errors on every request
• Provider status page shows outage

Diagnostic Steps

# 1. Check provider status
curl -s https://status.openai.com/api/v2/status.json | jq

# 2. Test API connectivity
curl -s https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  | jq '.data[0].id'

# 3. Check error rate in monitoring
# Look at error rate dashboard for last 30 minutes

# 4. Test fallback model
curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-3-haiku","max_tokens":10,"messages":[{"role":"user","content":"test"}]}'

Recovery Actions

1. Enable fallback model:
   - Set feature flag: FALLBACK_MODEL=claude-3-5-sonnet
   - Route all traffic to fallback provider
   - Notify users of potential quality differences

2. If fallback also down:
   - Enable cached responses mode (return last known good response)
   - Switch to static fallback responses for critical paths
   - Disable non-critical AI features

3. Monitor provider status:
   - Check status page every 5 minutes
   - Subscribe to provider status updates

4. When provider recovers:
   - Gradually shift traffic back (10% → 50% → 100%)
   - Monitor error rates and latency
   - Disable fallback after 30 minutes of stable operation

The traffic-shift percentages in step 4 matter more than they look: snapping back to 100% the moment the status page goes green is how you catch a provider’s shaky recovery and bounce the incident twice.

---

4. Incident: Hallucination Event

Users are getting wrong answers delivered with total confidence. Unlike an outage it fails silently; you’ll usually hear about it from support tickets or social media before any dashboard lights up.

Symptoms

• Users report incorrect information in responses
• Monitoring shows faithfulness score drop
• Social media or support tickets flag wrong answers

Diagnostic Steps

# 1. Identify affected queries
import json

def find_hallucinations(logs: list, threshold: float = 0.7):
    suspicious = []
    for log in logs:
        if log.get("faithfulness_score", 1.0) < threshold:
            suspicious.append({
                "query": log["query"],
                "answer": log["answer"],
                "score": log["faithfulness_score"],
                "timestamp": log["timestamp"],
            })
    return suspicious

# 2. Check if retrieval is returning wrong context
def check_retrieval(query: str, expected_source: str):
    results = vector_store.search(query, top_k=5)
    for r in results:
        if expected_source in r["metadata"]["source"]:
            return True
    return False

# 3. Check if prompt was recently changed
# Review prompt version history in metadata.yaml

Recovery Actions

1. Immediate:
   - Roll back to previous prompt version: cp versions/3.1.0.md prompt.md
   - If RAG issue: check if corpus was recently updated with incorrect data
   - Disable affected feature if hallucinations are severe

2. Investigation:
   - Analyze 20-50 affected queries
   - Determine root cause: prompt change, corpus change, model update, or edge case
   - Document findings in incident report

3. Prevention:
   - Add affected queries to test set
   - Strengthen system prompt grounding instructions
   - Add faithfulness check to CI/CD pipeline
   - Set up real-time hallucination monitoring

Root cause is usually one of four things: a prompt edit, a corpus update, a silent provider-side model change, or a new class of user queries. Check them in that order: prompt and corpus changes are yours to see, model changes require diffing outputs against your golden set.

---

5. Incident: Cost Spike

The bill jumped. Cost spikes are sneaky because nothing is “broken”: the system works exactly as designed, just at 10x the price.

Symptoms

• Daily LLM cost > 3x normal (SEV-2) or > 10x normal (SEV-1)
• Cost monitoring alert triggered
• Budget threshold notification

Diagnostic Steps

# 1. Identify cost source
def analyze_cost_spike(daily_logs: list):
    by_feature = defaultdict(float)
    by_user = defaultdict(float)
    by_model = defaultdict(float)
    
    for log in daily_logs:
        by_feature[log["feature"]] += log["cost"]
        by_user[log["user_id"]] += log["cost"]
        by_model[log["model"]] += log["cost"]
    
    return {
        "by_feature": sorted(by_feature.items(), key=lambda x: -x[1]),
        "by_user": sorted(by_user.items(), key=lambda x: -x[1]),
        "by_model": sorted(by_model.items(), key=lambda x: -x[1]),
    }

# 2. Check for anomalies
# - Single user sending abnormally long prompts?
# - Bug causing repeated retries?
# - Model changed without cost calculation update?
# - RAG retrieving too many chunks?

Recovery Actions

1. Immediate (SEV-1):
   - Enable cost throttle: set daily_budget to normal * 1.5
   - Disable non-critical features (image description, sentiment analysis)
   - Rate limit top spending users

2. Immediate (SEV-2):
   - Identify the cost driver (feature, user, or model)
   - If single user: apply rate limit
   - If bug: deploy fix or disable affected feature
   - If model change: revert to cheaper model

3. Investigation:
   - Review cost logs for the past 24 hours
   - Check for new traffic sources (bot, integration, viral event)
   - Verify pricing calculations match provider pricing
   - Review prompt token counts for unexpected increases

The usual suspects, in order of how often they’re the culprit: a retry loop gone wild, one user or integration hammering an endpoint, a feature shipping much longer prompts than budgeted, and a provider changing model pricing or silently routing to a pricier tier (the expensive one).

---

6. Incident: Safety Failure

A guardrail failed: a restricted action executed, harmful output got through, an injection bypassed your filters, or PII ended up in a response. Treat every safety failure as SEV-1 until proven otherwise; the downside of underreacting is far worse than the downside of disabling a feature for an hour.

Symptoms

• Agent performs restricted action
• Output contains harmful, biased, or inappropriate content
• Prompt injection bypasses guardrails
• PII leaked in responses

Diagnostic Steps

1. Capture the full conversation log
2. Identify the input that triggered the failure
3. Check if safety guardrails were active
4. Check if the input bypassed input validation
5. Determine if this is a novel attack or known pattern
6. Check if a recent prompt or model change weakened guardrails

Recovery Actions

1. Immediate:
   - Disable the affected agent/feature
   - Block the user if malicious intent
   - Notify security team and management
   - Preserve all logs for investigation

2. If PII leak:
   - Identify what PII was exposed and to whom
   - Notify affected users (legally required in many jurisdictions)
   - Notify legal team and DPO

3. If harmful output:
   - Add the input pattern to blocklist
   - Strengthen system prompt safety instructions
   - Add additional output validation layer
   - Re-test with adversarial inputs

4. Post-incident:
   - Conduct full security review
   - Update guardrails and test suite
   - Document in security incident log
   - Review with legal/compliance team

Step 1’s “preserve all logs” is easy to skip under pressure and painful to skip legally: if a PII leak turns into a disclosure obligation, those logs are your evidence of scope.

---

7. Incident: Quality Degradation

Nothing’s down and nothing’s expensive; the answers are just quietly worse. Quality drift is the hardest incident to detect and the easiest to ignore until churn shows up.

Symptoms

• User satisfaction scores dropping
• Error rate increasing
• Support tickets about wrong or unhelpful responses
• Monitoring shows accuracy drop

Diagnostic Steps

1. Check for recent changes:
   - Prompt version changes (check metadata.yaml)
   - Model version updates (provider may have updated model)
   - Corpus changes (new documents, updated content)
   - Configuration changes (temperature, max_tokens, top_k)

2. Run golden test set:
   python scripts/eval_prompt.py --prompt current --test-set golden.jsonl
   Compare results to baseline

3. Sample 50 recent responses:
   - Score each on accuracy, relevance, faithfulness
   - Identify patterns in low-scoring responses

Recovery Actions

1. If prompt change caused degradation:
   - Roll back to previous prompt version
   - Re-run evaluation to confirm recovery

2. If model update caused degradation:
   - Switch to fallback model
   - Report issue to model provider
   - Adjust prompt for new model behavior

3. If corpus change caused degradation:
   - Identify which documents were added/modified
   - Check if new documents contain incorrect or conflicting information
   - Remove or fix problematic documents
   - Rebuild vector index

4. If no recent changes:
   - Check for distribution shift in user queries
   - Monitor for adversarial inputs
   - Consider seasonal or event-driven query pattern changes

If your golden test set can’t detect the degradation users are reporting, the test set is the thing to fix, since it no longer reflects what users actually ask; that’s worth a separate action item.

---

8. Post-Incident Checklist

[ ] Incident documented in incident tracker
[ ] Timeline of events recorded
[ ] Root cause identified
[ ] Fix deployed and verified
[ ] Test set updated with regression cases
[ ] Monitoring thresholds adjusted if needed
[ ] Postmortem scheduled within 48 hours
[ ] Action items assigned with deadlines
[ ] Stakeholders notified of resolution
[ ] Runbook updated with new learnings

That last item is what makes this document better over time: every incident missing from this runbook is a gap you just discovered for free.

See Also

• Complete Guide to LLM Prompt Engineering
• Complete Guide to LangChain in Production
• Complete Guide to LLM Application Architecture
• Complete Guide to LLM Cost Optimization
• Complete Guide to LLM Evaluation