AI Data Preparation Checklist

Overview

This checklist covers data preparation for LLM and RAG systems. Run through every section before indexing data into your vector store or fine-tuning pipeline. Poor data quality is the most common cause of RAG failures.

When to Use This Checklist

• You’re building a new RAG pipeline and need to validate data quality before indexing.
• You’re migrating a corpus from one vector store to another and want to re-validate chunking and embeddings.
• Retrieval quality dropped after a corpus refresh and you need to audit which step regressed.
• You’re onboarding a new data source and need to confirm it meets the same quality bar as existing sources.
• You’re preparing a fine-tuning dataset and need to deduplicate, redact PII, and normalize text before training.

---

1. Data Collection

1.1 Source Inventory

• All data sources identified and documented
• Source format documented (PDF, HTML, Markdown, JSON, CSV, database)
• Source access verified (API keys, database credentials, file permissions)
• Source refresh cadence defined (real-time, hourly, daily, weekly)
• Source ownership documented (who keeps each source up to date)

Source              | Format    | Size     | Refresh   | Owner
────────────────────┼───────────┼──────────┼───────────┼────────────
Product docs        | Markdown  | 50 MB    | Weekly    | Docs team
API reference       | OpenAPI   | 5 MB     | On change | Eng team
Support tickets     | JSON      | 200 MB   | Daily     | Support team
Internal wiki       | HTML      | 500 MB   | Daily     | Knowledge mgmt
Code repository     | Markdown  | 30 MB    | On commit | Eng team
Customer FAQs       | CSV       | 2 MB     | Monthly   | Support team

1.2 Legal and Compliance

• Data usage rights verified (can you use this data in an AI system?)
• License terms reviewed for each source
• Personal data identified and documented
• Consent status checked for user-generated content
• Data retention policy applied
• GDPR/CCPA compliance verified for personal data

---

2. Data Cleaning

2.1 Text Quality

import re
from typing import List

def clean_text(text: str) -> str:
    # Remove HTML tags
    text = re.sub(r'<[^>]+>', '', text)
    
    # Remove excessive whitespace
    text = re.sub(r'\s+', ' ', text)
    
    # Remove non-printable characters
    text = re.sub(r'[^\x20-\x7E\n\r\t]', '', text)
    
    # Fix encoding issues
    text = text.encode('utf-8', errors='ignore').decode('utf-8')
    
    # Strip leading/trailing whitespace
    text = text.strip()
    
    return text

def remove_boilerplate(text: str, min_length: int = 100) -> str:
    # Remove navigation menus, footers, headers
    lines = text.split('\n')
    meaningful = [l for l in lines if len(l.strip()) > min_length]
    return '\n'.join(meaningful)

2.2 Cleaning Checklist

• HTML tags removed from web-scraped content
• Markdown formatting normalized
• Encoding issues fixed (UTF-8 throughout)
• Excessive whitespace collapsed
• Boilerplate removed (navigation, footers, cookie notices)
• Empty or near-empty documents removed (< 50 characters)
• Duplicate documents identified and handled
• Corrupted files identified and removed
• Non-text content (images, videos) extracted or referenced separately

---

3. PII and Sensitive Data Removal

3.1 PII Detection

import re

PII_PATTERNS = {
    "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
    "phone": r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
    "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
    "credit_card": r'\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b',
    "ip_address": r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b',
}

def detect_pii(text: str) -> dict:
    found = {}
    for pii_type, pattern in PII_PATTERNS.items():
        matches = re.findall(pattern, text)
        if matches:
            found[pii_type] = len(matches)
    return found

def redact_pii(text: str) -> str:
    for pii_type, pattern in PII_PATTERNS.items():
        text = re.sub(pattern, f'[REDACTED_{pii_type.upper()}]', text)
    return text

3.2 PII Checklist

• Emails redacted or removed
• Phone numbers redacted
• Social security numbers redacted
• Credit card numbers redacted
• IP addresses redacted (if sensitive)
• Home addresses redacted
• Names of customers/users evaluated (may need to keep for context)
• API keys and secrets removed
• Internal URLs and hostnames evaluated

---

4. Deduplication

4.1 Exact Deduplication

import hashlib

def find_exact_duplicates(documents: list) -> dict:
    hashes = {}
    duplicates = {}
    
    for doc in documents:
    content_hash = hashlib.sha256(doc["content"].encode()).hexdigest()
        if content_hash in hashes:
            duplicates[doc["id"]] = hashes[content_hash]
        else:
            hashes[content_hash] = doc["id"]
    
    return duplicates

4.2 Near-Duplicate Detection

For near-duplicate detection, scikit-learn’s TfidfVectorizer combined with cosine similarity is a common starting point. For larger corpora, consider MinHash or LSH-based approaches that scale better than O(n²) pairwise comparison.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def find_near_duplicates(documents: list, threshold: float = 0.95) -> list:
    vectorizer = TfidfVectorizer()
    tfidf_matrix = vectorizer.fit_transform([d["content"] for d in documents])
    
    similarity_matrix = cosine_similarity(tfidf_matrix)
    np.fill_diagonal(similarity_matrix, 0)
    
    duplicates = []
    for i in range(len(documents)):
        for j in range(i + 1, len(documents)):
            if similarity_matrix[i][j] > threshold:
                duplicates.append({
                    "doc_a": documents[i]["id"],
                    "doc_b": documents[j]["id"],
                    "similarity": similarity_matrix[i][j],
                })
    
    return duplicates

4.3 Deduplication Checklist

• Exact duplicates removed
• Near-duplicates identified (cosine similarity > 0.95)
• Near-duplicates reviewed — keep the most complete version
• Cross-source duplicates identified (same content from different sources)
• Version conflicts resolved (keep latest version of updated documents)

---

5. Chunking Strategy

5.1 Chunk Size Selection

Content type          | Recommended chunk size | Overlap
──────────────────────┼───────────────────────┼─────────
Technical docs        | 512-1024 tokens        | 50-100
API reference         | 256-512 tokens         | 0-50
Support tickets       | 256-512 tokens         | 0
Code documentation    | 512-1024 tokens        | 50-100
Legal documents       | 1024-2048 tokens       | 100-200
FAQ entries           | One Q&A per chunk      | 0
Tables                | One table per chunk    | 0

5.2 Chunking Implementation

from langchain.text_splitter import RecursiveCharacterTextSplitter

def chunk_documents(documents: list, chunk_size: int = 512, overlap: int = 50) -> list:
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=overlap,
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    
    chunks = []
    for doc in documents:
        doc_chunks = splitter.split_text(doc["content"])
        for i, chunk in enumerate(doc_chunks):
            chunks.append({
                "id": f"{doc['id']}_chunk_{i}",
                "content": chunk,
                "metadata": {
                    "source": doc["source"],
                    "title": doc["title"],
                    "chunk_index": i,
                    "total_chunks": len(doc_chunks),
                },
            })
    
    return chunks

5.3 Chunking Checklist

• Chunk size appropriate for content type
• Chunk overlap preserves context across boundaries
• Document structure respected (no chunks crossing section boundaries)
• Tables kept intact (not split across chunks)
• Code blocks kept intact
• Metadata attached to each chunk (source, page, section, title)
• Chunk length distribution checked (no extreme outliers)

---

6. Embedding Generation

6.1 Model Selection

Choose an embedding model based on your quality, cost, and latency requirements. The OpenAI embeddings guide covers the trade-offs between text-embedding-3-small and text-embedding-3-large. For self-hosted options, the LangChain embeddings documentation lists local providers like Sentence-Transformers and HuggingFace models.

Model                      | Dimensions | Cost / 1K tokens | Quality
───────────────────────────┼────────────┼─────────────────┼────────
text-embedding-3-small     | 1536       | $0.00002         | Good
text-embedding-3-large     | 3072       | $0.00013         | Best
text-embedding-ada-002     | 1536       | $0.00010         | Good
Cohere embed-english-v3    | 1024       | $0.00010         | Good
Sentence-Transformers      | 384-768    | Free (self-host) | Fair

6.2 Embedding Generation

from openai import OpenAI
import time

client = OpenAI()

def generate_embeddings(chunks: list, model: str = "text-embedding-3-small", batch_size: int = 100) -> list:
    embeddings = []
    
    for i in range(0, len(chunks), batch_size):
        batch = chunks[i:i + batch_size]
        texts = [c["content"] for c in batch]
        
        response = client.embeddings.create(
            model=model,
            input=texts,
        )
        
        for j, embedding in enumerate(response.data):
            embeddings.append({
                "chunk_id": batch[j]["id"],
                "embedding": embedding.embedding,
                "content": batch[j]["content"],
                "metadata": batch[j]["metadata"],
            })
        
        # Rate limit
        time.sleep(0.5)
    
    return embeddings

6.3 Embedding Checklist

• Embedding model selected based on quality and cost requirements
• All chunks embedded with the same model
• Embedding dimensions consistent across all chunks
• Batch size optimized for throughput without hitting rate limits
• Failed embeddings retried
• Embedding cost tracked and within budget
• Embeddings stored in vector database with metadata

---

7. Vector Index Configuration

7.1 Index Selection

The Pinecone documentation and the pgvector guide cover index configuration in depth. The table below summarizes the main options.

Vector store    | Index type | Best for                    | Max vectors
────────────────┼────────────┼─────────────────────────────┼────────────
Pinecone        | HNSW       | Managed, production         | 1B+
Weaviate        | HNSW       | Self-hosted, flexible       | 1B+
pgvector        | IVFFlat    | PostgreSQL integration      | 100M
Qdrant          | HNSW       | Self-hosted, Rust-based     | 1B+
Milvus          | IVF/HNSW   | Large-scale, self-hosted    | 10B+
Chroma          | HNSW       | Local development           | 1M

7.2 Index Parameters

# Pinecone example
from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="your-api-key")
index = pc.create_index(
    name="knowledge-base",
    dimension=1536,
    metric="cosine",
    spec=ServerlessSpec(
        cloud="aws",
        region="us-east-1",
    ),
)

# HNSW parameters (for self-hosted)
HNSW_CONFIG = {
    "m": 16,              # Max connections per node
    "ef_construction": 200,  # Build-time search width
    "ef_search": 50,      # Query-time search width
}

7.3 Index Checklist

• Vector store selected based on scale and latency requirements
• Index type appropriate for corpus size
• Distance metric selected (cosine for normalized embeddings)
• Index parameters tuned for recall vs latency tradeoff
• Metadata filters configured (source, date, category)
• Index built and verified with test queries
• Query latency measured (p95 < 200ms target)

---

8. Quality Validation

8.1 Data Quality Metrics

Metric                  | Target  | Measurement
────────────────────────┼─────────┼──────────────────────
Average chunk length    | 400-800 | tokens per chunk
Chunk length variance   | Low     | No chunks < 50 or > 2000 tokens
Duplicate chunk rate    | < 2%    | % of near-duplicate chunks
PII detection rate      | 0%      | After redaction
Empty chunk rate        | 0%      | No empty chunks indexed
Metadata completeness   | 100%    | Every chunk has source + title
Embedding success rate  | > 99%   | % of chunks successfully embedded

8.2 Retrieval Quality Test

def test_retrieval_quality(test_queries: list, vector_store, expected_sources: dict):
    results = []
    for query in test_queries:
        retrieved = vector_store.search(query, top_k=5)
        retrieved_sources = [r["metadata"]["source"] for r in retrieved]
        expected = expected_sources.get(query, [])
        
        # Check if expected source is in top-k
        hit = any(src in retrieved_sources for src in expected)
        
        results.append({
            "query": query,
            "hit": hit,
            "retrieved_sources": retrieved_sources,
            "expected_sources": expected,
        })
    
    hit_rate = sum(1 for r in results if r["hit"]) / len(results)
    return {"hit_rate": hit_rate, "results": results}

# Run with 20+ test queries
# Target: hit_rate > 0.85

8.3 Quality Checklist

• Average chunk length within target range
• No empty or near-empty chunks
• No PII detected after redaction
• Duplicate rate below threshold
• All chunks have complete metadata
• Embedding success rate > 99%
• Retrieval hit rate > 0.85 on test queries
• Query latency p95 < 200ms

---

9. Sign-Off Checklist

• All data sources collected and documented
• Legal compliance verified
• Text cleaned and normalized
• PII redacted
• Duplicates removed
• Chunking strategy applied and validated
• Embeddings generated for all chunks
• Vector index built and configured
• Quality metrics meet targets
• Retrieval test queries pass
• Monitoring and alerting configured
• Data refresh pipeline documented

Common Mistakes

Skipping the PII audit on internal documents

Internal wikis and support tickets often contain customer emails, phone numbers, and API keys. Even if the data never leaves your company, indexing PII into a vector store creates a retrieval path that bypasses normal access controls. Run detect_pii on every source, even ones you trust, and log what was redacted.

Using one chunk size for every content type

A single 512-token chunk size works for generic text but breaks on FAQs (too large, mixes questions), legal documents (too small, splits clauses), and tables (splits rows from headers). Match the chunk size to the content type using the table in section 5.1, and measure retrieval quality per content type, not just overall.

Embedding before deduplication

If you embed near-duplicates, the vector store returns two or more copies of the same answer, wasting context window tokens and confusing the retrieval ranking. Always run exact and near-duplicate detection before chunking and embedding. The cosine similarity threshold of 0.95 catches most paraphrased duplicates; lower it to 0.90 if you see too many redundant results.

Treating the test set as the training set

If you tune chunk size, embedding model, or index parameters using the same queries you use to validate retrieval quality, you’re overfitting to those queries. Hold out a separate set of 20+ queries that you never use for tuning, and report hit rate on that set only. Re-run the held-out set after every parameter change.

Rebuilding the entire index on every update

For corpora that change daily (support tickets, wiki edits), full index rebuilds waste compute and create downtime windows. Most vector stores (Pinecone, Weaviate, pgvector, Qdrant) support incremental upserts. Use them for daily updates, and reserve full rebuilds for quarterly maintenance or when you change the embedding model.

See Also

• Complete Guide to LLM Prompt Engineering
• AI RAG Evaluation Checklist
• Complete Guide to RAG in Production
• Build a RAG Pipeline with LangChain and Vector Databases
• Implement Semantic Search with Embeddings