Vector Databases — AI/ML Embeddings and Similarity Search

Overview

Vector databases store high-dimensional numerical vectors (embeddings) generated by machine learning models and enable similarity search via Approximate Nearest Neighbor (ANN) algorithms. They power semantic search, recommendation systems, image retrieval, and Retrieval-Augmented Generation (RAG) for LLMs. Unlike traditional databases that search by exact match or range, vector databases find the “closest” vectors in embedding space — the mathematical representation of meaning, image features, or audio signatures.

When to Use

• For alternatives, see Complete Guide to Vector Databases.

• You need semantic search (find similar meaning, not just keyword match)

• LLM RAG pipelines require retrieving relevant context chunks

• Recommendation systems suggest items similar to user preferences

• Image, audio, or video retrieval by content similarity

• You have pre-trained embedding models and need growth-ready vector storage

How Vector Search Works

1. Embedding: A model (OpenAI, BERT, CLIP) converts text/image into a dense vector (e.g., 768-1536 dimensions)
2. Indexing: Vectors are organized into an ANN index (HNSW, IVF, PQ) for fast retrieval
3. Query: The query is embedded and the index returns the K nearest neighbors
4. Metadata filtering: Combine vector similarity with traditional filtering (date, category, user ID)

Comparison

Database	Deployment	Index	Best For
Pinecone	Managed cloud	HNSW, metadata filters	Production RAG, no ops overhead
Weaviate	Self-hosted / cloud	HNSW, BM25 hybrid	Multi-modal, GraphQL interface
pgvector	PostgreSQL extension	ivfflat, hnsw	Teams already on Postgres
Chroma	Embedded / local	HNSW	Prototyping, small-scale local RAG
Milvus/Zilliz	Self-hosted / cloud	IVF, HNSW, GPU	Large-scale, high throughput

pgvector Example

-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    content TEXT,
    embedding vector(1536)
);

-- Create HNSW index for fast ANN search
CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

-- Insert a document with embedding
INSERT INTO documents (title, content, embedding)
VALUES ('Vector DB Guide', 'A guide to vector databases...', '[0.12, -0.03, ...]');

-- Semantic search: find 5 most similar documents
SELECT id, title, content,
    1 - (embedding <=> '[0.11, -0.02, ...]') as similarity
FROM documents
ORDER BY embedding <=> '[0.11, -0.02, ...]'
LIMIT 5;

Hybrid Search (Vector + Keyword)

# Weaviate: combine vector similarity with BM25 keyword search
import weaviate

client = weaviate.connect_to_local()

results = client.collections.get("Article").query.hybrid(
    query="vector database architecture",
    vector=[0.12, -0.03, ...],
    alpha=0.5,  # 0 = pure BM25, 1 = pure vector
    limit=10
)

RAG Pipeline Example

from openai import OpenAI
import chromadb

# 1. Load and chunk documents
chunks = load_and_chunk_documents("knowledge_base/")

# 2. Embed and store in Chroma
client = chromadb.Client()
collection = client.create_collection("docs")
embeddings = openai_client.embeddings.create(input=chunks, model="text-embedding-3-small")
collection.add(ids=ids, documents=chunks, embeddings=[e.embedding for e in embeddings.data])

# 3. Retrieve relevant chunks for a query
query_embedding = openai_client.embeddings.create(input="How do vector indexes work?", model="text-embedding-3-small")
results = collection.query(query_embeddings=[query_embedding.data[0].embedding], n_results=5)

# 4. Augment LLM prompt with retrieved context
context = "\n".join([r["document"] for r in results["documents"][0]])
prompt = f"Context:\n{context}\n\nQuestion: How do vector indexes work?"
response = openai_client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": prompt}])

ANN Algorithms

Algorithm	Type	Speed	Memory	Best For
HNSW	Graph-based	Fast	High	General purpose, high recall
IVF	Clustering	Medium	Medium	Large datasets, memory-constrained
PQ	Quantization	Fast	Low	Billions of vectors, acceptable recall loss

Common Mistakes

• Wrong distance metric — cosine similarity for semantic text, Euclidean for image features, dot product for normalized embeddings
• No metadata filtering — pure vector search returns irrelevant results; always combine with metadata filters
• Ignoring index tuning — default HNSW parameters may not suit your recall/latency requirements
• Storing raw vectors without indexing — full brute-force scan is O(n) and unusable at scale
• Using a vector DB for structured queries — combine with a relational database; vector DBs are poor at aggregation and joins

Troubleshooting

• Query is slow after an index change: check execution plans and cardinality estimates. Rebuild statistics and verify the index is being used.
• Replication lag grows: monitor network, disk I/O, and long transactions. Split large writes and consider parallel replication.
• Connections exhausted: review connection pool size, idle timeouts, and leaked connections. Use prepared statements and close connections in finally blocks.
• Backup takes too long: enable compression, incremental backups, and off-peak scheduling. Test restore times against RTO targets.
• Deadlocks in high concurrency: access tables and rows in a consistent order. Keep transactions short and retry deadlocked operations.

FAQ

Do I need a dedicated vector database or can I use Postgres? For small-to-medium scale (< 1M vectors) and teams already on Postgres, pgvector is sufficient. For high-scale, multi-tenant, or managed needs, Pinecone or Weaviate are better.

How do I choose embedding dimensions? Use the output dimension of your chosen model (OpenAI text-embedding-3-small = 1536, BERT-base = 768). Do not arbitrarily reduce dimensions without quantization-aware training.

Can I update vectors in place? Yes, but re-indexing may be required depending on the database and index type. Some systems support incremental updates; others require full rebuilds.

How do I get started with this in an existing project?

Start with a small, isolated part of your codebase. Apply the concepts from this guide to one module or service. Measure the impact, then expand to other areas.

What tools do I need?

The tools mentioned throughout this guide are listed in each section. Most are open-source and widely adopted. Check the related resources for setup instructions.

How do I measure success after implementing this?

Define clear metrics before starting: performance benchmarks, error rates, or maintainability indicators. Compare before and after. Iterate based on the data, not on assumptions.

Advanced Topics

Detailed Scenario: RAG System for Technical Documentation

System: RAG chatbot over 50,000 pages of technical documentation
Stack: OpenAI text-embedding-3-small (1536 dim) + pgvector + GPT-4o
Requirements: Accurate answers with citations, latency < 3s

Pipeline:
  1. Load documents (PDF, Markdown, HTML)
  2. Chunk into ~500 token segments with 50 token overlap
  3. Generate embeddings with OpenAI
  4. Store in PostgreSQL with pgvector
  5. On query: embed question, search similar chunks, augment LLM

pgvector schema:
  CREATE EXTENSION IF NOT EXISTS vector;

  CREATE TABLE doc_chunks (
      id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
      doc_id VARCHAR(100) NOT NULL,
      chunk_index INT NOT NULL,
      content TEXT NOT NULL,
      embedding vector(1536) NOT NULL,
      metadata JSONB DEFAULT '{}',
      created_at TIMESTAMPTZ DEFAULT NOW()
  );

  CREATE INDEX idx_doc_chunks_embedding
  ON doc_chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

  CREATE INDEX idx_doc_chunks_doc ON doc_chunks(doc_id);

Ingestion (Python):
  from openai import OpenAI
  import psycopg2

  client = OpenAI()
  conn = psycopg2.connect("dbname=ragdb")

  def embed_text(text: str) -> list[float]:
      resp = client.embeddings.create(
          input=text, model="text-embedding-3-small"
      )
      return resp.data[0].embedding

  def ingest_document(doc_id: str, chunks: list[str]):
      for i, chunk in enumerate(chunks):
          emb = embed_text(chunk)
          with conn.cursor() as cur:
              cur.execute(
                  "INSERT INTO doc_chunks (doc_id, chunk_index, content, embedding) "
                  "VALUES (%s, %s, %s, %s)",
                  (doc_id, i, chunk, emb)
              )
      conn.commit()

RAG query:
  def rag_query(question: str, top_k: int = 5) -> str:
      q_emb = embed_text(question)
      with conn.cursor() as cur:
          cur.execute(
              "SELECT content, doc_id, chunk_index, "
              "1 - (embedding <=> %s) AS similarity "
              "FROM doc_chunks "
              "ORDER BY embedding <=> %s "
              "LIMIT %s",
              (q_emb, q_emb, top_k)
          )
          results = cur.fetchall()

      context = "\n\n".join([r[0] for r in results])
      sources = [f"doc:{r[1]} chunk:{r[2]} sim:{r[3]:.3f}" for r in results]

      prompt = (
          f"Context:\n{context}\n\n"
          f"Question: {question}\n"
          f"Answer based on the context. Cite the source."
      )
      response = client.chat.completions.create(
          model="gpt-4o",
          messages=[{"role": "user", "content": prompt}]
      )
      return response.choices[0].message.content + "\n\nSources: " + ", ".join(sources)

Results:
  | Metric | Value |
  |--------|-------|
  | Embedding time | 200ms |
  | Vector search time | 15ms |
  | LLM time | 1.5s |
  | Total latency | ~1.7s |
  | Recall@5 | 92% |
  | Answer accuracy | 87% (with correct citations) |

HNSW index tuning:
  - m=16: balance between recall and memory (default)
  - ef_construction=64: build quality (higher = better, slower)
  - ef_search=40: tune at query time for recall/latency trade-off
  - For 50K vectors: 50MB index, search < 20ms

Lessons learned:
  - pgvector is sufficient for < 1M vectors
  - Chunk size affects quality: 500 tokens with overlap works well
  - Filter by metadata before vector search to improve relevance
  - HNSW outperforms IVF in recall for small-to-medium datasets

How do I handle embedding updates when the model changes?

Version your embeddings: store the model and version in metadata. When changing models, re-embed all documents into a new column or table. Keep both versions during the transition. Update queries to use the new version. Drop the old one when there is no traffic. Schedule re-embedding in batches during off-peak hours to avoid impacting latency.

End of document. Review and update quarterly.

Common Production Pitfalls

• Treating the guide as a checklist to complete once rather than a practice to evolve.
• Adopting every recommendation at once instead of starting with one measured change.
• Skipping the maturity assessment and forcing advanced practices on an unprepared team.
• Not updating runbooks and on-call expectations as new practices are introduced.
• Ignoring real incident data when prioritizing which parts of the guide to apply first.
• Failing to assign an owner who reviews decisions quarterly.
• Copying examples without adapting them to the team’s actual tooling and constraints.
• Forgetting to measure outcomes before adding the next improvement.