Build a RAG Pipeline with LangChain and Vector Databases

Overview

Retrieval-Augmented Generation (RAG) pairs an LLM with a retrieval step: instead of answering only from what the model memorized during training, the pipeline pulls relevant passages out of your own documents and injects them into the prompt. That grounding cuts hallucinations sharply and lets the model answer questions about private or recently updated data.

This recipe builds the full pipeline: load and chunk documents, generate embeddings, store them in a vector database, retrieve the most relevant chunks at query time, and generate an answer with LangChain. The examples use current LangChain packages (langchain-openai, langchain-chroma, langchain-community) — the legacy from langchain.embeddings import … imports you still see in older tutorials no longer work on a fresh install.

When to Use

Use this resource when:

• You want an LLM to answer questions based on your own documents, not just its training data
• You need up-to-date answers the base model hasn’t seen
• You want to reduce hallucinations in domain-specific applications
• You’re building a knowledge base, help desk bot, or document Q&A system

Don’t reach for RAG when the whole corpus fits in the model’s context window: pasting 30 documents into the prompt is simpler and cheaper than running a retrieval stack.

Solution

Python

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

# 1. Load and chunk documents
loader = TextLoader("docs/knowledge_base.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

# 2. Embed chunks and store them in a vector DB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    chunks, embeddings, persist_directory="./chroma_db"
)

# 3. Wire the retriever + LLM into a retrieval chain
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = ChatPromptTemplate.from_template(
    "Answer using only this context. If the answer isn't in the "
    "context, say you don't know.\n\nContext:\n{context}\n\n"
    "Question: {input}"
)
qa_chain = create_retrieval_chain(
    vectorstore.as_retriever(search_kwargs={"k": 4}),
    create_stuff_documents_chain(llm, prompt),
)

# 4. Query
result = qa_chain.invoke({"input": "What is the refund policy?"})
print(result["answer"])
for doc in result["context"]:
    print(f"Source: {doc.metadata.get('source')}")

JavaScript

import { TextLoader } from 'langchain/document_loaders/fs/text';
import { RecursiveCharacterTextSplitter } from '@langchain/textsplitters';
import { OpenAIEmbeddings, ChatOpenAI } from '@langchain/openai';
import { Chroma } from '@langchain/community/vectorstores/chroma';
import { createRetrievalChain } from 'langchain/chains/retrieval';
import { createStuffDocumentsChain } from 'langchain/chains/combine_documents';
import { ChatPromptTemplate } from '@langchain/core/prompts';

async function buildRag() {
  // 1. Load and chunk
  const loader = new TextLoader('docs/knowledge_base.txt');
  const docs = await loader.load();
  const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 500, chunkOverlap: 50 });
  const chunks = await splitter.splitDocuments(docs);

  // 2. Embed and store
  const embeddings = new OpenAIEmbeddings({ model: 'text-embedding-3-small' });
  const vectorstore = await Chroma.fromDocuments(chunks, embeddings, {
    collectionName: 'knowledge_base',
  });

  // 3. Wire retriever + LLM
  const llm = new ChatOpenAI({ model: 'gpt-4o-mini', temperature: 0 });
  const prompt = ChatPromptTemplate.fromTemplate(
    'Answer using only this context.\n\nContext:\n{context}\n\nQuestion: {input}'
  );
  const chain = await createRetrievalChain({
    retriever: vectorstore.asRetriever({ k: 4 }),
    combineDocsChain: await createStuffDocumentsChain({ llm, prompt }),
  });

  // 4. Query
  const result = await chain.invoke({ input: 'What is the refund policy?' });
  console.log(result.answer);
}

buildRag();

Java

// Java RAG with Spring AI 1.x
import org.springframework.ai.document.Document;
import org.springframework.ai.reader.TextReader;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.chat.messages.SystemMessage;
import org.springframework.ai.chat.messages.UserMessage;

// 1. Load and split documents
var reader = new TextReader("docs/knowledge_base.txt");
List<Document> documents = reader.get();
var splitter = TokenTextSplitter.builder().withChunkSize(500).build();
List<Document> chunks = splitter.split(documents);

// 2. Store embeddings (vectorStore is auto-configured: Chroma, PGvector, ...)
vectorStore.add(chunks);

// 3. Retrieve relevant chunks
List<Document> relevant = vectorStore.similaritySearch(
    SearchRequest.builder().query("What is the refund policy?").topK(4).build()
);

// 4. Build the augmented prompt
String context = relevant.stream()
    .map(Document::getText)
    .collect(Collectors.joining("\n---\n"));
Prompt prompt = new Prompt(List.of(
    new SystemMessage("Answer using only the provided context.\nContext:\n" + context),
    new UserMessage("What is the refund policy?")
));
String answer = chatModel.call(prompt).getResult().getOutput().getText();

How the Pipeline Works

Four stages, and each one can fail independently:

1. Ingestion: documents are loaded, cleaned, and split into chunks. For document handling, see file upload validation. Chunk size (typically 200–1000 tokens) trades granularity against context preservation.
2. Embedding: each chunk becomes a high-dimensional vector. Vectors capture semantic meaning, not keyword overlap.
3. Retrieval: the user’s question gets embedded too, and the vector store returns the k most similar chunks by cosine similarity or Euclidean distance. When nothing clears your relevance threshold, the pipeline should fall back instead of guessing.
4. Generation: the retrieved chunks go into the prompt as context, and the model answers grounded in that text.

Trade-offs:

• Chunk size: smaller chunks give precise retrieval but thin context; larger chunks dilute relevance.
• k value: too low misses context; too high wastes tokens and can confuse the model.
• Embedding model quality drives retrieval accuracy more than any other single choice.

Variants

Technology	Approach	Notes
LangChain	Orchestration + chaining	Mature ecosystem, handles prompt templates and output parsing
LlamaIndex	Data-centric framework	Built-in connectors for PDFs, SQL, APIs; better for complex data
Haystack	Modular pipeline	Strong for evaluation and production monitoring
Vector DBs	Chroma / Pinecone / Weaviate / pgvector	Chroma for local, Pinecone for managed, pgvector if you already use Postgres
Embeddings	OpenAI / Cohere / local (BGE, E5)	OpenAI for quality; local models for privacy and cost

Best Practices

• Add metadata (source filename, page number) to every chunk so answers can cite their origin.
• Use chunk_overlap (10–20% of chunk size) so splits don’t land mid-sentence.
• Filter retrieved chunks with a relevance threshold; injecting unrelated text hurts more than it helps.
• Re-rank with a cross-encoder (bge-reranker, Cohere Rerank) after the first vector pass. It scores query-chunk pairs directly and lifts precision.
• Use parent-child chunking on dense documents: index small chunks (~200 tokens) for retrieval, but hand the LLM the parent chunk (~1000 tokens) for context.
• Set up incremental indexing: track document versions and only re-embed what changed instead of rebuilding the corpus.
• Log the retrieved chunks, their similarity scores, and user feedback per query. You can’t improve retrieval quality you don’t measure. Track recall@k and MRR over time and alert on drops.
• Add a guardrails layer before the LLM call: filter retrieved chunks for PII, toxic content, or off-topic material.

Common Mistakes

• Chunks too large: a 2000-token chunk dilutes relevance and wastes prompt context. Tune size per document type: code docs, contracts, and news articles chunk differently.
• No overlap: splitting mid-sentence drops context at chunk boundaries.
• Skipping deduplication: ingesting the same document twice skews retrieval and wastes storage. Hash content or key on document ID before inserting.
• Wrong distance metric: cosine and Euclidean produce different rankings. OpenAI embeddings expect cosine; check what your embedding model was trained with.
• Storing raw HTML/PDF: extract clean text first; markup noise pollutes embeddings.
• Tiny or empty chunks: chunks under ~10 tokens produce noisy embeddings. Filter them or merge them with neighbors during ingestion.
• Ignoring embedding versioning: switching embedding models invalidates every stored vector. Record the model version in config and re-index on change.
• No fallback behavior: when nothing meets the threshold, the LLM should say “I don’t know” rather than improvise from training data.
• Evaluating with synthetic queries only: real users phrase questions differently than your test set. Mine production logs for the evaluation corpus.
• Missing metadata: without source, page, or section metadata you can’t filter or cite results. Attach it at ingestion time.

Troubleshooting

• Answers inconsistent between runs: set temperature: 0, use a seed where supported, and version your prompt template.
• Prompt injection through retrieved content: retrieved chunks are untrusted input. Keep user text and system instructions clearly separated, and consider filtering chunks before generation.
• High token costs: cache embeddings, cap retrieved context, and route simple questions to a smaller model.
• Retrieval returns irrelevant chunks: tune chunk size, overlap, and metadata filters. Measure retrieval quality separately from generation: a good answer built on the wrong chunks is luck, not signal.
• Evaluation scores disagree with human judgment: define a rubric, use more than one judge (human or LLM), and track disagreement. Human review stays the ground truth.

Production Checklist

• Chunks deduplicated by content hash before insertion
• Distance metric matches the embedding model (cosine for OpenAI)
• Empty or sub-10-token chunks filtered during ingestion
• Embedding model version recorded in configuration
• Metadata attached to every chunk (source, page, section)
• Chunk size and overlap tuned per document type
• Fallback answer when no chunk meets the relevance threshold
• Re-ranking model applied after initial vector retrieval
• Incremental indexing pipeline for document updates
• Retrieval metrics (recall@k, MRR) tracked and alerted

Scaling Considerations

• Vector store choice: under ~100K documents, pgvector is enough and saves you a separate service. From 100K to ~1M, Pinecone or Weaviate managed tiers are the practical route. Past 1M, self-host Milvus or Qdrant with GPU-accelerated indexes.
• Embedding cost: indexing 100K documents with text-embedding-3-small costs around $1, but re-embedding the whole corpus on every change adds up, so incremental indexing matters at scale.
• Retrieval latency: vector search over 1M documents takes 10–50 ms on an HNSW index; a cross-encoder re-rank adds 50–100 ms. For sub-100 ms total, use ANN search and only re-rank the top-20.
• Context window: GPT-4o accepts 128K tokens, but pushing more than ~10K tokens of context raises cost and can degrade answers. Feed the top 3–5 chunks after re-ranking.
• Update frequency: for content that changes often (news, catalogs), trigger re-indexing from webhooks; for static content (docs, policies), a nightly batch job is enough.

Cost Estimation

Component	Cost	Notes
Embedding (text-embedding-3-small)	$0.02/1M tokens	OpenAI API
Embedding (self-hosted e5-large)	$0	GPU cost only
Vector store (Pinecone starter)	$0	100K vectors, 1 namespace
Vector store (Pinecone standard)	$70/month	5M vectors, unlimited namespaces
LLM generation (GPT-4o)	$5/1M input, $15/1M output	Per query: ~2K input + 500 output
LLM generation (GPT-4o-mini)	$0.15/1M input, $0.60/1M output	30x cheaper, good for simple Q&A

At 10K queries/day on GPT-4o you’re looking at roughly $150/day. Routing simple questions to GPT-4o-mini cuts that by about 90%.

When Not to Use RAG

• Small corpus (<50 documents): if everything fits in the context window, paste it; RAG adds infrastructure you don’t need.
• Verbatim answers required: RAG returns chunks, not full documents. For legal clauses or code samples where exact wording matters, use keyword search over full documents.
• Highly structured data: tables and forms embed poorly; a SQL query usually beats vector similarity there.
• Real-time requirements: indexes are eventually consistent. If the answer must reflect data from seconds ago, query the source directly.
• Claim-level citations required: RAG can show source chunks but can’t guarantee every sentence traces back to one. Legal and academic use cases need quote-constrained generation instead.

Performance Benchmarks

Component	Latency	Throughput	Notes
Embedding (text-embedding-3-small)	50-200ms	10K tokens/req	OpenAI API
Vector search (pgvector, 100K docs)	5-20ms	100 QPS	HNSW index
Vector search (Pinecone, 1M docs)	10-50ms	200 QPS	Managed service
Re-ranking (cross-encoder)	50-100ms	20 docs/batch	bge-reranker-large
LLM generation (GPT-4o)	1-5s	50 concurrent	Streaming enabled

End-to-end, expect 200 ms to 6 s per query, and the LLM generation step dominates. Pre-computed embeddings and query caching pull p99 down.

Further Reading

• LangChain retrieval tutorial: the canonical walkthrough for the chain setup used above.
• pgvector: vector similarity inside PostgreSQL, the lowest-friction starting point if you already run Postgres.
• Ragas: automated RAG evaluation; pairs well with the metrics described in the FAQ below. See also RAG evaluation with Ragas.
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — the paper that named the technique.