LangChain RAG: Document Loading, Chunking, Embeddings and Retrieval

RAG Pipeline Stages

RAG stands for retrieval augmented generation. Instead of asking the model to answer from memory, you retrieve relevant chunks from your own documents and pass them into the prompt. This improves freshness, domain accuracy, and source grounding.

Good RAG is not simply embeddings plus a vector database. It requires thoughtful document preparation, chunking, metadata, retrieval tuning, context packing, source display, and evaluation. Most weak RAG systems fail because retrieval quality is poor, not because the model is bad.

The pipeline begins before the user asks a question. You ingest documents, split them into chunks, embed the chunks, and store vectors with metadata. At query time, you embed the user question, retrieve similar chunks, optionally rerank them, and ask the model to answer using those chunks.

Load: read PDFs, HTML, Markdown, tickets, docs, or database rows.
Split: create chunks that preserve meaning and fit model context.
Embed: convert chunks into vectors.
Retrieve: find relevant chunks for each user question.
Generate: answer with citations and refusal rules.

Chunking Strategy

Chunk size should reflect document structure. API references, policies, and tutorials need different chunking. Preserve headings and metadata because they help both retrieval and citation display.

Use overlap when ideas continue across paragraph boundaries.
Store source path, title, section, date, and permissions as metadata.
Test retrieval with realistic questions before judging answer quality.

Small RAG Chain with Citations

This example shows the shape of a RAG chain. Swap the vector store for your production choice when needed.

Small RAG Chain with Citations

from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

docs = [
    Document(
        page_content="Refunds are available within 14 days for annual plans.",
        metadata={"source": "billing-policy.md", "section": "refunds"},
    ),
    Document(
        page_content="Enterprise customers can request SSO using SAML or OIDC.",
        metadata={"source": "security.md", "section": "sso"},
    ),
]

vectorstore = FAISS.from_documents(docs, OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

def format_docs(items):
    return "\n\n".join(
        f"Source: {doc.metadata['source']}#{doc.metadata['section']}\n{doc.page_content}"
        for doc in items
    )

prompt = ChatPromptTemplate.from_template("""
Answer using only the context below. If the answer is not present, say you do not know.
Include the source name when possible.

Context:
{context}

Question: {question}
""")

chain = (
    {
        "context": retriever | format_docs,
        "question": RunnablePassthrough(),
    }
    | prompt
    | ChatOpenAI(model="gpt-4o-mini", temperature=0)
    | StrOutputParser()
)

print(chain.invoke("Can annual customers get a refund?"))

The prompt tells the model not to answer beyond context.
Metadata makes citations possible.
In production, evaluate whether retrieved chunks actually contain the answer.

Inspect Retrieval Before Generation

Testing retrieval independently reveals whether weak answers come from search or generation.

Inspect Retrieval Before Generation

query = 'How long are audit logs retained?'
documents = [
    {'source': 'policy.md', 'content': 'Audit log retention is 90 days.'},
    {'source': 'faq.md', 'content': 'Password reset links expire after one hour.'},
]

for rank, document in enumerate(documents, start=1):
    print(rank, document['source'], document['content'])

assert 'retention' in documents[0]['content'].lower()

Output

1 policy.md Audit log retention is 90 days.
2 faq.md Password reset links expire after one hour.

Replace the deterministic fixture with retriever.invoke(query) in the project, then retain the same ranking and relevance assertions.

Before you move on