LangChain Embeddings and Vector Stores: Chunking, Indexing and Search Quality

Chunking Strategy

Embeddings convert text into numeric vectors so related meaning can be searched by distance. Vector stores save those vectors with document text and metadata. In a RAG system, this layer often decides answer quality before the LLM ever sees the prompt.

A serious developer should not treat vector search as magic. You need to design chunk boundaries, choose metadata, test top-k results, filter by document type or tenant, and measure whether the right evidence appears in retrieval.

Chunking is not only about character count. Good chunks preserve meaning. Policy documents may split by headings, code docs by functions, API docs by endpoint, and support articles by question-answer pairs.

Use overlap when an answer depends on text near a boundary. Too little overlap loses context; too much overlap wastes tokens and creates duplicate results.

Start with 700 to 1200 characters for documentation-style text.
Preserve source, section title, product, version, tenant, and URL in metadata.
Inspect the actual chunks before indexing; do not tune blind.
Evaluate top-k retrieval separately from final answer generation.

Vector Store Design

A vector store becomes part of your application contract. You need an indexing process, an update strategy, deletion behavior, and metadata filters so users do not retrieve documents they should not see.

Use local stores for learning and prototypes.
Use managed stores when you need persistence, scaling, backups, filtering, and multi-user operations.
Store metadata that supports authorization and query narrowing.
Rebuild or migrate indexes when chunking or embedding models change.

Index Documents with Metadata

This example creates searchable chunks and stores metadata used later for citations and filtering.

Index Documents with Metadata

from pathlib import Path
from langchain_community.document_loaders import TextLoader
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

docs = []
for file_path in Path("data/docs").glob("*.md"):
    loaded = TextLoader(str(file_path), encoding="utf-8").load()
    for doc in loaded:
        doc.metadata.update({
            "source": file_path.name,
            "doc_type": "support",
            "version": "2026-05",
        })
        docs.append(doc)

splitter = RecursiveCharacterTextSplitter(chunk_size=900, chunk_overlap=150)
chunks = splitter.split_documents(docs)

vectorstore = FAISS.from_documents(
    chunks,
    OpenAIEmbeddings(model="text-embedding-3-small"),
)
vectorstore.save_local("storage/support_index")

Source metadata is what lets final answers cite evidence.
Version metadata helps you remove or filter old documentation later.

Inspect Retrieval Before Calling the Model

Debug retrieval by printing sources and snippets. Do this before blaming the prompt.

Inspect Retrieval Before Calling the Model

retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
docs = retriever.invoke("How do enterprise customers configure SSO?")

for rank, doc in enumerate(docs, start=1):
    print(f"\n#{rank} source={doc.metadata.get('source')}")
    print(doc.page_content[:500])

If the right evidence is not in the retrieved docs, improve chunking, metadata, query rewriting, or search settings.
A retrieval inspection script is one of the highest-value RAG debugging tools.

Before you move on