LangChain Conversational RAG: Chat History, Query Rewriting and Source-Grounded Answers

Why Query Rewriting Matters

Conversational RAG is harder than single-turn RAG because users ask follow-up questions like “what about enterprise plans?” or “explain step two again.” The retriever needs a standalone search query, while the answer model needs both retrieved evidence and conversation context.

A reliable design separates query rewriting from answering. The first model call rewrites the user question for search. The second model call answers using retrieved documents and visible conversation history.

Retrievers do not understand pronouns and vague follow-ups as well as chat models. If the user asks “does it support that?”, the retriever needs a rewritten query such as “Does the product support SAML SSO for enterprise customers?”

Rewrite only the search query; do not answer during rewriting.
Keep the original user message for the final response.
Limit history length so old turns do not pollute retrieval.

Conversation State

Use chat history when it changes the meaning of the current question. Do not stuff the entire conversation into every prompt forever. Summarize old turns or store only relevant state when conversations become long.

Persist history by session ID.
Store human and assistant messages, not hidden internal tool traces.
Handle topic changes by allowing retrieval to ignore irrelevant history.

Standalone Question Rewriter

The rewriter turns follow-up questions into search-friendly questions.

Standalone Question Rewriter

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_openai import ChatOpenAI

rewrite_prompt = ChatPromptTemplate.from_messages([
    ("system", "Rewrite the latest user question as a standalone search query. Do not answer it."),
    MessagesPlaceholder("chat_history"),
    ("human", "{question}"),
])

rewriter = rewrite_prompt | ChatOpenAI(model="gpt-4o-mini", temperature=0) | StrOutputParser()

The rewriter should produce a query, not a final answer.
This improves retrieval for pronouns, ellipsis, and follow-up language.

Answer with History and Retrieved Context

The answer prompt uses history for continuity and documents for factual claims.

Answer with History and Retrieved Context

answer_prompt = ChatPromptTemplate.from_messages([
    ("system", """Answer using only the context.
If the context is insufficient, say you do not know from the documents.
Cite source filenames."""),
    MessagesPlaceholder("chat_history"),
    ("human", "Question: {question}\n\nContext:\n{context}"),
])

def answer_question(question, chat_history, retriever, model):
    rewritten = rewriter.invoke({"question": question, "chat_history": chat_history})
    docs = retriever.invoke(rewritten)
    context = "\n\n".join(
        f"Source: {d.metadata.get('source')}\n{d.page_content}"
        for d in docs
    )
    return (answer_prompt | model | StrOutputParser()).invoke({
        "question": question,
        "chat_history": chat_history,
        "context": context,
    })

The final answer receives the original question so tone and intent are preserved.
Retrieved context is the evidence boundary for factual claims.

Before you move on