Agents need external knowledge when facts are private, recent, domain-specific, or too large to fit reliably in model parameters. Retrieval-augmented generation supplies selected evidence at the moment a decision needs it.
Agentic RAG adds control around retrieval: the agent may rewrite queries, choose among search tools, inspect results, request more evidence, or stop when sources are sufficient.
A trustworthy knowledge workflow preserves source identity, separates evidence from instructions, and refuses to invent answers when retrieval fails.
RAG does not give an agent truth. It gives the agent evidence candidates. The system must still retrieve well, rank sources, preserve provenance, and admit when the evidence is insufficient.
A practical RAG system ingests content, splits it into meaningful units, attaches metadata, creates searchable representations, retrieves candidates, reranks them, and sends a small evidence set to the model.
Every stage affects quality. Poor document parsing or chunking cannot be repaired by a clever final prompt.
Not every question requires search. The agent should retrieve when the answer depends on external facts, then select the correct knowledge source for the task.
Query rewriting can turn a conversational request into useful search terms, but the original user meaning must remain visible so the rewritten query does not drift.
Every retrieved passage should carry a source identifier and enough metadata to create a citation or audit record. Access checks must happen before retrieval results enter model context.
Filtering after generation is too late: the model may already have seen information the current user was not permitted to access.
Documents and webpages may contain instructions such as "ignore previous rules" or attempts to exfiltrate data. Retrieved text is evidence, not authority, and must never be allowed to redefine system policy.
Mark content boundaries, remove active content where appropriate, limit tool permissions during research, and instruct the model to extract facts rather than follow embedded commands.
When a grounded answer is wrong, determine whether the correct source was missing, ranked too low, ignored by the model, or contradicted by another source. Retrieval and generation require separate metrics.
Useful measures include retrieval recall, precision, citation correctness, answer faithfulness, abstention quality, and performance on questions with no valid answer.
Retrieval-augmented generation for agents is not just a vector search call before an answer. It is evidence management inside a decision-making loop. The agent may retrieve documents to answer, choose a tool, justify an escalation, or decide that it does not have enough evidence to continue.
The retrieval layer should separate search, fetch, rerank, cite, and verify. Search finds candidates. Fetch retrieves authoritative source text. Reranking chooses the most relevant evidence. Citation logic preserves provenance. Verification checks whether the final answer is actually supported by the retrieved material.
Agents need retrieval permissions. A user may be allowed to ask a question but not allowed to read every document that could answer it. Enforce access before search results enter context, not after the model has already seen them. Retrieved content can also contain prompt injection, so it must be treated as untrusted evidence rather than instruction.
Good RAG systems include no-evidence behavior. If sources are missing, stale, contradictory, or outside permission boundaries, the correct answer may be a clarification or refusal. A grounded agent is valuable partly because it knows when not to invent.
Different agent tasks need different retrieval strategies. A customer-support agent may need policy snippets plus account-safe order facts. A research agent may need broad exploration followed by source comparison. A coding agent may need exact file search before semantic search. One retrieval pattern rarely fits all workflows.
For high-precision tasks, prefer structured filters, keyword search, and exact identifiers before semantic expansion. For exploratory tasks, use hybrid search and reranking. For long documents, retrieve sections and summaries separately so the agent can cite precise evidence without flooding the context window.
Retrieval should also be observable. Track the query, filters, candidate count, selected sources, citations used, and final groundedness score. If the agent gives a bad answer, you need to know whether retrieval failed, reranking failed, or the model ignored good evidence.
Keep retrieval fresh. Index versions, document updates, deleted content, and permission changes must propagate into the agent. A good answer from stale or unauthorized content is still a product failure.
A strong RAG review separates retrieval quality from answer quality. First check whether the correct sources were available and retrievable. Then check whether the agent selected the right pieces. Only after that should you judge whether the answer used the evidence correctly.
Create a small review table with the user question, expected source, retrieved source, cited source, final claim, and verdict. This makes grounding failures visible. Sometimes the retriever misses the right document. Sometimes the model ignores the right document. Sometimes the citation points to a source that does not support the claim.
Also include no-evidence cases. The system should know how to say that approved sources do not contain enough information. In many business settings, a careful no-answer is more valuable than a fluent guess.
The search layer filters documents before returning context to the agent.
documents = [
{"id": "public-1", "team": "all", "text": "Refunds are allowed within 14 days."},
{"id": "finance-2", "team": "finance", "text": "Manual refund approval limit is $500."},
]
def retrieve(query: str, user_team: str) -> list[dict]:
words = set(query.lower().split())
results = []
for doc in documents:
if doc["team"] not in {"all", user_team}:
continue
score = len(words.intersection(doc["text"].lower().split()))
if score:
results.append({**doc, "score": score})
return sorted(results, key=lambda item: item["score"], reverse=True)
for result in retrieve("refund approval", user_team="support"):
print(result["id"], result["text"])The final response must contain citations or explicitly report insufficient evidence.
def build_grounded_answer(question: str, evidence: list[dict]) -> dict:
if not evidence:
return {
"answer": "I do not have enough approved evidence to answer.",
"citations": [],
"grounded": False,
}
return {
"answer": evidence[0]["text"],
"citations": [item["id"] for item in evidence],
"grounded": True,
}
print(build_grounded_answer("What is the refund window?", [
{"id": "policy-17", "text": "Refunds are available within 14 days."}
]))No. RAG retrieves external knowledge. Memory stores information about prior interactions, users, or workflow state. A system may use both.
Retrieve when the answer depends on external facts. Simple conversational or transformation tasks may not need search.
Surface the conflict, prefer authoritative and current sources using explicit rules, and escalate when the decision is high risk.
Explore 500+ free tutorials across 20+ languages and frameworks.