RAG Strategies

This is part of the AI Agents series. All code is at github.com/achintmehta/langchain.

What is RAG?

Retrieval-Augmented Generation (RAG) is the technique of giving an LLM access to relevant documents at the time it generates an answer, rather than relying on knowledge baked into its weights. You retrieve relevant chunks from your vector database, include them in the prompt, and ask the model to answer based on them.

The advantages are significant: the model can answer questions about private data it was never trained on, it can cite sources, and you can update the knowledge base without retraining anything. The disadvantages are that the answer quality is only as good as retrieval quality, and you are consuming context window space with the retrieved chunks.

Naive RAG, the baseline

The simplest possible pipeline:

1. Embed the user's question.
2. Do a similarity search against your vector database to get the top-k chunks.
3. Put those chunks + the question into a prompt.
4. Send to the LLM and return the answer.

In LangChain (llm is the ChatOpenAI client from Part 2, db the pgvector store from Part 4, and as_retriever the wrapper introduced in Part 5):

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

retriever = db.as_retriever(search_kwargs={"k": 4})

rag_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a helpful assistant. Answer the question using only
the context below. If the answer is not in the context, say so.

Context:
{context}"""),
    ("human", "{question}")
])

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | rag_prompt
    | llm
    | StrOutputParser()
)

answer = chain.invoke("What is the capital of France?")
print(answer)

RunnablePassthrough() is a LangChain component that passes its input through unchanged. Here it routes the raw question to the prompt's {question} slot while the retriever | format_docs branch handles the {context} slot. Both branches run in parallel before the prompt is assembled.

This naive baseline works well for simple, precise queries against focused document sets. Its weaknesses show when the query is ambiguous, when the answer spans multiple chunks, or when vocabulary in the query doesn't match the phrasing in the documents.

First upgrades: transform the query

Before changing the pipeline itself, remember that the cheapest improvements to naive RAG all live on the query side, and you already have them from the last part: rewrite the raw question into document vocabulary, generate multiple query variants and fuse their results with Reciprocal Rank Fusion (RAG-Fusion), embed a hypothetical answer instead of the question (HyDE), or route the query to the right data source. Measure naive RAG first, apply those where it fails, and only then reach for the heavier patterns below, which restructure what gets indexed and how the pipeline runs rather than what gets asked.

RAPTOR, recursive summaries for long documents

RAPTOR addresses a structural problem: with naive RAG you can retrieve individual chunks, but you lose the big picture. A user asking "summarise the main conclusions of this report" will get back detail-level chunks rather than a high-level overview.

RAPTOR builds a tree of summaries over your corpus. The leaves are your original chunks. The model clusters similar chunks and summarises each cluster; those summaries become new nodes one level up. The process repeats until you have a single root summary.

At retrieval time, you can search at any level of the tree. A high-level question retrieves from the upper tree; a specific factual question retrieves from the leaves. Retrieval can also start at a high level and descend.

This works best for long, structured documents, technical manuals, research papers, legal agreements, where both high-level and detailed queries need to be supported.

GraphRAG, knowledge graphs for multi-hop questions

Standard vector similarity search finds chunks that are locally similar to the query. It struggles when the answer requires combining information from multiple places in the corpus that are not individually similar to the query, the "multi-hop" problem.

GraphRAG adds a knowledge graph layer. During indexing, an LLM extracts entities (people, products, concepts) and relationships (works_for, depends_on, contradicts) from chunks, building a graph. At query time, vector search finds a starting set of nodes, and graph traversal expands outward along edges to collect related context.

This makes GraphRAG much stronger for questions like "Which engineers worked on the project that uses Library X?", a question that requires hopping through multiple document relationships.

Agentic RAG, retrieval inside an agent loop

Rather than running retrieval as a fixed pipeline step, agentic RAG wraps the whole thing in a LangGraph agent loop. The agent can:

• Try a retrieval, decide the results are insufficient, and retry with a rephrased query.
• Decompose a complex question into sub-questions, retrieve for each, and synthesise.
• Fall back to a web search if the document store doesn't have the answer.
• Combine document retrieval with SQL queries or API calls when the question spans both.

We have not built agents yet, that is where this series is headed, and this pattern is covered properly in the agent architectures part.

Self-RAG and CRAG, evaluating retrieval quality

Even with good chunking and embedding, retrieval sometimes returns chunks that are not actually relevant to the question. Self-RAG and CRAG add an evaluation step that scores each retrieved chunk for relevance before it is sent to the LLM.

If the scores are low, the pipeline can re-retrieve with a different query, fetch from a different source, or fall back to a parametric answer (the model's own knowledge). This self-correction loop significantly improves answer quality on questions where naive retrieval would return poor context.

Once you know LangGraph (two parts from now), this is straightforward to implement as a conditional edge: after the retrieval node, route to a grader node; if the grade is below a threshold, loop back to re-query. File the pattern away for then.

What's next

Retrieval lets an LLM know things it was never trained on. The next step is letting it do things: the next part explains tool calling, the mechanism that lets a model ask your code to run a function, which is the foundation every agent in the rest of this series is built on.