What Is RAG? Retrieval-Augmented Generation Explained

The shortest useful mental model: RAG turns a closed-book exam into an open-book one. A plain LLM answers from what it memorized during training; a RAG system lets the model look up the right pages first and answer from them. The original 2020 paper by Lewis et al. named the pattern and showed that pairing a retriever with a generator beat a generator alone on knowledge-intensive tasks. The idea has since become the default way to put a company's own data behind a chatbot.

The word to hold onto is grounding. In a RAG answer, the facts come from documents you control and can inspect, not from the opaque weights of the model. That single shift — answering from retrieved evidence rather than memory — is what makes RAG answers fresher, more specific to your world, and checkable against a source.

A standalone LLM has three structural gaps, and RAG was built to close all three. First, a knowledge cutoff: the model only knows what existed in its training data, so it cannot tell you about last week's release or a policy that changed yesterday. Second, no access to private data: your internal wiki, your ticket history, your product docs were never in the training set, so the model has no way to know them. Third, hallucination: when a model does not know something, it will often produce a fluent, confident, and wrong answer rather than admit the gap.

RAG addresses each by moving knowledge out of the weights and into a searchable store that is consulted on every question. Want the answer to reflect today's docs? Update the documents — no retraining. Need the model to know your internal runbook? Index it. Worried about invented facts? Give the model real passages and instruct it to answer only from them. As AWS puts it, RAG lets you extend a model to your organization's internal knowledge base without retraining the model itself.

1. Ingest and index (done ahead of time). You take your source material — docs, PDFs, wiki pages, support tickets — and split it into chunks a few hundred tokens long. Each chunk is passed through an embedding model that turns text into a vector: a list of numbers that captures its meaning, so that passages about similar topics land near each other in vector space. Those vectors go into a vector database that can search by similarity. This indexing job runs once up front and again whenever your content changes.

2. Retrieve (on every query). When a question comes in, you embed it with the same model, then ask the vector store for the handful of chunks whose vectors sit closest to the query's — typically the top 3 to 8. Those are your candidate evidence.

3. Augment (on every query). You paste the retrieved chunks into the prompt as context, usually wrapped with an instruction like "answer using only the context below, and cite it." This is the step the name refers to: the prompt is augmented with retrieved knowledge.

4. Generate (on every query). The LLM reads the question plus the retrieved context and writes an answer grounded in those passages, ideally with citations back to the sources so a human can verify.

Embeddings are the translation layer. An embedding model maps text to a vector so that meaning becomes geometry — "reset my password" and "I forgot my login" end up close together even though they share no words. That is what lets retrieval find relevant chunks by topic rather than by exact keyword match.

The vector store is the searchable index of those embeddings. Its one job is fast similarity search: given the query vector, return the nearest chunk vectors. Popular options include pgvector, Pinecone, Weaviate, and FAISS, but the role is the same regardless of which you pick.

Chunking is how you cut documents before embedding. Chunk too large and each vector blurs several topics together, so retrieval gets imprecise; chunk too small and you sever the context a passage needed to make sense. Chunk size and overlap are among the most consequential knobs in a RAG system, which is why teams tune them per corpus.

Reranking is an optional refinement step. A first retrieval pass fetches a broad set of candidate chunks cheaply; a slower, more accurate reranker model then reorders them by true relevance and keeps only the best few for the prompt. It trades a little latency for noticeably better context.

The upside is why RAG is everywhere. It gives a model fresh and private knowledge without retraining — you update documents, not weights. It cuts hallucinations by handing the model real passages to reason over. It produces citeable answers, because every claim can point back to a source chunk, which is essential for trust in support, legal, and internal tooling. And it is usually far cheaper than fine-tuning: indexing a corpus costs a fraction of a training run, and refreshing knowledge is just a re-index. NVIDIA frames RAG as connecting a model to authoritative external sources so answers stay current and verifiable.

The distinctions are simple. Fine-tuning changes the model's weights, so use it to change how the model behaves — a voice, a strict output format, a specialized skill — not to feed it facts that change weekly. Long context means stuffing documents directly into a very large prompt; it works well for a handful of documents you already have in hand, but it cannot scale to a knowledge base of millions of tokens and pays the cost of re-reading everything on every call. RAG selects only the relevant slice of a large, changing corpus per query. The honest summary: fine-tune to change behavior, use RAG (or long context for small inputs) to change knowledge — and plenty of production systems use both together.

It is easy to confuse RAG with tool-calling because both bring outside information to the model, but the direction matters. RAG is fundamentally a read: fetch relevant text, ground the answer. Tools — including those exposed over the Model Context Protocol — let the model do things and get back live, structured results: query a database, file a ticket, trigger a deploy. Retrieval is one specific tool an agent might have, but most tools are about action, not lookup. For the boundaries between MCP, plain APIs, and function calling, see MCP vs API vs function calling; for how these pieces compose into something autonomous, see what is an AI agent. In practice a real agent uses both: RAG to know, tools to act.

There is a lesson in RAG that carries straight over to debugging. A RAG answer grounded in retrieved documents beats an ungrounded one for the same reason a coding agent grounded in a real reproduction beats one guessing from a description. When BugMojo hands an agent a captured bug over MCP — the rrweb session replay, the console output, the failing network request — the agent is reasoning over the actual state that broke, not a secondhand summary. That is grounding by another name: the model answers from evidence it can inspect, and the answer gets better because of it.