RAG Explained: How to Make AI Answer From Your Data

What problem does RAG solve?

RAG solves two limits that every LLM (large language model) has: it knows nothing about your company, and it can answer confidently when it does not know. You met both limits in Article 2:

1. A knowledge cutoff. The model knows nothing after its training date, and nothing about your company at all. Your HR policy, your course syllabus and your Q3 pricing sheet were never in the training data.
2. A tendency to hallucinate. Ask about something it does not know, and it will often produce a fluent, confident, wrong answer instead of saying "I don't know."

You could try pasting your documents into the prompt every time. For one PDF, that works, and for one PDF it is the right answer (more on that below). But you cannot paste 4,000 support tickets or a 900-page policy manual into a context window on every query. The context window is the amount of text a model can read in one go, measured in tokens, which are small chunks of text, often a word or part of a word. Even models with million-token windows get slower, costlier and less accurate as you fill them.

RAG is the standard engineering answer to this problem.

What is RAG, in one sentence?

RAG, or Retrieval-Augmented Generation, means that before the LLM answers, a search step finds the most relevant pieces of your documents and pastes them into the prompt, so the model answers from evidence, not memory. The term comes from Lewis et al., 2020.

That is all there is to it. RAG is not a new kind of model. It is a pipeline wrapped around an ordinary LLM. The model itself is unchanged. What changes is what you feed it.

A useful mental model: RAG turns a closed-book exam into an open-book exam. The student (the LLM) is the same. But now it gets to look at the right page of the textbook before answering, and it can cite the page.

How does a RAG pipeline work, step by step?

A RAG pipeline has two phases: an indexing phase, done once and offline, and a query phase, run every time a user asks something.

Indexing phase. Ingest your documents, split them into chunks of roughly 200 to 800 tokens, embed each chunk, and store the vectors in a vector database. A chunk is one passage cut from a document. An embedding turns a chunk into a list of numbers that captures its meaning (Article 5 explains how). A vector database stores those lists of numbers so they can be searched by meaning.

Query phase. Embed the user's question with the same embedding model, otherwise the numbers cannot be compared. Then run a similarity search, which scores how close the question's numbers are to each chunk's numbers (a measure called cosine similarity) and returns the top-k chunks. Here k is the number of best matches you ask for, usually 3 to 10. Next, build an augmented prompt, and let the LLM generate a grounded, cited answer.

The augmented prompt typically reads: "Answer the question using only the context below. If the context doesn't contain the answer, say so." That instruction does a great deal of anti-hallucination work. It converts "make something up" into "say I don't know," which for business applications is the difference between usable and dangerous.

A worked example. Illustration only, using a made-up policy. Suppose the indexed HR handbook contains these chunks:

• Chunk A: "Sick leave: employees get 12 days of paid sick leave a year. A medical certificate is needed for absences longer than 2 days."
• Chunk B: "Casual leave: employees may take casual leave with their manager's approval."
• Chunk C: "Work from home: requests go to the reporting manager."

An employee asks: "How many days of medical leave do I get?" The handbook says "sick leave", not "medical leave", but the two phrases have similar meaning, so their embeddings are close and Chunk A scores highest. The augmented prompt sent to the LLM is:

Answer the question using ONLY the context below.
If the context does not contain the answer, reply exactly:
"The provided documents do not cover this."

Context:
[Chunk A] Sick leave: employees get 12 days of paid sick leave a year...
[Chunk B] Casual leave: employees may take casual leave with their manager's approval.
[Chunk C] Work from home: requests go to the reporting manager.

Question: How many days of medical leave do I get?

The model can now answer "12 days of paid sick leave a year, and a medical certificate is needed beyond 2 days," and point to Chunk A as the source. If the same employee asks about maternity leave, none of the chunks cover it, so the instruction makes the model reply "The provided documents do not cover this." Without the instruction, many models would fill the gap with a plausible guess.

Why does RAG beat fine-tuning for most business use cases?

For most business use cases, RAG beats fine-tuning because the need is usually knowledge, and RAG supplies knowledge. Learners often assume "teach the model our data" means fine-tuning, which means training an existing model further on your own examples. For knowledge, it usually does not do what people expect.

The rule of thumb we teach: fine-tuning teaches skills, RAG supplies knowledge. If your problem is "the model doesn't know our stuff," reach for RAG. Many production systems eventually use both, but almost nobody should start with fine-tuning.

Where does RAG go wrong?

RAG goes wrong in five common ways, and most of them are problems with retrieval, not with the LLM. Vendor demos can make RAG look easy. Production tells a different story. The failure modes, ranked roughly by how often we see them:

1. Retrieval misses. The right chunk exists but does not get retrieved. The question's wording differs too much from the document's wording, or the answer is spread across three chunks and only one comes back. The LLM cannot fix what retrieval never gave it. Most "RAG doesn't work" complaints are actually retrieval problems.
2. Bad chunking. A table split down the middle, or a policy clause separated from its exception. The retrieved chunk is technically relevant but incomplete or misleading.
3. Garbage in the corpus. The corpus is the full set of documents you indexed. If it holds outdated PDFs, duplicates, or drafts sitting next to finals, RAG faithfully retrieves the wrong version and cites it confidently. RAG makes your document hygiene problem visible. It does not solve it.
4. The "lost in the middle" effect. Stuff 20 chunks into the prompt and models tend to pay most attention to the first and last ones (Liu et al., 2023). More context is not automatically better.
5. Questions retrieval cannot answer. "Summarise all complaints from last quarter" is an aggregation query, one that must count, sum or scan everything. Similarity search returns a few passages, so it cannot do that. These questions need a database query or an agent, not vanilla RAG. Our guide on connecting LLMs to SQL shows how a text-to-SQL agent handles them.

When an answer is wrong, work out which part of the pipeline failed:

A debugging tip that saves weeks: always log which chunks were retrieved for each answer. When an answer is wrong, check the chunks first. In most cases the diagnosis takes minutes once you can see them.

How should you chunk documents?

Chunk by structure and test the size on your own documents. If retrieval quality is the number one failure point, chunking is the number one lever. Three practical guidelines:

• Respect structure. Split on headings, paragraphs and sections, not on a blind character count. A chunk should be a self-contained thought.
• Size for the content. FAQ-style content works with small chunks of about 200 to 300 tokens. Dense policy or legal text often needs 500 to 800 tokens. There is no universal right number, so test on your documents.
• Use overlap. An overlap of 10 to 20% between consecutive chunks means a sentence at a boundary appears in both, so meaning is not cut mid-thought.

One more technique worth knowing by name is hybrid search. It combines vector similarity with keyword search (a classic method called BM25). Vectors catch paraphrases ("sick leave" and "medical absence"). Keywords catch exact terms that vectors can miss, such as product codes, names and "Form 16". Most mature RAG systems use both.

Do you even need full RAG?

You may not need RAG at all. In the same spirit as Article 5 ("you may not need a vector database"), match the tool to the size of the problem:

• One document, occasional questions? Paste it into the context window. Modern models accept very long inputs, so there is no infrastructure to build.
• A few dozen documents, a small team? Tools such as NotebookLM, or Claude and ChatGPT Projects with uploaded files, let you ask questions about your documents without building anything.
• Hundreds of documents, many users, citations, access control and frequent updates? Now you are building real RAG: a vector database, a chunking strategy, an evaluation set and the rest.

Also skip plain RAG when the job is not about knowledge. Counting and summing across everything needs a database query or an agent, and a change in tone or format is a job for fine-tuning.

Build the pipeline when the scale or governance demands it, not because the architecture diagram looks impressive.

Hands-on: how do you build a mini-RAG in about 20 minutes?

You will build a small RAG system in one Google Colab notebook, using the free tier. Colab is Google's free notebook that runs Python in your browser.

1. Corpus. Take any policy-like document: an HR handbook excerpt, a course brochure or your website's FAQ. Split it manually into 10 to 15 paragraph chunks in a Python list.
2. Index. Using sentence-transformers (the all-MiniLM-L6-v2 model, the same as in Article 5), embed all the chunks.
3. Retrieve. Write a function that embeds a question, computes cosine similarity against all the chunks, and returns the top 3.
4. Augment and generate. Paste the top-3 chunks and your question into any LLM using this template:

Answer the question using ONLY the context below.
If the context does not contain the answer, reply exactly:
"The provided documents do not cover this."

Context:
[chunk 1] [chunk 2] [chunk 3]

Question: [your question]

5. Break it. Ask three questions: one clearly answered in the corpus, one partially answered, and one not covered at all. Watch how the "only from context" instruction changes behaviour on question 3. Then remove the instruction and ask again. That difference is the whole anti-hallucination argument for RAG, demonstrated in one minute.

Stretch goal: deliberately re-chunk your corpus badly, splitting mid-sentence every 100 characters, and re-run question 1. You will watch retrieval quality collapse, and chunking stops being abstract.

What should you do next?

1. Run the mini-RAG exercise above on a document you know well, so you can judge the answers yourself.
2. Write an evaluation set of 30 to 50 real questions with known answers and known source passages, before you build anything larger.
3. Log the retrieved chunks for every answer in any RAG system you try, so a wrong answer can be diagnosed in minutes.

Frequently asked questions

Is RAG a type of model? No. RAG is an architecture: a retrieval step wrapped around a standard LLM. Any capable LLM can be the "G" (generation) in RAG.

Does RAG eliminate hallucinations? It substantially reduces them for questions your documents cover, especially with a strict "answer only from context" instruction. It does not eliminate them. The model can still misread a chunk, blend two chunks incorrectly, or answer confidently when retrieval returned junk. Evaluation and citations remain necessary.

Won't million-token context windows kill RAG? Long contexts shrink the territory where RAG is mandatory, but retrieval survives for three reasons: cost (a million tokens per query adds up fast), latency (speed of response), and accuracy (models still attend unevenly across huge contexts). The practical answer for 2026: long context for small corpora, RAG for large ones, and hybrids in between.

What does a basic RAG stack cost? For a pilot, it is effectively the LLM API cost plus very little else. Open-source embedding models are free, and vector stores such as Chroma or FAISS run locally. Costs grow with scale and managed infrastructure, not with getting started.

Can RAG work in Hindi, Bengali or mixed-language documents? Yes. Use a multilingual embedding model so a question in one language can retrieve chunks in another. Test retrieval quality for each language, because it is rarely uniform.

How do I measure whether my RAG system is any good? Build a small evaluation set of 30 to 50 real questions with known answers and known source passages. Measure retrieval hit rate (did the right chunk come back?) and answer accuracy separately. That tells you which half of the pipeline to fix.

Key takeaways

• RAG = search first, then generate. It is an open-book exam for LLMs, and it is a pipeline, not a model.
• Fine-tuning teaches skills, RAG supplies knowledge. For "the model doesn't know our data," RAG is almost always the right first move.
• Most RAG failures are retrieval failures, usually traceable to chunking and document hygiene. Log your retrieved chunks.
• The "answer only from the provided context" instruction is the cheapest, highest-leverage hallucination control in applied GenAI.
• RAG cannot count or aggregate. When questions become "do things," you need agents.

What is next in this series?

Next up: AI Agents Explained, When Your AI Stops Answering and Starts Doing. You will see how an LLM can use tools, check its own work and carry out multi-step tasks, which is what RAG on its own cannot do.

Related articles

• What an embedding actually is
• How do LLM context windows work?
• RAG vs fine-tuning: which one should you use?
• Connecting LLMs to SQL databases

More in GenAI / LLM.