RAG explained: how AI answers from your own documents

A language model such as the ones behind modern chat assistants has read a vast amount of text, but it has never read your documents. It does not know your refund policy, your project reports or last month’s meeting notes. If you ask it about them, it may give a confident answer that is simply invented. This is often called hallucination, and it is the main reason organisations hesitate to rely on AI for real work.

Retrieval-augmented generation, or RAG, is the most widely used way to fix this. Instead of hoping the model remembers the right facts, you hand it the relevant passages at the moment of the question and ask it to answer from those.

The idea in one picture

Think of an open-book exam. A student who relies on memory may guess. A student who is allowed to look things up in the right chapter, and then writes the answer in their own words, is far more accurate. RAG turns the AI into the second student. The “retrieval” step finds the right chapter, and the “generation” step writes the answer.

The five steps

1. 01Ingest. Gather the documents you want the assistant to know: policies, manuals, reports, web pages, tickets. Clean them up and note useful details such as title, date and owner.
2. 02Chunk. Split each document into passages small enough to be specific, often a few paragraphs, with a little overlap so ideas are not cut in half.
3. 03Embed and index. Convert each passage into a list of numbers that captures its meaning, called an embedding, and store them in a searchable index.
4. 04Retrieve. When someone asks a question, convert it the same way and fetch the passages whose meaning is closest.
5. 05Generate. Give those passages to the language model with clear instructions: answer only from them, say when the answer is not there, and cite the sources.

A concrete example

A company loads its HR policies into a RAG assistant. An employee asks, “How many days of leave can I carry over into next year?” The system finds the two paragraphs of the leave policy that mention carry-over, gives them to the model, and the model replies with the rule and a link to the policy page. If the policy does not say, a well-built assistant replies that it could not find the answer and suggests contacting HR, instead of making one up.

Why RAG is popular

• Answers are grounded in your own, up-to-date information.
• You can show where an answer came from, which builds trust and makes errors easy to spot.
• Updating knowledge means updating documents, not retraining a model.
• You can control who sees what, by limiting retrieval to documents each user is allowed to read.

RAG or fine-tuning?

Fine-tuning changes the model itself, by training it further on examples. It is good for teaching a style, a format or a specialised skill. It is poor at teaching facts that change, and it cannot show its sources. RAG is better when answers must reflect current documents and be traceable. Many teams start with RAG, because it is faster to build and easier to check, and add fine-tuning only if a clear need appears.

What makes a RAG system good

• Document quality. Out-of-date, duplicated or contradictory files produce confusing answers. Cleaning the sources is often the biggest improvement.
• Sensible chunking. Passages that are too long blur several ideas. Passages that are too short lose context.
• Useful metadata. Dates, document types and owners let you filter, for example to prefer the latest version.
• Hybrid search. Combining meaning-based search with keyword search finds exact codes and names that meaning-based search can miss.
• Re-ranking. A second pass that reorders the retrieved passages by relevance often improves accuracy noticeably.
• Clear instructions. The prompt should say what to do when the answer is missing, and require citations.

Where it still fails

• The right passage is never retrieved, so the model answers from weak or irrelevant text.
• The answer is spread across several documents and the system fetches only one.
• The model ignores the instructions and adds details that are not in the sources.
• Documents contain instructions that try to steer the model, which is a security risk when you index content you do not control.
• No one tests it, so quality is judged by a few good demos.

Do you need RAG?

RAG fits when people repeatedly ask questions whose answers sit in a body of documents: policies, product manuals, research reports, case files. It is a poor fit when the answer needs live data from a database, in which case a tool or query is better, or when the question needs deep reasoning across many facts that no single passage contains.

Getting started

• Pick one body of documents and one group of users.
• Collect 30 to 50 real questions with known answers.
• Build the simplest pipeline, then measure it against those questions.
• Improve retrieval before changing models.
• Add access control and logging before opening it to more people.

A day in the life of one question

It helps to follow a single question through the system. An employee types: “Can I claim a taxi home if I work past nine?”

1. 01The question is converted into an embedding, a list of numbers that captures its meaning.
2. 02The index returns the six passages whose meaning is closest. Two come from the travel policy, one from an overtime guide and three are weak matches.
3. 03A re-ranking step reorders them and keeps the best four.
4. 04The prompt tells the model to answer only from those four passages, to cite them and to say so if the answer is missing.
5. 05The model replies: taxis are reimbursed after nine in the evening with a receipt, and the answer cites the two policy passages.
6. 06The employee clicks a citation, sees the original paragraph and trusts the answer, or spots a mistake and reports it.

Notice how many steps sit before the language model. This is why improving retrieval usually helps more than changing the model.

Costs, speed and privacy

• Cost: you pay for embedding documents once, for each question that is asked, and for the model call that writes the answer. Caching common questions reduces it.
• Speed: retrieval takes a fraction of a second, and generation takes a few seconds, which is why many assistants stream the answer as it is written.
• Privacy: if you use outside services for embedding or generation, your text is sent to them. Check their terms, or run models on your own servers for sensitive material.
• Access: the system must only retrieve documents that the person asking is allowed to read.

Frequently asked questions

Does RAG stop the model from hallucinating?

It greatly reduces it by giving the model real source text, but it does not eliminate it. The model can still misread a passage or add details, which is why evaluation and citations matter.

How much data do I need?

There is no minimum. RAG works for a handful of documents and for millions. What matters is that the documents are accurate and current.

Can RAG read tables, scanned PDFs and images?

Only if they are converted to text first. Scanned files need text recognition, and tables need careful handling so that rows keep their meaning.

Should I fine-tune a model instead?

Usually not at first. Fine-tuning teaches style or skills, not facts that change. RAG is easier to update and to audit, so most teams start there.

Done well, RAG gives you an assistant that knows your organisation, shows its sources and admits what it does not know. That combination is what makes it safe enough to use.