How to evaluate a RAG assistant before you trust it

A demo that answers three questions well proves very little. Before a RAG assistant reaches your staff or your customers, test it the way you would test any system that people will rely on. The good news is that careful evaluation does not need special software. It needs a good list of questions, an honest marking scheme and the discipline to repeat the test.

RAG systems can fail in two separate places, and it is important to tell them apart. Retrieval can fail, which means the right passage is never found. Or generation can fail, which means the right passage is found, and the answer is still wrong or unsupported. Good evaluation measures both.

Step 1: build a test set

Collect 30 to 50 real questions from the people who will use the assistant, drawn from emails, support tickets or chat logs. As the project matures, grow the set to 100 or more. For each question, write the correct answer and note the document and passage where it lives.

• Easy questions, whose answer sits in a single clear paragraph.
• Hard questions, which need several passages or careful reading.
• Unanswerable questions, which your documents do not cover. The right response is to say so.
• Variations of the same question in different words, to test robustness.

Write the answers before you see what the system says, to keep yourself honest. Keep a part of the set aside that you never use while tuning, as a final fair test.

Step 2: check retrieval first

For every question, check whether the right passage appeared in the passages given to the model. Record the hit rate, which is the share of questions where it did. This single number tells you a great deal.

• A low hit rate means the assistant cannot answer correctly however good the model is. Fix chunking, search settings, document quality or the embedding model first.
• A high hit rate shifts attention to the next step.
• Also look at the rank: if the right passage is usually fifth, not first, a re-ranker may help.

Step 3: check the answers

Read each answer against the correct one and the retrieved sources. Mark it on a few simple points.

• Correct: does it match the right answer?
• Complete: does it include what the person needs, such as conditions and exceptions?
• Grounded: is every claim supported by the retrieved passages?
• Cited: do the citations point to passages that really support the statement?
• Appropriate: is the tone and length right for the audience?

Use at least two people to mark a sample, and compare. If they disagree often, your marking guide is unclear, and it needs sharpening before the numbers mean anything. Automated checks, such as using a second model to judge groundedness, can help at scale, but always check a sample by hand.

Step 4: test the edges

Real users will not stick to your tidy questions. Probe the awkward cases deliberately.

• Out-of-scope questions: it should say it does not know, and point to a human or a better source.
• Contradictory documents, or an old version beside a current one: it should prefer the right one or flag the conflict.
• Vague or misspelled questions: it should ask for clarification rather than guess.
• Sensitive topics and access: it must not reveal documents the user is not allowed to see.
• Attempts to change its behaviour, including instructions hidden inside a document.

Step 5: keep measuring after launch

Quality drifts. Documents change, the people asking change, and models are updated. Build evaluation into routine.

• Re-run the full test set after every change to documents, prompts, search settings or models.
• Each month, review a sample of real questions and add the failures to the test set.
• Give users an easy way to flag a bad answer, and read the flags.
• Track the share of questions where the assistant declined, and check that it declined for the right reasons.

A simple scorecard

• Retrieval hit rate: the right passage found in the top results.
• Answer accuracy: marked correct and complete.
• Groundedness: no claims beyond the sources.
• Correct refusals: unanswerable questions handled properly.
• Escalations: questions passed to a person, and whether they should have been.

Decide the minimum score for each before launch, for example “at least 90 percent correct on answerable questions, and at least 95 percent correct refusals on unanswerable ones”. If the assistant does not reach it, you have a clear list of what to fix.

Common mistakes

• Judging quality from a handful of impressive demos.
• Testing only with questions written by the people who built the system.
• Looking only at the final answer, so that you cannot tell whether retrieval or generation failed.
• Never testing what happens when the answer is missing.
• Evaluating once, and not again after changes.

A worked mini-example: marking five answers

Suppose your test set has five questions, and you mark each answer for retrieval and correctness.

• Q1, leave carry-over: right passage found, answer correct and cited. Pass.
• Q2, travel approval: right passage found, but the answer omits the exception for managers. Retrieval pass, answer fail: improve the prompt or the chunking.
• Q3, equipment returns: right passage not retrieved. Retrieval fail: look at how the document is split, or add keyword search.
• Q4, a question the documents do not cover: the assistant invents a policy. Refusal fail: strengthen the instruction, and test again.
• Q5, a rephrased version of Q1: correct. Pass.

The score is three passes out of five, and the table tells you what to fix, in what order. This is far more useful than a general feeling that it is “mostly fine”.

Tools that can help

• A spreadsheet is enough to start. List the question, expected answer, retrieved passages, the answer given and your marks.
• Open-source evaluation libraries can compute metrics such as faithfulness and context precision automatically. Use them to scale up, but calibrate them against human marking.
• Logging and tracing tools show each step of a real question, which makes debugging much faster.

Frequently asked questions

How big should my test set be?

Start with 30 to 50 questions, and grow to 100 or more for production systems, adding every failure you discover.

Can another AI model mark the answers?

It can help at scale, particularly for groundedness. Calibrate it against human marks on a sample, and keep people in the loop for important cases.

How often should I re-test?

After every change to documents, prompts, search or models, and on a regular schedule such as monthly.

What score is good enough?

It depends on the stakes. Set targets in advance, with stricter ones for areas where a wrong answer is costly.

Evaluation is what turns an assistant from a clever demonstration into a tool people can depend on. It also gives you something rare in AI projects: evidence. When someone asks whether the assistant is good enough, you can show them the numbers.