- 16 Oct, 2026
- 0 Comments
- 5 Mins Read
RAG Evaluation: How to Tell If Your LLM App Actually Works
Retrieval-augmented generation (RAG) is the most common way companies put large language models to work on their own documents. A demo takes an afternoon. Knowing whether it is good enough to ship takes discipline. Most teams test a RAG chatbot by asking it ten questions, liking eight answers, and calling it done. Then real users arrive, ask questions nobody imagined, and trust drops fast.
This guide gives you a practical RAG evaluation process you can run with a small team: what to measure, how to build a test set, and how to turn scores into release decisions.
Why RAG Apps Fail Quietly
A RAG pipeline has two halves, and either can fail without an obvious error:
- Retrieval finds the wrong chunks, misses the right one, or returns an outdated version of a policy.
- Generation ignores good context, blends two documents, or confidently invents an answer when the context was empty.
Because the output is fluent text, both failures look like success. That is why you need to evaluate the two halves separately, then together.
Step 1: Build a Golden Test Set
Everything starts with questions where you already know the right answer.
How many questions?
Start with 50 to 100. That is enough to spot patterns and small enough for a subject-matter expert to review in a day. Grow it over time from real user logs.
What to include
- Common questions your users ask every week.
- Hard questions that need two documents combined.
- Out-of-scope questions the bot should refuse (“What is our CEO’s salary?”).
- Trick questions with outdated or conflicting sources.
- Phrasing variants: the same question asked formally, casually and with typos.
For each question, record the expected answer and the document (or chunk) that contains it. That source label is what lets you score retrieval on its own.
Step 2: Measure Retrieval First
If retrieval is wrong, no prompt will rescue the answer. Measure it before you touch generation.
Key retrieval metrics
- Hit rate (recall@k): does the correct chunk appear in the top k results? Aim high here first; it is the ceiling for everything else.
- MRR (mean reciprocal rank): how near the top the right chunk lands. Higher means less noise in the prompt.
- Context precision: what share of retrieved chunks are actually relevant. Low precision wastes tokens and confuses the model.
Common retrieval fixes
- Change chunk size and overlap; tables and FAQs often need different chunking than long prose.
- Add hybrid search (keyword plus vector) so product codes and names are not missed.
- Add a re-ranker on the top 20 results.
- Attach metadata (department, date, version) and filter on it.
Re-run the test set after each change. One change at a time, so you know what helped.
Step 3: Measure Answer Quality
Once retrieval is solid, score the generated answers.
Faithfulness (groundedness)
Is every claim in the answer supported by the retrieved context? This is your main hallucination metric. An answer can be correct by luck and still be unfaithful, and that will bite you later.
Answer relevance and correctness
Does the answer address the question, and does it match the expected answer from your golden set? Partial answers count as partial.
Refusal behaviour
For out-of-scope questions, the right answer is “I don’t have that information.” Track how often the bot refuses correctly and how often it refuses questions it should answer. Both matter.
Citations
If your app shows sources, check that the cited document actually contains the claim. Wrong citations destroy trust faster than no citations.
Step 4: Use LLM-as-Judge, Carefully
Scoring hundreds of answers by hand does not scale. A common approach is to use a strong LLM as a judge with a clear rubric, for example: “Score 1 to 5 for faithfulness. Quote the unsupported sentence if any.”
Open-source frameworks such as Ragas, DeepEval and TruLens package these metrics so you do not have to write every judge prompt yourself.
Three rules keep LLM judges honest:
- Calibrate against humans. Have an expert score 30 to 50 answers and check the judge agrees most of the time before you trust it.
- Ask for reasons. A judge that must quote evidence is far more reliable than one that just outputs a number.
- Keep humans on the edge cases. Review every answer the judge scores low, plus a random sample of the high ones.
Step 5: Turn Scores Into Release Gates
Metrics only help if they drive decisions. Agree thresholds with the business owner before launch, for example:
- Retrieval hit rate above your agreed target on the golden set.
- No unfaithful answers on high-risk topics (pricing, legal, HR policy).
- Correct refusal on all out-of-scope questions.
Run the full evaluation automatically in CI whenever you change the prompt, the model, the chunking or the embedding model. A model upgrade that “feels better” can quietly break a third of your answers.
Step 6: Keep Evaluating in Production
Your test set is never finished. In production:
- Log questions, retrieved chunks and answers (with privacy controls in place).
- Collect feedback with a simple thumbs up or down.
- Review failures weekly and add them to the golden set.
- Watch cost and latency next to quality; a slower, pricier answer is not always a better one.
- Re-index when source documents change, and test that the new version is retrieved.
A Simple RAG Evaluation Checklist
- ☐ 50 to 100 golden questions with expected answers and source documents
- ☐ Retrieval scored separately (hit rate, MRR, context precision)
- ☐ Faithfulness, correctness and refusal scored for answers
- ☐ LLM judge calibrated against human scores
- ☐ Release thresholds agreed with the business owner
- ☐ Evaluation runs automatically on every pipeline change
- ☐ Production feedback loop feeding new test cases
Common Mistakes to Avoid
- Testing only happy-path questions. Real users are messier than your demo.
- Tuning prompts before fixing retrieval. You are polishing answers built on the wrong context.
- Trusting one overall score. A single average hides the few dangerous answers that matter most.
- Never re-testing after a model upgrade. Treat model changes like code changes.
How PITC Solutions Can Help
PITC Solutions helps teams move RAG and LLM projects from impressive demo to dependable product. Through our AI and cloud consulting, we can review your pipeline, build a golden test set with your experts, and set up automated evaluation on AWS or your existing stack. If your team wants to build these skills in-house, our corporate GenAI and RAG training covers retrieval design, evaluation and production monitoring with hands-on labs.
Not sure if your LLM app is ready for real users? Book a free RAG review and we will tell you where it is likely to fail, and how to fix it.
For more background, read where AI actually helps in business or browse more AI and cloud insights.
