Gains:
- Defining metrics that measure retention and generation quality separately
- Set up a gold question set and run automatic evaluation with LLM-as-judge
- Maintaining quality through feedback, monitoring and regression testing in production
The sentence "I installed the assistant, it seems to be working fine" is not an engineering statement. RAG systems break down silently: a new document type fools the retrieval, a prompt change reduces accuracy, the index becomes stale. The only way to realize this is to measure. In this unit, we cover how to measure RAG quality (retrieval and generation separately), automatic evaluation (LLM-as-judge) and maintain quality in production (monitoring, regression). "You can't improve what you don't measure" is the motto of this unit.
Measure Two Separate Things
RAG has two legs and must be measured separately because the problem may be in either of them:
- Retrieval quality: Did the correct part arrive?
- Generation quality: Was the correct answer produced from the incoming piece?
If the answer is bad, you need to know which leg is bad first. If the right part never arrives, even the best prompt can't save (retrieval problem). If the correct part arrived but the model misread it, improving the retrieval is futile (generation problem).
Retrieval Metrics
Retrieval is a sorting/access problem; measured by classical information retrieval metrics. For this you must have the golden cluster: the knowledge of which piece is "correct" for each question.
metric
What measures
Simple definition
Recall@k
Is the correct piece in the top k?
Correct part capture rate
precision@k
How many of the k pieces returned are relevant?
Cleaning of what is brought
MRR (Mean Reciprocal Rank)
In which order is the correct piece?
Rewards being in the top ranks
Hit rate
Did at least one correct piece arrive?
The most basic measure of success
Practical comment: If Recall@k is low, chunking or search strategy (hybrid, k, re-ranking) should be reworked. If precision is low but recall is high, adding re-ranking is a good move.
Generation Metrics
When the correct part arrives, we measure the quality of the response produced by the model. Three basic dimensions:
- Faithfulness: Is each claim in the answer supported by context? Is there any fitting? It is a direct measure of hallucination.
- Answer relevance: Does the answer actually answer the question or is it off-topic?
- Completeness: Has all the relevant information in the context been used or is it missing?
These are often scored on a graded basis (e.g. 1-5), rather than binary like “true/false.”
Tip: Track Faithfulness as a separate metric. If faithfulness decreases as accuracy decreases, the problem is generation; If faithfulness is high but the answer is wrong, the problem is the wrong piece (retrieval). Together, these two metrics are a compass that shows the location of the fault.
Establishing a Golden Question Set
Each measurement requires a golden set / evaluation dataset: realistic questions + expected correct answers + correct source pieces. Starting with 30-50 well-chosen questions is better than 500 random questions. Include the following in the set: frequently asked real questions, known difficult questions, trap questions with no answers (one should say "I don't know"), questions with contradictory sources.
# Golden cluster example (conceptual)[ {"question": "How many days of annual leave?", "expected_answer": "14 days for 1-5 years of seniority", "correct_part_id": "two-part-3", "category": "leave"}, {"question": "Where is the company's Mars office?", "expected_answer": "NO_INFORMATION", # trap: I don't know "correct_part_id": null, "category": "trap"}]
LLM-as-Judge: Automatic Assessment
Scoring hundreds of answers by hand is tiring. LLM-as-judge model is when one model scores and justifies another model's answer based on certain criteria. A good judge prompt clearly defines the criteria, gives examples and asks for justification.
# LLM-as-judge prompt (conceptual)You are an impartial evaluator. Evaluate the ANSWER below according to the given CONTEXT and EXPECTED ANSWER. Score (1-5) and justify:- faithfulness: is each claim in the answer supported in context?- accuracy: does the answer match the expected answer?- completeness: is the relevant information complete? Particularly: if the answer contains information not in the context, give faithfulness1 and indicate which claim is fabricated.CONTEXT: {context}EXPECTED: {expected}ANSWER: {answer}Output: {faithfulness, accuracy, completeness, justification}
Caution: LLM-as-judge is not perfect; They may have their own biases (long answer, preferring their own style). Also verify Judge: have some answers scored by both the judge and the human and measure the agreement between them. If Judge is consistent with human scores, you can trust him.
Weak/Strong Rating
Weak ("it was good for me"):
I asked a few questions and the answers seemed good. I've got it live.# Problem: no measurements, regression imperceptible, improvement blind.
Powerful (gold cluster + discrete metrics + automatic judgment + regression):
Golden cluster of 40 questions. With each change, recall@5, faithfulness and accuracy are measured automatically. If the score drops, the change is rolled back. In production, user feedback is collected and added to the set.
Monitoring and Regression in Production
Evaluation is not done once and finished. Three constant practices:
- Regression testing: Auto-run golden cluster on every prompt/retrieval/model change. If the score is reduced, the change is reversed. This prevents "breaking it while trying to make it better".
- Production monitoring: "I couldn't find information" rate in real questions, average delay, cost, user feedback (👍/👎) are monitored. A sudden increase in "I don't know" is often the first sign of an index or retrieval malfunction.
- Feedback loop: The real questions provided by the user 👎 are reviewed and added to the golden pile; Thus, the set becomes richer over time and the blind spots of the system are closed.
Three Mini Cases
Case 1 — Silent regression. A team modified the prompt to "improve" it; General accuracy increased, but faithfulness decreased by 30% in trap questions (the model started to fit more). It wouldn't have been noticed if it weren't for the trap questions in the golden cluster; Regression testing reverted the change.
Case 2 — Straightening the wrong leg. In one assistant the answers were bad; The team worked on the prompt for weeks. When we measured the retrieval metrics, recall@5 was only 48% — the problem was in retrieval, not generation. When hybrid + re-ranking was added, recall increased to 89% and accuracy increased as well.
Case 3 — Production alert. One day, the "I couldn't find information" rate for a support assistant jumped from 6% to 34%. The trackpad warned; The reason was that the indexing job, which runs at night, silently failed and new articles were not uploaded. Without monitoring, incorrect "I don't know" statements would have continued for days.
Common mistakes
- Being satisfied with “it worked well for me”: Without measurement, regression goes unnoticed.
- Not separating retrieval and generation: You will correct the wrong leg and waste time.
- Not asking trap questions: The tendency to make things up does not appear in the golden cluster.
- Not verifying Judge: A biased jury gives false confidence.
- Not monitoring production: Index failure, cost explosion continues silently.
In summary
- In RAG, retrieval and generation quality are measured separately; It must be determined first which leg is damaged.
- recall@k, precision@k, MRR for Retrieval; Faithfulness, suitability, completeness are used for generation.
- Each measurement requires a gold cluster; Put real, difficult, trap and contradictory questions in it.
- LLM-as-judge large sets auto-scores; but the judge himself must be justified against man.
- Regression testing, production monitoring, and a feedback loop maintain quality over time.
Application task
(1) Create a golden set of at least 15 questions for your own assistant: include at least 3 traps (no answers), 3 difficult, 2 contradictory source questions. Write the expected answer and the correct part for each question. (2) Manually compare two different prompt versions with this set; Give each answer 1-5 points for faithfulness and accuracy. (3) Adapt the LLM-as-judge prompt above to your own criteria. (4) Identify 3 metrics you will track in production and for each ask “at what threshold do I alarm?” write the value.
checklist
- [ ] I can measure retention and generation quality with separate metrics.
- [ ] I know what metrics like recall@k, faithfulness mean.
- [ ] I can build a golden cluster that includes real, difficult, trap and contradictory questions.
- [ ] I can set up automatic evaluation and verify the judge with LLM-as-judge.
- [ ] I can operate regression testing, production monitoring and feedback loop.