A few weeks ago I watched a demo fall apart in real time. A team had spent a quarter building a RAG assistant over their support docs, and the internal RAG evaluation report showed 94% accuracy on a hand-labeled test set. Good enough to demo in front of the people with budgets. Ten minutes in, a stakeholder asked about a pricing change that had shipped that Monday. The assistant retrieved the old page, cited it, and answered with total confidence. Wrong number, clean citation, in front of everyone who signs the checks.
Nobody asked about the 94% after that.
This gap between what your RAG evaluation measures and what your users do is the most expensive blind spot in RAG right now. Leaderboard scores keep climbing. Enterprise deployments keep face-planting on questions a plain search engine would handle. When teams dig into the failures, the model is rarely the culprit. The evaluation is.
The specifics are worth naming. Most RAG evaluation setups rest on three shaky supports: a static golden dataset built once at kickoff, an LLM judge that scores answers by general vibe, and public benchmark numbers that get quoted in meetings and never reproduced. Each fails in a predictable way. Each has a fix.
This post covers seven of them. Some change what you test, some change how you score it, and two move testing closer to production. All are tool-agnostic, so LangChain, LlamaIndex, or a homegrown pipeline makes no difference. The cost is a scheduler, some logging, and about an hour of labeling per week.
One fair warning before we start. Honest evals will make your numbers worse before they make your system better. That 94% was always a story you were telling yourself. The fixes below tell you what’s real.
Why benchmark scores stopped predicting anything
The gap between test scores and production behavior is not anecdote anymore. It keeps showing up in careful measurement.
In July 2025, Apple engineers published an evaluation of deep research systems, the agentic cousins of RAG that search, read, and cite across the open web. Across the systems tested, 30 to 41% of generated statements were unsupported by the sources cited. These are systems that ace standard evals. The failures live in the citations nobody scores.
Meta’s CRAG benchmark ran into the same wall from another direction. Its 4,400 questions sit behind a mock API that lets the underlying facts change over time, which is exactly what your documentation does. Systems that looked sharp on static QA sets dropped hard once the right answer depended on recently changed or conflicting sources.
Then there is the leaderboard problem. The Leaderboard Illusion, a 2025 paper from researchers at Cohere and several universities, showed that top labs privately test many model variants on Chatbot Arena and only publish the best performer. Leaderboards partly measure release strategy, not raw capability.
Your internal RAG evaluation inherits all three problems because it uses the same ingredients. Stale data, lenient judges, contaminated questions. The fixes are just as specific.
Three ways your RAG evaluation lies to you
Your golden dataset is rotting
Docs change. The index gets rebuilt, and products ship on Mondays. A test set built at kickoff measures kickoff performance forever. A question about the old pricing tier now has two “correct” answers, and only one of them is still true.
There is a second rot, and it is quieter. If your test questions came from tutorials, public FAQs, or model-generated examples, the model may already know the answers from training. Retrieval then looks flawless while going untested. You’re scoring memory, not the pipeline.
Your judge has known biases
The MT-Bench paper that made LLM-as-judge popular, from Zheng and colleagues in 2023, documented the problems on arrival. Judges prefer the first answer presented. Judges prefer longer answers. A 2024 follow-up, LLM Evaluators Recognize and Favor Their Own Generations, showed evaluators can pick their own model family’s outputs out of a lineup.
If your judge and your generator come from the same vendor, family ties inflate your scores. Nobody at the vendor did this on purpose. It still costs you.
Public benchmarks leak
Contamination is old news for model benchmarks. Enterprise RAG has its own version. Benchmark questions are generic, your documents are not, and a model that scored well on public QA has proven nothing about your 40,000-page wiki or the ticket comments your users search.
Worse, generic questions skip the hard parts: conflicting sources, stale pages, half-migrated docs. Those are the questions that break demos. They never show up in the test set.
Seven fixes for honest RAG evaluation
1. Split the score by component
A single accuracy number for the whole pipeline hides two systems. Split your RAG evaluation in two. Grade retrieval on its own with recall@k against known relevant chunks, plus MRR or nDCG for ranking quality. Grade generation on its own with faithfulness, meaning is the answer supported by the retrieved context, and answer correctness against a reference answer. When the demo fails, you’ll know within minutes whether to fix the retriever or the prompt. RAGAS and DeepEval both ship these metrics, free and open source.
2. Build a golden set that ages
Treat the test set as a living artifact with a changelog. Add 10 to 20 real queries from production logs every week and label them. Retire questions that reference deprecated docs, and date every entry. A set that changes weekly resists the slow overfitting a frozen set invites. The pricing question that killed that demo would have been caught here, because the set would have noticed the pricing page changed.
3. Judge the judge
Keep the LLM judge, but audit it. For pairwise tests, swap answer order and rerun. If preferences flip, position bias is live in your eval. Spot-check 20% of judgments by hand each week and log every disagreement, because those disagreement logs teach you more than the scores do. Before trusting a judge at scale, measure its accuracy against a human-labeled sample. It’s an hour of setup, and it recalibrates everything downstream.
4. Score unsupported claims directly
Answer accuracy can look fine while the citations are fiction. Attribution metrics close that hole. For each statement in an answer, check whether a cited source supports it. The ALCE benchmark formalized citation precision and citation recall for exactly this purpose, and Apple’s 30 to 41% unsupported-statement finding is exactly what this metric catches. If your system cites sources, you measure this. No exceptions.
5. Test refusal behavior
Feed the system questions your docs cannot answer and score what happens next. Models tend to answer confidently instead of abstaining, a failure the FaithEval benchmark surfaced across leading models. Build “no answer possible” cases into every test set, roughly 10 to 15% of it, and treat abstention as a first-class outcome with its own metric. A system that says “I don’t know” beats one that’s 94% right and never admits doubt. Users forgive ignorance. They don’t forgive confident fabrication.
6. Hold out data nobody on your team touched
The cheapest upgrade to eval integrity is also the simplest: build test questions from real user queries, after launch, labeled by someone who didn’t build the pipeline. Dev-written tests test the developer’s assumptions. Users don’t share those assumptions. If pre-launch testing is unavoidable, freeze a holdout set no prompt engineer has seen, keep it out of every tuning loop, and rerun it on every index change. Contamination you create yourself is still contamination.
7. Move part of the eval into production
Offline tests are snapshots. Production keeps moving. Sample real traffic weekly and run faithfulness checks asynchronously, off the user path. Log judge scores next to explicit feedback, thumbs and escalations. Alert on drift: citation precision falling, abstention climbing, retrieval latency creeping up. Langfuse, Arize Phoenix, and LangSmith all support this loop today. A weekly drift review catches in days what a quarterly eval catches in quarters. That’s RAG evaluation on live traffic, not last quarter’s questions.
A loop a small team can run
The fixes above sound like a lot of work. Assembled, they come to about four hours a week for a team of one or two.
- Monday: sample 50 queries from last week’s logs, label 15 of them
- Tuesday: run component metrics on all 50, full-pipeline metrics on the labeled 15
- Wednesday: review judge disagreements and fix the pattern, not the instance
- Thursday: add new edge cases from support tickets, plus one unanswerable case
- Friday: update the dashboard and write three lines of changelog for the team
The cadence matters more than the tooling. What you want is a habit: real queries in, honest scores out, one fix per week. Six months of that beats any one-time RAG evaluation project, and it compounds.
What the honest number buys you
To recap: static golden sets rot, LLM judges carry documented biases, and public benchmarks leak. Those three inflate your score while telling you nothing about question 10,001, asked against a doc that changed on Monday. The seven fixes move RAG evaluation toward reality with component metrics, aging golden sets, audited judges, attribution checks, refusal tests, untouched holdouts, and production sampling.
Start with fixes 1 and 4 this week. They’re the cheapest and they surface the most. One afternoon of wiring, and you’ll know whether your citations are real.
That demo team from the intro now samples every Monday. Their score dropped to 81% in the first month, which was the first honest number they’d ever produced. Nobody has asked about the 94% since. The 81 comes with a changelog.
If you’re building RAG at a small company and want this process written down properly, that’s our lane. Rag About It produces the implementation guides and documentation that keep teams honest about their systems. Subscribe for weekly technical deep dives, or reply with your RAG evaluation setup and we’ll cover the interesting parts.
