IBM ran an internal RAG system against real questions about its own product documentation. In July, 40% of valid questions had no article in the corpus capable of answering them at all, not a retrieval problem, a coverage problem. Writers spent the next several months closing those gaps. By December, an answering article existed for 75% of valid questions.
The team expected accuracy to follow. Instead they found a second, different problem: search was failing to find the relevant article in 47% of cases where the article existed. Fixing the content hadn't fixed retrieval, because they were never the same failure. IBM's evaluation had asked three separate questions all along: does an answer exist, does search find it, and is the final answer good. Collapsing those into one "the model got it wrong" score would have sent someone chasing a fix in the wrong layer for months.
Separate the failure layers
That's the pattern behind most disappointing pilots. The model gets blamed first, and sometimes that's fair. More often it never received the right evidence: the current policy was buried under an older version, a table's rows and headers got scrambled in extraction, or retrieval returned three near-duplicates and displaced the one document that actually answered the question. A bigger model can't fix any of that. It can only get better at working with whatever it's handed.
OCR compounds the problem in a specific, measurable way: research comparing OCR-derived text against human-verified content found the OCR version degraded both retrieval and generation, meaning a document can look fine to a person skimming it and still be quietly unusable to the retriever underneath. And staleness has its own failure mode, independent of coverage or extraction: one large evaluation found that adding outdated context could reduce answer performance by at least 20%, even when the current, correct information was sitting right next to it in the same context window.
The pre-pilot questions
- Does the collection contain the authoritative answer, or merely documents on the same topic?
- Can the system distinguish current, historical, and superseded versions?
- Does the chunk preserve the structure that gives the passage its meaning?
- Can retrieval quality be measured separately from answer quality?
IBM's own fix tells you how to use that list: they didn't just re-run the pilot and hope. They built an evaluation interface that recorded, for every question, whether it was valid, whether an answering article existed, whether search found it, and whether the final answer was good, four separate signals instead of one. That's the difference between a pilot that tells you what to fix next and one that just tells you it didn't work.