Your Hallucination Is a Retrieval Problem

In January 2024, a team led by Scott Barnett at Deakin University published the now-standard taxonomy of where retrieval-augmented-generation systems fail in production. The paper (“Seven Failure Points When Engineering a Retrieval Augmented Generation System,” presented at CAIN 2024 (arXiv:2401.05856)) was drawn from three real deployments across research, education, and biomedical domains. The headline finding, two years before it became conventional wisdom in the industry: every one of the seven failure points they cataloged happened before the generator produced its first token.

Missing content. Missed top-K. Not in context. Not extracted. Wrong format. Incorrect specificity. Incomplete answer. None of these are model failures. All of them are misdiagnosed as model failures by production teams the moment a confident wrong answer reaches a user, because the user-visible artifact is always the same: fluent prose, confidently asserted, factually off. The model wrote it. So the model gets blamed.

The two years of follow-on research since Barnett have only reinforced the picture. The ASTUTE RAG paper (Google + USC, ACL 2025, arXiv:2410.07176) reports that under realistic conditions roughly 70% of retrieved passages don’t directly contain the true answer, even when using a real-world search engine over the open web as the corpus. Stable-RAG (early 2026, arXiv:2601.02993) demonstrates a separate generation-side failure mode: even when the gold document is in the retrieved set, simply reordering the passages causes the generator to produce a different answer, sometimes ignoring the gold document entirely. The convergent finding from these and related studies is consistent: the majority of production RAG failures are retrieval failures, not generation failures. The model is doing its job. The retriever is not.

The next sections work backward from that observation. First, why the steelman version of “it’s the model” doesn’t survive contact with production data. Then a five-question debug flowchart you can apply to your own logs on Monday. Then the architectural point that determines whether any of this is debuggable in the first place.

The strongest version of “it’s the model”

Before taking the model-blaming response apart, the steelman deserves a fair hearing. There is a real version of the argument that a smarter model would have caught the bad retrieval and refused to answer, and it’s worth engaging with honestly.

Frontier models in 2026 are better at admitting “I don’t know.” They are more cautious about extrapolating. They are more likely to flag low-evidence answers as uncertain. Some of them, when given a retrieval result that doesn’t contain the answer to the user’s question, will say so explicitly rather than confabulating from partial context. The Anthropic and OpenAI teams have both made measurable progress on this in the last eighteen months. I am not going to pretend that part isn’t real.

So the steelman is this: yes, retrieval is broken, but a sufficiently smart generator would notice when the context doesn’t support an answer, and refuse to make one up. Upgrade the model, get a smarter generator, the retrieval problem becomes self-correcting.

There’s partial truth in that. The part that’s true is worth getting right before unpacking why the rest doesn’t survive contact with production data.

The partial truth is real. A 2026 frontier model, on a clean retrieval miss (the retrieved passages have nothing to do with the question) is meaningfully more likely to abstain than a 2024 model was. That capability is improving and will keep improving. If you held everything else constant and only upgraded the model, hallucination rates would come down marginally. The published evidence on prompt and model-only improvements past a reasonable baseline puts them in the single-digit-percentage range, and that range is real, and you get it.

The part that doesn’t survive is that production retrieval failures rarely look like clean misses. The Stable-RAG paper from earlier this year reproduced this in controlled experiments: even when the correct document was in the retrieved set, simply reordering the passages caused the generator to produce a different answer. In a non-trivial fraction of cases, the model ignored the gold document even when it was present. A separate paper, ASTUTE RAG, reported that under realistic conditions roughly 70% of retrieved passages didn’t directly contain the true answer. So the production environment your “smarter model” is operating in is not “no relevant context, please abstain.” It is “partially relevant context, sometimes contradictory, often misordered, with the right document buried under three plausible-looking near-misses.” A smarter model is better at that than a dumber model, marginally. But the cliff is not in generation. The cliff is in everything upstream that determines what the generator is looking at.

The marginal-vs-cliff version: yes, generator improvement helps at the margins, but it doesn’t help the way upgrading retrieval helps. The marginal generator improvement is a few percent. The retrieval improvement available to most production systems is an order of magnitude. The two are not in the same league.

Where the failure actually lives

In 2024 a team led by Barnett published what is now the standard taxonomy of where RAG systems fail in production, drawn from three real deployments. They identified seven failure points. Every one of them happens before the generator sees its first token. Missing content: the answer simply isn’t in the corpus, but the system answers anyway. Missed top-K: the right document exists but the retriever didn’t rank it high enough. Not in context: the retrieval was right but the context window got truncated. Not extracted: the chunk contained the answer but the generator didn’t surface it. Wrong format. Incorrect specificity. Incomplete answer.

None of those are model failures. All of them feel like model failures to a user.

The reason production teams misdiagnose this so consistently is that the user-visible artifact is always the same: a confident wrong answer in natural language. The model wrote it. So the model is to blame. The retriever, the chunker, the embedding model, the reranker, the context window assembler: those layers are invisible to the user and almost always invisible to the team trying to debug the failure. If you don’t log what the retriever returned, every failure looks like a generation failure. If you do log it, the picture inverts almost overnight.

A debug flowchart you can run on Monday

The diagnostic I now run on any production RAG system that’s producing wrong answers has five questions, in order. Each one isolates a different failure mode. Each one has a different fix.

Question one. Was the right document in the corpus at all? If no, you have a content gap. No retrieval improvement will help. The fix is upstream: ingestion, coverage, deduplication of obsolete versions. Surprisingly often the right document was removed during a content cleanup, or never indexed in the first place, or exists only as an attachment the ingestion pipeline silently skipped.

Question two. Did the retriever rank it in the top-K? If no, you have a recall failure. Almost always, the fix is hybrid search. Pure vector retrieval misses on vocabulary mismatch: the user asked about “compensation” and the document said “salary,” or the user typed an exact identifier like “Section 4.2.1” that an embedding model fuzzes into something semantically close but lexically wrong. Adding BM25 alongside the vector search and fusing the rankings catches the misses that either method alone would lose. This is the single highest-ROI change you can make to a naive RAG pipeline in production. I have watched it cut hallucination rates in half by itself.

Question three. Did the reranker keep it? If you’re not using a reranker, this is moot, but you should be. A cross-encoder reranker that re-scores the top-50 retrieved chunks against the original query, then passes the top-5 to the generator, catches relevance signals that initial embedding similarity misses. If you are using one and the right document still got dropped, you have a rerank tuning problem, and the fix is usually either a domain-specific reranker or a different scoring threshold.

Question four. Was it actually in the context window the generator saw? Long context windows in 2026 are large enough that this question feels obsolete, but it isn’t. The “lost in the middle” effect, documented in research that has now been replicated across every frontier model, shows that information placed in the middle of a long context window is systematically underweighted by the generator. If your retrieved chunks are ordered by retrieval score and the gold document happened to rank fourth, it lands in the dead zone of the context window where the model is least likely to attend to it. The fix is reordering: put the highest-relevance chunks at the end of the context (closest to the question), not the middle.

Question five. Did the generator actually use it? This is the faithfulness question, and it’s the one that requires a separate evaluation pipeline to answer at all. The standard approach is RAGAS-style faithfulness scoring: a separate model reads the answer and the retrieved context, then judges whether each claim in the answer is supported by the context. If a claim isn’t supported, you have a faithfulness failure: the generator ignored or misused the context it was given. This is the closest thing to a real “model hallucination,” and it is the only one where upgrading the model meaningfully helps. Faithfulness failures are a minority of what users perceive as hallucinations. The much larger category is upstream of the generator entirely.

Five questions. Five fixes. None of them are “upgrade the model.”

What this requires architecturally

Honest accounting: the debug flowchart is only runnable if you have the data. None of the five questions can be answered from a black-box production system. To answer question one, you need to log the corpus state at query time. To answer question two, you need to log every retrieved chunk and its score. To answer question three, you need to log the rerank decisions. To answer question four, you need to log the assembled context window the generator actually received. To answer question five, you need a faithfulness eval pipeline running over your production traffic.

Almost no production RAG system I’ve ever audited had any of this when I arrived. Most of them had the bare minimum: prompt-in, completion-out, maybe a Datadog metric for latency. The retrieval layer was a black box, the reranker was a separate service nobody owned, the context assembly was happening inside a vendor SDK with no logging hook, and the faithfulness scoring did not exist. Of course they thought it was a model problem. They had no way to know it wasn’t.

The architectural lesson (and I’ll keep saying it until it stops being interesting) is that the gateway is where this instrumentation has to live. Not as an afterthought, not as a separate observability tool bolted on six months after launch, but as the way every call flows from the start. If every model call already passes through one layer in your stack, every layer of the retrieval pipeline can be logged there, and the debug flowchart above stops being a research project and starts being a query. That is what AOCore is for, on the build-it-yourself path. The gateway captures the retrieved passages, the rerank scores, the assembled prompt, the model used, the completion, the faithfulness score, and the user feedback, all in one place. Most teams trying to debug hallucinations are running blind. The gateway is the difference between debugging blind and debugging from the data.

For the organizations that aren’t going to staff a retrieval engineering team to learn these lessons the hard way, the same architecture shows up as a product. AODex ships RAG as a finished feature: knowledge bases that users upload documents into, retrieval pipelines that handle the chunking and embedding and reranking decisions, source citations surfaced to the user on every answer, and (through the AOSentry governance layer inside AOCore) the same audit trail that makes the debug flowchart possible. Buyers don’t run the five-question diagnostic on AODex because the failures it would catch have been engineered out before the customer touches the product. That is, in the end, what “buy versus build” actually means in this category. Either you do the retrieval engineering, or you buy a product where it has already been done.

What you cannot do (what the team I opened with was about to do) is keep upgrading the model and hoping the generator gets smart enough to compensate for a retrieval layer that nobody is measuring. The 3% you’d get from a frontier model upgrade is real. The 30% you’d get from fixing retrieval is sitting on the table, and the only reason most teams aren’t picking it up is that nobody told them where to look.

Now you know where to look.