Why most RAG systems fail on real documents

The demo always goes well. Someone loads twenty clean PDFs into a retrieval-augmented generation system (RAG for short), asks it five questions, and it answers all of them with neat citations. Everyone is impressed.

Three months later the same system is connected to the real document store, and people have quietly stopped using it. The answers are vague, or confidently wrong, or they quote a policy that was replaced two years ago.

The language model is rarely the problem. A RAG system has two jobs: find the right passages, then write an answer from them. Almost all the failures happen in the first job. If the search hands the model the wrong three paragraphs, no model in the world will give you the right answer.

Here's where it usually goes wrong.

The documents are messier than anyone admits

Real company knowledge doesn't live in twenty tidy PDFs. It lives in scanned contracts with a stamp across the text, Word files with tracked changes nobody ever accepted, Excel sheets where the important part is a comment in cell F14, and folders called Archiv_alt_NICHT_LÖSCHEN.

And there are versions. Spesenreglement_2022.pdf, Spesenreglement_2024_final.pdf, Spesenreglement_2024_final_v2.pdf. To a search system these are three equally valid sources. Ask about meal allowances and it may happily blend the old and the new rules into an answer that matches neither.

None of this is an AI problem, and a better model won't fix it. Someone has to decide which documents are current, mark the rest as archived, and get the scans through proper text recognition. It's unglamorous work, and it does more for answer quality than anything else on this list.

Chunking cuts in the wrong places

Before documents can be searched, they're split into chunks. The default in most tutorials is a fixed size, say 500 words with a bit of overlap. That's fine for blog posts. For contracts and policies it's a disaster.

A fixed-size split will cheerfully cut a table in half, so the column headers end up in one chunk and the numbers in another. It separates a clause from the sentence that says "unless otherwise agreed in Annex B". It drops the section heading, so a chunk that says "the period is 90 days" no longer says which period.

Better chunking follows the structure of the document: sections, clauses, table rows with their headers repeated. Each chunk should also carry its context (document title, section, date) so it still makes sense on its own.

People don't ask in the words the documents use

Vector search is good at meaning and less good at exact terms. Someone asks "how much notice do we give suppliers?" and the relevant contract says Kündigungsfrist. Someone searches for an article number, and the closest matches by meaning have nothing to do with it.

In a Swiss company it gets worse, because the question might be in English, the policy in German and the supplier contract in French.

What helps is combining methods. Keyword search catches exact codes and names, vector search catches meaning, and a re-ranking step on top picks the passages that actually answer the question. Rewriting the question before searching (spelling out abbreviations, adding the term in the other language) helps more than people expect.

Permissions get added at the end

This one isn't about quality, but it's the one that stops projects. If the assistant can find a document, it can quote it. If HR files or board minutes are in the index, sooner or later someone asks an innocent question and gets an answer from them.

Access rights have to be enforced at search time, using the permissions the document already has in SharePoint or on the file server. A line in the prompt saying "don't reveal confidential information" isn't access control.

Nobody measures the search

Most teams judge their RAG system by trying a few questions and seeing how the answers feel. That makes it impossible to tell whether a change made things better or worse.

What works is boring. Collect 50 to 100 real questions from the people who'll use the system, and for each one note which document and section holds the answer. Then measure two things separately: did the search find the right passage, and, given the right passage, was the answer correct? If the search is the weak point (it usually is), you now know exactly where to work.

It also helps to let the system say "I couldn't find that". An honest non-answer does far less damage than a fluent wrong one.

Where we'd start

If we were handed a RAG project that isn't working, the first week wouldn't involve the model at all. We'd sort out which documents are current, fix the text extraction on the worst files, rebuild the chunking around document structure and write the test questions. Only then would we look at prompts and models. That order sounds backwards, but it's where most of the answer quality comes from.