Alessandro Nigro.

Enterprise RAG: Where It Actually Breaks

Written by Alessandro Nigro, Senior Software Engineer, IT consultant since 2009 · updated

An enterprise RAG system usually breaks before the model does: in retrieving the right passages, in permissions, in stale documents left in the index, and in how the model fills the gaps when retrieval comes back empty. That is why, on a RAG project, I spend more time on retrieval and evaluation than on choosing the model. Here are the four break points I check and how I measure a RAG before real users see it.

Enterprise RAG system: the short answer

RAG (retrieval augmented generation) means that, before answering, the system searches your documents for relevant passages and passes them to the language model with the question. The answer is only as good as three things: retrieval found the right passages, the user was allowed to see them, and the model stuck to what it was given.

Building one that holds up means:

  • preparing documents so each chunk makes sense on its own;
  • enforcing permissions inside the search, before the model sees anything;
  • keeping the index in sync with the documents that are actually in force;
  • evaluating the system on a set of real questions with expected answers, before and after every change.

What retrieval augmented generation does

The term comes from the paper by Lewis and colleagues, presented at NeurIPS 2020, which describes models that "combine pre-trained parametric and non-parametric memory for language generation": what the model learned in training, plus an external store it can search. The same paper noted that, for language models alone, "providing provenance for their decisions and updating their world knowledge remain open research problems." Those are exactly the two reasons a company wants RAG: show where an answer came from, and update knowledge without retraining anything.

In a production application the flow looks like this:

  1. documents are split into chunks and indexed, usually as embeddings in a vector database and often with a keyword index too;
  2. when a question arrives, the system retrieves the closest chunks;
  3. the model receives the question and the chunks and writes the answer, with references.

Any of these steps can fail silently. The final answer always sounds confident.

Break point one: retrieval

If retrieval brings back the wrong chunks, no model can answer well. The most common cause is chunking that strips context. Anthropic's write-up on Contextual Retrieval states the problem directly: "traditional RAG solutions remove context when encoding information". A chunk that says "the term is thirty days" is useless if it does not say which contract it belongs to.

In the same article, Anthropic reports its own tests. Adding a short context to each chunk cut the top-20 retrieval failure rate by 35%. Combined with keyword search (BM25), the reduction reached 49%. With a reranking step, 67%. Those numbers come from their datasets and will differ on yours, but they show where the effort pays off.

Long context has limits too. The Lost in the Middle study (Liu and colleagues, 2023) found that performance is often highest when the relevant information sits at the beginning or end of the input, and degrades significantly when it sits in the middle of a long context. Sending the model fifty chunks "just in case" makes answers worse.

What I check:

  • each chunk carries the document title, section and date;
  • search combines meaning with exact terms (product codes, contract numbers, acronyms);
  • how many chunks actually reach the model, and in what order.

Break point two: permissions

A RAG that indexes everything into one store can show a user a document they could not open in the source system. The OWASP Top 10 for LLM Applications covers this as LLM08:2025 Vector and Embedding Weaknesses and recommends "fine-grained access controls and permission-aware vector and embedding stores", with data partitioned between different classes of users or groups.

In practice the permission filter belongs in the search itself: the system retrieves only from documents that user can read. Filtering the answer afterwards is too late, because the text has already passed through the model.

The content of the documents matters as well. A file from outside can carry instructions aimed at the model. OWASP calls this indirect prompt injection, which happens when an LLM "accepts input from external sources, such as websites or files." In TreZeroUno, the platform I built, text that comes from websites is treated as data and never as instructions. The same rule applies to an enterprise RAG.

Break point three: stale and conflicting documents

A RAG answers from what is in the index, and indexes often hold several versions of the same price list or procedure. If retrieval picks the outdated one, the answer will be precise, well written and wrong.

The fixes are about process first, code second:

  • one source of truth per document type, with a named owner;
  • the index updated when a document changes or is retired, not once a month;
  • date and status (current, superseded) stored with each chunk and used in the search;
  • the date of the cited document shown in the answer, so readers can spot it.

Break point four: the model fills the gaps

When retrieval does not find the answer, the model tends to produce one anyway. This is the most dangerous failure because it is invisible: the sentence is plausible and arrives in the same tone as the correct answers.

The defenses I build in:

  • explicit instructions to answer only from the retrieved passages and to say "I can't find this in the documents" when the information is missing;
  • mandatory citations of the passages used, one click away;
  • an automated check that the answer actually cites at least one retrieved passage;
  • for high-risk questions (contracts, safety, health), a person approves before the answer goes out.

How to evaluate a RAG system before launch

A RAG is measured with an evaluation set: real questions, each with the expected answer and the documents where it is found. I build it with the people who know the work, before writing code, and rerun it on every change.

Keep the measures separate by stage:

What I measureThe question it answers
RetrievalIs the right passage among the top results?
FaithfulnessDoes the answer only state what the retrieved passages say?
CorrectnessDoes the answer match the expected one?
RefusalWhen the information is not there, does the system say so?
PermissionsDoes a user without access ever receive that content?

Separate measures tell you where to act. If retrieval is good and answers are not, the problem is the prompt or the model; if retrieval is poor, a better model will not help. After launch, real user questions and their outcomes become new cases in the set.

When you do not need RAG

If the knowledge base is small, you can give the model all of it. In the Contextual Retrieval article, Anthropic notes that below 200,000 tokens (about 500 pages) you can include the whole knowledge base in the prompt, and that prompt caching makes this faster and cheaper.

You also do not need RAG when the question asks for an exact value from a business system. There, the model should query the database through a controlled tool. I cover that pattern in the note on AI agents in production.

Frequently asked questions

What is RAG in simple terms?

RAG lets a language model look things up before it answers. The system searches your documents for relevant passages and gives them to the model with the question, so the answer is based on current, citable information as well as on what the model learned in training.

Does RAG stop hallucinations?

No. It reduces them when retrieval finds the right passages, but the model can still fill gaps or mix sources. You need instructions for when information is missing, verifiable citations, and an evaluation set that also measures refusals.

What is the difference between RAG and agentic RAG?

In basic RAG the system runs one search and answers. In agentic RAG the model decides which searches to run, in what order, and whether to call other tools such as a database query. It is more flexible, but it costs more and is harder to control, so I use it only when the questions require it.

Should you use RAG or fine-tuning for company documents?

For answering questions on documents that change, usually RAG: you update the index and the answer changes, with its source cited. Fine-tuning teaches a model a style or a format; it is a poor way to keep it current on prices and procedures. The two can be combined.

Sources

Related service

How I design, build and run AI agents in production, with integrations, security, monitoring and indicative prices.