A retrieval-augmented prototype is a weekend. Point an embedding model at a folder of documents, drop the nearest neighbours into a prompt, and you have something that demos beautifully. The demo is not the hard part, and treating it as though it were is why so many of these projects stall three months later with a system nobody in the business trusts.

The failures are surprisingly consistent. Almost none of them are generation failures.

The model is rarely the problem

When a RAG system gives a wrong answer, the instinct is to blame the model, then to reach for a bigger one. It usually does not help, because in most cases the model answered the question correctly given the context it was handed. The context was simply wrong, incomplete, or contained the right passage buried under four irrelevant ones.

A useful diagnostic: take twenty failed answers and, for each, check whether the passage containing the correct answer was in the retrieved context at all. In our experience the answer is no far more often than teams expect. That is a retrieval problem, and swapping the generation model will not touch it.

Chunking destroys meaning before you ever embed it

The default in most tutorials is to split documents every 500 or 1000 characters with some overlap. This is fast, and it is wrong for almost every real corpus.

Splitting on a fixed character count cuts tables in half, separates a heading from the paragraph it governs, and severs a clause from the definition it depends on. The embedding for a fragment that begins mid-sentence encodes something close to noise.

What tends to work better:

  • Split on structure, not length. Use the document's own headings, sections, list items and table boundaries. A chunk should be a thing a person would recognize as a unit.
  • Carry ancestry into the chunk. Prefix each chunk with its document title and heading path. A paragraph reading "this does not apply to contractors" is meaningless in isolation and precise under the heading "Section 4.2 Overtime eligibility".
  • Keep tables whole, and store a text rendering alongside them. Chunking a table row-wise is a reliable way to produce confidently wrong numeric answers.

Pure vector search misses the things people actually search for

Embeddings are good at conceptual similarity and bad at exact tokens. Product codes, error numbers, policy identifiers, surnames, version strings: these are precisely the terms your users type, and they are exactly what dense retrieval handles worst. "ERR-4021" and "ERR-4012" sit close together in embedding space and mean entirely different things.

Hybrid retrieval fixes most of this. Run BM25 or another lexical search alongside the vector search, fuse the two result sets with reciprocal rank fusion, then re-rank the fused set with a cross-encoder. The re-ranking step is the one teams most often skip and the one that usually produces the largest single jump in answer quality, because a cross-encoder scores the query and passage together rather than comparing two independently computed vectors.

Retrieve broadly, perhaps thirty to fifty candidates. Re-rank aggressively. Pass five to eight to the model.

Nobody knows whether it got better

Ask a team how accurate their RAG system is and you will often get an anecdote. This is the deepest problem on the list, because without measurement every subsequent change is a guess and every regression ships silently.

The minimum viable evaluation setup is smaller than people fear:

  1. A golden set. One hundred to two hundred real questions from real users, each with a correct answer and the document that supports it. Have subject matter experts write them. This is a week of somebody's time and it is the highest-return week in the project.
  2. Retrieval metrics, separately from answer metrics. Recall at k tells you whether the right passage was even available. Measure it on its own, because it isolates the failure mode described at the top of this piece.
  3. Answer scoring. Faithfulness (is every claim supported by the retrieved context) and correctness (does it match the reference). An LLM judge is acceptable here provided you validate it against human labels on a sample first, and provided you know its biases: it favours longer answers, it favours answers that echo the question's phrasing, and it is unreliable at judging arithmetic.
  4. A regression gate in CI. Any change to chunking, embeddings, prompt or model runs the suite. A drop beyond a threshold fails the build. Without this, quality erodes one well-intentioned tweak at a time.

Fresh questions against stale indexes

Documents change. Policies get superseded, prices move, procedures are revised. An index built once and never refreshed will confidently cite last year's policy, and the answer will look exactly as authoritative as a correct one.

Treat the index as a pipeline with the same rigour as any other data pipeline: incremental updates on document change, deletion propagated rather than orphaned, a freshness timestamp attached to every chunk, and the source document's own effective date surfaced in the answer so a person can spot staleness themselves.

The answer is right and nobody believes it

The last failure is social rather than technical. A system that produces a paragraph of prose with no indication of where it came from asks the user for trust it has not earned. The first time it is wrong, they stop using it, and they tell colleagues.

Cite at passage level, not document level, and link to the exact section. Show the retrieved context on request. Most importantly, let the system decline: if retrieval confidence is low, saying "I could not find this in the documents I have" is vastly better than a fluent guess. Teams resist building the refusal path because it makes the system look less capable in a demo. It is what makes people trust it in month six.

What this means in practice

If you are staring at a RAG prototype that impresses in a demo and disappoints in use, work the list in this order:

  1. Build the golden set and measure retrieval recall separately. You cannot fix what you have not localised.
  2. Fix chunking to follow document structure and carry heading context.
  3. Add hybrid retrieval and a re-ranker.
  4. Add citations and a refusal path.
  5. Only then consider a larger generation model.

The order matters. Most teams start at step five, because it is the one that requires no new understanding of their own data.