What Is RAG? Retrieval-Augmented Generation

Why plain LLMs hallucinate

An LLM only knows what was in its training data, and it has no reliable memory of your documents. Ask it about a private file and it will guess — confidently, fluently, and often wrong.

This follows directly from the mechanism. The model always produces the most plausible next token, and it has no built-in way to say "I don't know". When the answer is not in its weights, it produces something statistically smooth rather than nothing. A fluent wrong answer is the expected behaviour, not a malfunction.

The RAG pipeline

Retrieval-augmented generation fixes this by putting the right information into the context window at question time. Three stages:

• Index — your documents are split into chunks and converted into vector embeddings, which are numerical representations that capture meaning.
• Retrieve — when a user asks a question, that question is also embedded, and the system finds the chunks whose vectors are most similar.
• Generate — the LLM receives those chunks as context and answers using them, ideally citing the source.

The model is unchanged. What changed is that the facts are now in front of it, so "most plausible next token" lands on the correct answer.

Where the real difficulty is

The pipeline is easy to describe and genuinely hard to build, because almost all the difficulty lives in the retrieval half rather than the generation half.

Chunking decides what units of meaning exist. Split too small and answers lose context; split too large and the model drowns in irrelevant text. Documents with tables or code need special handling.

Embeddings determine what "similar" means. A general embedding model treats "car" and "automobile" as close, which is good, but it may miss that "refund policy" and "money back" mean the same thing in your domain.

Hybrid search combines vector similarity with plain keyword matching, because exact terms like error codes, product names, and numbers are often missed by pure vector search.

Reranking takes the top candidates from a fast first pass and re-scores them with a more expensive, more accurate model. This reliably improves answer quality and is the single highest-leverage addition to a basic RAG system.

Evaluating a RAG system

This is where most student projects stop early, and it is the part that separates an engineer from a hobbyist. You need to know whether the system is retrieving the right chunks and whether the final answer is faithful to them. Measure those separately, because a wrong answer can be a retrieval failure (the right chunk was never found) or a generation failure (the chunk was found and ignored). The fixes are completely different.

Why students should care

• An answer that cites your own data is trustworthier than one that guesses.
• Almost every company holds knowledge in private documents, so RAG skills map directly to real jobs.
• It is a genuine engineering discipline, not an API wrapper — retrieval quality is where the real thinking happens.

Build one this week

Create a small RAG app over a directory of markdown files: index them, ask questions, and print the retrieved chunks alongside the answer so you can see the failure directly. Once you can read the retrieval and judge whether the evidence was sufficient, you understand most production RAG systems.

Go deeper

• RAG is a core module of the Agentic AI Engineer program — see the full curriculum.
• Understand the hallucination problem first: how LLMs really work.
• Decide when retrieval is the wrong tool: RAG vs fine-tuning.