What Is RAG? How LLMs Use External Documents

RAG (Retrieval-Augmented Generation) is a technique where an LLM first retrieves relevant content from external documents before generating an answer, then answers using that content as reference. An LLM only knows information up to its training cutoff; RAG lets it incorporate information published after training, or information it was never trained on in the first place — like internal company documents — into its answers.

Why it’s needed

An LLM doesn’t know what isn’t in its training data — and rather than admitting it doesn’t know, it can exhibit “hallucination,” confidently making something up instead. RAG substantially reduces this problem, because the answer is grounded in documents actually retrieved rather than relying solely on the LLM’s memory.

How it works, step by step

1. Prepare documents: split reference documents (internal wikis, manuals, recent news, etc.) into small chunks.
2. Embedding: convert each chunk into a numeric vector a computer can use to compute semantic similarity, and store it in a vector database.
3. Retrieval: when a user asks a question, convert the question into a vector the same way and find the stored chunks closest to it in meaning.
4. Generation: pass the retrieved chunks to the LLM along with the question, and have it write an answer using that content as reference.

Steps 2-3 make up the “Retrieval” half, and step 4 is the LLM’s normal “Generation” step — combining the two is where the name RAG comes from.

Where it’s used

• Internal-document chatbots (answering using the latest policies and manuals)
• Services dealing with constantly-changing information like breaking news or market prices
• An AI Agent finding and using information it needs mid-task — Anthropic describes retrieval, alongside tools and memory, as one of the basic building blocks of an “augmented LLM.”
• Local tools like LM Studio use a similar approach internally (chunking + embedding search) when you attach a document.

Difference from fine-tuning

Fine-tuning is a concept RAG is frequently compared with. Both are ways to adapt an LLM for a specific purpose, but the approaches differ — a detailed comparison is covered in RAG vs Fine-tuning.

Limitations and caveats

• Inaccurate retrieval leads to inaccurate answers — RAG “reduces” hallucination, it doesn’t eliminate it entirely.
• Quality varies significantly with design details like how finely documents are chunked and which embedding model is used.
• Retrieving more documents means passing more context to the LLM, which can affect cost and speed.

Conclusion

RAG is a structure that lets an LLM “look up what it doesn’t know and answer using that.” It’s one of the most widely used ways to work around an LLM’s training-cutoff limitation, used everywhere from internal-document chatbots to information-gathering for AI Agents.