Fundamentals · 24 July 2026
What is RAG? And why search alone is not enough
Every AI chat over your own documents is built on RAG. What is behind it, what it does well, and the kind of question it fails at.
By Aidocr
Key takeaways
- RAG (Retrieval Augmented Generation) answers questions by searching your own documents before it responds.
- It reduces invented answers and enables source references, without training a model of your own.
- RAG reaches its limit on questions that span several sources and on documents that supersede each other.
- For those cases you additionally need a knowledge graph.
Anyone who gives an AI chat their own documents is almost certainly using RAG: Retrieval Augmented Generation, loosely translated as answers with a search step in front. The principle sits behind practically every tool that answers questions about your own files, from document uploads in ChatGPT to internal company assistants.
How RAG works
Three steps. First, a question comes in. Second, the system searches the stored documents for the most relevant passages, matching by meaning rather than exact words. Third, the language model writes an answer from those passages.
The strength lies in the second step. The model does not answer from its training memory but from your documents. That noticeably reduces invented answers, makes source references possible, and is quick to introduce because no model needs to be trained.
Where RAG reaches its limit
RAG treats knowledge as loose blocks of text. One question from building operations makes the problem visible: which maintenance contracts cover the ventilation units in building B? The answer is written in no single paragraph. The asset list lives in the CAFM, the contract is a PDF, the mapping of unit to building sits in a spreadsheet. A similarity search finds passages that mention ventilation. The connection between the sources is invisible to it.
Add to that a peculiarity of construction and real estate: documents supersede each other. A change order replaces a line item in the bill of quantities, a new contract version replaces the old one. To a pure text search, both versions look almost identical. Which one applies today is not a question of similarity but of the relationship between documents.
What this means in practice
RAG is the foundation, and for many questions exactly right: where does it say that? What does this document state? But as soon as questions concern relationships, meaning what belongs to what and what applies today, you additionally need a structure that stores relationships explicitly: a knowledge graph. What that is and how an ontology helps is covered in the next article: What is a knowledge graph?
Frequently asked questions about RAG
- What does RAG mean?
- RAG stands for Retrieval Augmented Generation. The system searches the stored documents for passages relevant to a question, and the language model writes an answer from them with a source reference.
- What is RAG used for?
- For practically every tool that answers questions about your own files, from document uploads in ChatGPT to internal company assistants.
- Where does RAG reach its limits?
- On questions whose answer is spread across several sources, and on documents that supersede each other, such as a change order that replaces a line item. A pure similarity search does not recognise the connection or which version applies today.
- What is the difference between RAG and GraphRAG?
- RAG searches text by meaning. GraphRAG adds a knowledge graph that knows the relationships between things, and gives the model both: evidence and context.