A three-dimensional word map with glowing nodes and connections. Library, Public library and Books form one group; Cooking and Kitchen form another. Music and Travel occupy other areas. An artistic illustration with invented positions, not measured embeddings or an actual map of knowledge.

The previous chapter explored plausible false claims. An LLM can write a convincing answer without a reliable basis for it. An application can therefore search for relevant sources first and give the model the passages it finds. [1]

A question for a library archive

Imagine a library archive containing historical newspaper articles. Someone wants to know when the local public library opened. The application searches the archive for relevant articles and gives the LLM the passages it finds, together with the question. The model is instructed to base its answer on these sources. It can then use information from the archive without having been trained on the entire collection. If no suitable evidence is found, the application should make that clear rather than provide an unsupported answer. [1]

The question and the newspaper articles may use different words. A search for the exact phrase “public library” might miss an article that refers to the “town library”. We therefore also want passages relevant to the question even when they use different words. Text can be converted into a numerical representation that lets software calculate similarity. Searching by this kind of relevance is called semantic search. [2]

What an embedding is

An embedding represents data through numbers that software can use to calculate relationships between the data. In our archive example, the data is the content of texts. A specially trained embedding model processes a question or a passage and produces an ordered list of numbers. The model is trained to give texts with similar content similar numerical patterns. A question about the opening of the “public library” and an article about the inauguration of the “town library” can therefore be matched even though they use different words. The search software compares the numerical lists for the question and the stored passages. Each list is called a vector, and each position corresponds to a dimension. The individual numbers usually have no meaning we can read directly. The comparison depends on how they work together. An embedding does not replace the original text; it provides a numerical representation for the search. [2]

The relationships an embedding model captures well depend on what it was trained to do. A model for individual words is therefore not automatically suitable for searching longer newspaper articles. Our archive needs a model that represents questions and passages so that their vectors can be used to find relevant content. Whether this works reliably with the articles in the archive must be checked by testing the application with specific search questions. [2]

Think of embeddings as positions on a map of a galaxy. Each passage is a star on that map. Texts with similar content should lie close together, even when they use different words. The question also receives a position on the map, and the search software looks for passages nearby. Unlike a spatial map of a galaxy, however, this numerical space has not just three dimensions but often hundreds or thousands. We cannot display it in full, but software can calculate how close the positions are. The galaxy map illustrates these relationships; it is not a literal map of knowledge. [2]

Illustrative word map. Library-related words form a group on the left, cooking-related words on the right, music and travel below. All positions and connections are invented.

This word map illustrates the galaxy analogy. Its positions and connections are invented, not calculated embeddings. It does not search an archive. Rotate the map with the buttons or by dragging with the mouse. On a touchscreen, use two fingers to rotate; one finger still scrolls the page. The buttons also work with Tab and Enter. Nothing moves automatically.

Full description of the word map

The illustration contains 214 points, 18 of them labelled, and 547 connections. The library group contains Library, Public library, Books, Reading, Borrowing, Archive, Newspaper and Article. Another group contains Cooking, Kitchen, Recipe and Food. The travel group contains Train, Bus, Travel and Station; the music group contains Music and Concert.

The groups extend in three spatial directions. Rotating the view reveals depth that a flat image hides. The unlabelled points add density to the illustration; they do not represent particular texts. The lines connect nearby points in this invented arrangement. Neither distance, colour nor a connection expresses a measured similarity, a source reference or a judgement about truth. Real text embeddings usually have far more than three dimensions, and this illustration is not a projection of them.

Similarity is not a verdict on truth. Two conflicting newspaper articles can cover the same event and both be relevant to a question. One article might announce the library's planned opening, while a later one reports that it has been postponed. Embeddings help find relevant passages. They do not check which details actually answer the question or whether the source is reliable.

From document to search result

For the search, longer documents are usually divided into smaller passages. This is called chunking. In our archive, a long newspaper article might cover several topics. If the application produced just one embedding for the whole article, a brief item about the library could be lost among the other topics. Each passage should therefore keep related content together. It must not be so short that important context is lost, such as the connection between an opening date and the library it refers to. The right way to divide the text depends on the material. [3]

The software produces embeddings for individual passages and prepares them for searching. It uses a search index, a data structure that helps locate relevant entries. A vector database can store the vectors and support these similarity searches. Each vector must remain linked to its original text. In the library archive, it should also be clear which newspaper, issue and page a passage came from. The LLM will later need the passages found, not their numerical vectors. The source references then allow the answer to be checked against the original documents. [1]

When someone asks a question, the software also produces an embedding for it and searches the index for similar vectors. For the comparison to be meaningful, the question and the stored passages must be represented in compatible numerical spaces. In our analogy, they must lie on the same map of a galaxy. Embeddings from arbitrary different models cannot simply be compared, even when their lists of numbers have the same length. One common solution is therefore to use the same suitable embedding model for both questions and passages. [2] [3]

In our archive example, the search might return several relevant passages, such as the announcement of the library's opening, the report of its postponement and a later account of the actual inauguration. The search software initially supplies these results. It has not yet formulated an answer or determined which date answers the question. The original texts must be considered in context. In the next step, the LLM receives the selected passages so it can use them to formulate an answer.

What RAG adds

Combining search with subsequent text generation is called RAG, short for Retrieval-Augmented Generation. Retrieval means searching for and fetching relevant content. Augmentation here means that the application puts this content into the prompt alongside the question and instructions. Generation is the LLM's subsequent production of text. RAG therefore refers not to the embedding model or database, but to the whole process in which retrieved information is intended to inform the language model's answer. Retrieval can use embeddings, but also other search methods, such as keyword search. [5] [1] [3]

A · Prepare the archive

From newspaper articles to a search index Split articles into passages. An embedding model creates their vectors. Store the vectors in a search index, linked to the original passages and source references. Newspaper articles Historical documents from the library archive Split the text Related content stays together in individual passages Embedding model Creates a vector for each passage Search index Vectors linked to original text + source reference The original texts are retained. Preparation is not model training.

B · Answer a question

From a question to an answer The same suitable embedding model creates a question vector. Search compares it with the index prepared in A and retrieves original passages. The instructions, question and passages form the prompt for a separate answering LLM. Its answer still needs checking. Question When did the town library open? Embedding model The same suitable model as in A creates the question vector Search the index from A Compare the question vector with the stored vectors Retrieve original passages Selected text + source references Not the numerical vectors Build the prompt Instructions + question + selected original passages Answering LLM A separate model generates text from the prompt Answer With source references Still needs checking
Simplified RAG with embeddings in our hypothetical library archive. The vectors help find passages; the original texts go into the LLM’s prompt. RAG can also use other search methods.
Read the full diagram description

In A, the software prepares the archive before questions are asked. It splits historical newspaper articles into passages that keep related content together. An already trained embedding model creates a numerical vector for each passage. The search index stores these vectors with links to the original texts and their newspaper, issue and page references. The texts remain available. This preparation does not train the model.

In B, someone asks when the town library opened. The same suitable embedding model creates a vector for the question. The search software compares it with the stored vectors in the index from A and retrieves selected original passages with their source references. These might cover the announcement, postponement and actual opening. The vectors help select passages; they do not decide which date answers the question.

The software combines the instructions, question and selected original passages into a prompt. A separate, already trained LLM generates an answer from it. The numerical vectors do not go into the prompt. Neither model is trained by this process. The source references allow the answer to be checked against the articles, but do not guarantee that it is correct.

The embeddings help with search. The original texts of the selected passages go into the prompt, not their numerical vectors. The LLM receives them alongside the question and instructions for the answer. In our library archive, these could be the reports of the planned opening, its postponement and the actual inauguration. The application therefore selects relevant excerpts rather than putting the entire archive into the prompt. These texts take up space in the context window, just like the question and any conversation history. The number of passages that can be included therefore also depends on the available context. [1] [3]

The LLM could now use the retrieved articles to formulate an answer about the public library's opening, distinguishing between the announced date and the actual inauguration. The application can identify the articles used by newspaper, issue and page. Readers can then check the original sources to see whether they support the date given and the other claims. A citation alone does not prove this. Even an answer with source references can misrepresent an article or make a claim that is not in it. [3]

What this does not change in the model

In our library example, two separate models have different jobs. The embedding model produces numerical vectors for the archive texts and the question. The search software uses these vectors to find relevant passages. The language model that answers the question then receives the original texts in its prompt and uses them to formulate an answer. Both models have already been trained. When the embedding model converts new articles into vectors, it applies what it has already learnt; it is not being retrained. Giving the language model passages does not train it either. Unlike training, this process does not change any model parameters. New or corrected articles therefore require updates to the archive and its search data, not automatically another round of model training. [3]

A RAG system can still involve additional training. The embedding model or the language model that produces the answer can be trained further to improve their respective tasks. This adaptation of an already trained model is called fine-tuning. The original RAG paper by Lewis and colleagues also describes a system with fine-tuning. The important distinction is between two processes. Retrieval supplies information as text in the prompt; training changes model parameters. Both can occur within one system, but they are not the same thing. [4]

Persistent memory is different again. The newspaper articles remain available because the application keeps them in files or a database, not because the language model permanently remembers them after reading them in a prompt. RAG allows relevant content to be retrieved again for a later question. Whether the application also stores earlier conversations or personal details and reuses them later depends on its design. “Uses RAG” therefore tells you neither that it remembers you nor that it stores no inputs.

In our library archive, embeddings help find articles about the opening of the public library even when they refer to a “town library”. The embedding model supplies the numerical vectors that the search software uses to compare the question with the passages. The language model that produces the answer then receives the original passages and can use them to distinguish between the announced date and the actual inauguration. RAG connects these steps, making an external collection of sources available for answering a question without having to train that collection into the language model's parameters.

In the next chapter

A clear instruction, additional sources and further training affect different parts of working with an LLM. The next chapter looks at when prompting, RAG or fine-tuning makes sense.

Deeper into the Rabbit Hole

Videos in English

  • Computerphile — A Helping Hand for LLMs (Retrieval Augmented Generation) — In roughly 14 minutes, Mike Pound moves from the RAG workflow to a practical implementation with a small demonstration. The description and chapter list were checked; the YouTube transcript could not be retrieved. The video was included on Mirco's recommendation. The episode is sponsored by Jane Street.
  • IBM Technology — Is RAG Still Needed? Choosing the Best Approach for LLMs — In roughly eleven minutes, Martin Keen compares RAG with putting larger collections of documents directly into the context window. He discusses the cost of retrieval, missed results and when whole documents may be more useful than individual excerpts. The full English transcript was read, not the full video watched. The discussion focuses on vector-based RAG; its portrayal of a vector database as the only possible solution for large collections is not adopted here.

Sources

  1. Kim Martineau, IBM Research, What is retrieval-augmented generation?, 22 August 2023. Explains how relevant passages are retrieved and added to the prompt. Also discusses possible failures in retrieval and text generation.
  2. Dale Markowitz, Google Cloud, Meet AI’s multitool: Vector embeddings, 23 March 2022. Explains numerical representations, similarity comparisons and model-dependent relationships. Its word examples do not transfer directly to models for whole passages.
  3. Ivan Belcic, IBM Think, What is retrieval augmented generation (RAG)?, updated 2 October 2026. Describes chunking, vector-based retrieval, prompt augmentation and the distinction from fine-tuning.
  4. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020. The original paper combines a language model with an external search index and describes fine-tuning the system. Only the abstract page was inspected, not the full paper.
  5. deepset / Haystack, InMemoryBM25Retriever. Describes keyword-based retrieval and explicitly shows a RAG pipeline that adds retrieved documents to the prompt before calling a language model, without an embedding model or vector search. The page content and code example were read; the code was not run.