What is RAG and Why is it Necessary?

Gains:

• Explaining that RAG injects context without changing the weights of the model and works with an 'open book exam' logic
• Comparing RAG with fine-tuning and long context approaches according to cost, timeliness and usage scenario
• Listing the steps of a typical RAG pipeline consisting of indexing and query phases

No matter how powerful a language model (artificial intelligence that understands and produces text; we'll call it model for short from now on) is, it doesn't know the contract your company signed yesterday, your internal wiki (internal knowledge base) page, or the release note published this morning. The model is limited to general knowledge up to the date it was trained; This is called the "education cut-off date". RAG (Retrieval-Augmented Generation) fills exactly this gap: it finds the company documents related to the question, gives it to the model as context (that is, the additional text it will read while producing the answer) and has the answer produced based on this context.

In this unit, we will clearly see what RAG is, when it is preferred over which alternatives, and the steps of a typical RAG pipeline. All subsequent units will deepen the parts of this map one by one.

The Basic Idea of RAG: Open Book Exam

Let's explain RAG in one sentence: "First find the relevant document, then have the model read that document and print the answer accordingly."

The most useful analogy is this: RAG moves the model from a “closed book exam” to an “open book exam.” In the closed book exam, the student answers only from memory; There is a high risk of making up what you don't remember. In the open book exam, the student answers by looking at the source placed in front of him. In RAG, the model no longer answers from its own memory, but from the current and specific text you give it.

Critical point: RAG does not change the weights of the model, that is, the billions of numerical parameters that the model has learned. You don't retrain the model. For each question, you inject chunks of text relevant to that question into the prompt (instruction text sent to the model). So you don't have to retrain the model when a document is updated; you simply refresh the relevant record in the search database.

Hint: Two questions determine the quality of the RAG: (1) Did you find the right document? (2) Did the model read it correctly? The first is "retrieval quality", the second is "generation quality". The two are measured and improved separately.

RAG, Fine-Tuning or Long Context?

Three paths are often confused when looking for a solution to an organizational problem. Let's clarify their differences. Fine-tuning is updating the model's weights with your data and teaching it a new behavior/style. Long context means filling all documents directly into the prompt without any selection.

Approach

What does

When is it appropriate?

Cost / Risk

RAG

Injects the relevant document as context

Frequently changing, extensive, specific information

Low; easy to update, source can be cited

Fine-tuning

Updates weights with new data

Fixed style/format/language teaching

High; Retraining required with every update

Long context only

Fills all documents into the prompt

Small, stationary document set

Token cost and risk of "losing the middle part" increases

As a rule: Fine-tuning teaches the model how to talk; RAG tells the model what to know. In most enterprise scenarios, RAG is tried first because it is cheap, updatable, and can show the source of the answer. Long context is reasonable if the document set is really small and fixed (e.g. a single 20-page manual); But with thousands of pages, it is expensive and the model may miss information in the middle of long text.

A Typical RAG Pipeline

RAG consists of two main phases: indexing (preparation, done once or periodically) and querying (runs on every user question).

Step by step indexing (offline, without user waiting):

1. Collect: Pull documents from sources (PDF, wiki, ticket system, database, email).
2. Chunking: Break long text into smaller manageable pieces.
3. Embed: Convert each part into embedding (the number vector that carries the meaning of the text).
4. Save: Write the vectors along with the text and metadata (source, date, authorization information) to the vector database.

Step-by-step query (online, while user is waiting):

1. Convert the user's question to embedding.
2. Retrieve the most similar parts from the vector database.
3. Place these pieces + question into a prompt template.
4. Get the contextual answer and its sources from the model.

# Conceptual outline of the inquiry phase (not dependent on language)question = "How many days of annual leave?"question_vektor = embed(question)parts = vektor_db.search(question_vektor, top_k=4) # most similar partsprompt = f"""Answer the QUESTION using the CONTEXT below. If the answer is not in context, say "I have no information about this." Fitting.CONTEXT:{parts}QUESTION: {question}"""answer = model.uret(prompt) # e.g. model: claude-opus-4-8

This flow is a map of each stage, which we will unpack one by one in subsequent units.

Weak Prompt / Strong Prompt

Even with the same RAG context, the quality of the prompt changes the answer.

Weak prompt (open to model fitting, does not require resources):

Use this information and say annual leave: {parts}. Question: {question}

Powerful prompt (grounding + "I don't know" permission + resource request):

Answer only based on the CONTEXT below. If there is no clear answer in the context, write "I couldn't find information about this in the documentation"; Don't guess. Add the [Source: file_name] tag of the piece you are relying on at the end of your answer. CONTEXT: {pieces} QUESTION: {question}

Three Mini Cases

Case 1 — HR assistant (Human Resources). A company has a 340-page HR handbook and employees ask an average of 90 questions a day. Fine-tuning was tried, but since the manual was updated monthly, retraining was required each time; The cost reached thousands of dollars per month. After switching to RAG, the update was reduced to the "re-index the document" step (minutes) and the correct-answer rate increased from 71% to 93% in manual measurement.

Case 2 — Customer support. The support team has 12,000 resolved tickets and 800 help articles. It takes an average of 4 minutes for a representative to manually find an answer. When the RAG assistant brought the 5 most relevant records and produced a draft response, the time was reduced to 40 seconds; But the team realized the risk of "looking unsure by bringing the wrong article" and made citing the source mandatory.

Case 3 — Law. A contracting team asked "in which contracts does the confidentiality clause last 5 years?" he asks the question. In the long context trial, 60 contracts were filled into a single prompt; the model skipped the middle two contracts. When only the relevant items were introduced with RAG, the token cost decreased by 80% and the missing skipping was reset.

Why is RAG needed?

• Currentness: You access information after the training cut-off date.
• Special information: Your internal documents are not included in the training of any model; Only you can give.
• Verifiability: You can cite the source of the answer (citation) — essential for auditing and trust.
• Hallucination control: It relies on the text placed in front of it rather than making up a model.
• Cost: It is much cheaper and faster to put into operation than fine-tuning.

Caution: RAG is not magic. If you bring in the wrong piece, the model arrives at the wrong answer looking “confident.” Keep in mind the phrase "Retrieval quality = RAG quality".

Common mistakes

• Mistaking RAG for fine-tuning: RAG does not change the weights; It just adds context. Confusing these two will lead to choosing the wrong architecture.
• Not allowing "I don't know": If the prompt leaves the model free to fill in the blank, it will make up.
• Not citing sources: An answer without a source cannot be checked; The user cannot notice the mistake.
• Cramming everything into one prompt: Long context looks cheap but is expensive and misses the middle information.
• Getting stuck in generation without measuring the retrieval: If the answer is bad, first ask "Did the right part arrive?" should be asked.

In summary

• RAG is an approach that injects documents relevant to the question into the model as context; does not change the weights ("open book exam").
• Fine-tuning teaches style/format, RAG gives current and specific information; long context works well for small fixed sets. In most scenarios, RAG is tried first.
• The pipeline has two phases: offline indexing (chunk + embedding + save) and online querying (retrieval + prompt + generate).
• RAG provides timeliness, specific information, verifiability, hallucination control, and low cost.
• The quality of the system directly depends on the quality of the retrieval: wrong piece means wrong answer.

Application task

Choose a genuine source of information from your own team (e.g. a procedure document or FAQ page). (1) Write 5 factual questions about this source. (2) Note which part of the document contains the correct answer for each question — this becomes your “golden answer” list. (3) Using the "strong prompt" template above, manually paste the relevant section as context and ask a model. (4) Compare the answer given by the model with the golden answer and mark as true/false. This is the first manual version of the assessment that you will automate in future units.

checklist

• [ ] I can explain in one sentence that RAG does not change the weights, it just adds context.
• [ ] I can distinguish between RAG, fine-tuning and long context and when which is appropriate.
• [ ] I can count the indexing (collect-shred-embed-save) and query (embed-fetch-prompt-generate) phases in order.
• [ ] I know why I added the "if it's not in context, say I don't know" and "cite source" instructions to the prompt.
• [ ] I can adapt the principle of "Retrieval quality = RAG quality" to my own case.