Retrieval-augmented generation (RAG) gives a language model your knowledge at the moment it answers, by searching your documents and placing the relevant passages in the prompt; fine-tuning retrains the model on examples so that it behaves differently every time. If your problem is facts that change, or answers that must point to a source, you need retrieval. If your problem is a consistent format, tone or narrow task that prompting cannot pin down, fine-tuning is the candidate, and most business systems need the first long before they need the second.
What is RAG?
RAG was introduced in a 2020 paper by Lewis and colleagues, which described models that combine pre-trained parametric and non-parametric memory for language generation. The parametric memory is what the model learned in training. The non-parametric memory is an external index it can search, in the original work a dense vector index of Wikipedia accessed with a neural retriever. The authors reported that RAG models generate more specific, diverse and factual language than a parametric-only baseline, and they named the two problems it addresses: providing provenance for decisions and updating world knowledge.
A modern RAG pipeline looks much the same. Anthropic's primer describes the preprocessing: break the knowledge base into chunks of usually no more than a few hundred tokens, convert them into vector embeddings, and store them in a vector database. At question time, the most relevant chunks are found and added to the prompt sent to the model. Good systems also add keyword search, because embeddings can miss crucial exact matches such as a specific error code.
The important property for a business is that the model's weights never change. Update a document, re-index it, and the next answer uses the new version.
What is fine-tuning?
Fine-tuning continues training an existing model on your own examples. Google's documentation describes it as retraining the base model on your own labeled dataset, adapting its weights to your task and data. OpenAI's guide puts the goal as giving the model the kinds of inputs and outputs you expect in your application so it excels at that task.
The documented benefits are about behavior and efficiency, not knowledge. OpenAI lists more examples than fit in one prompt, shorter prompts that save tokens and latency, training on proprietary data without sending it in every request, and training a smaller, cheaper model to excel at a particular task. Google's list is similar: alignment with custom syntax and domain rules, more consistent results, and lower inference cost by removing few-shot examples and long instructions. Google recommends supervised fine-tuning for classification, sentiment analysis, entity extraction, summarization of content that is not complex, and writing domain-specific queries.
There are variants. OpenAI documents supervised fine-tuning, vision fine-tuning, direct preference optimization and reinforcement fine-tuning. Google distinguishes parameter-efficient tuning, which updates a small subset of parameters, from full fine-tuning, which updates all of them and costs more to train and serve.
Which one is better for adding knowledge?
Retrieval, according to a direct comparison published in 2023. Ovadia and colleagues compared unsupervised fine-tuning and RAG on knowledge-intensive tasks and found that while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge. They also found that LLMs struggle to learn new factual information through unsupervised fine-tuning, and that exposing the model to numerous variations of the same fact during training could ease the problem.
That matches how the two methods work. A fine-tuned model absorbs your documents as statistical tendencies in its weights, with no record of which document a sentence came from. A RAG system hands the model the passage itself, so the answer can quote it and a person can check it. If your price list changes next month, a RAG system needs one re-indexed file; a fine-tuned model needs a new training run.
How do you decide which one your system needs?
Four questions settle most cases.
| Question | If yes | If no |
|---|---|---|
| Does the knowledge change more often than you would retrain a model? | RAG | Either |
| Must answers cite or link the source they came from? | RAG | Either |
| Is the problem the model's format, tone or task behavior rather than missing facts? | Fine-tuning candidate | RAG or prompting |
| Do you have 100 or more clean, labeled examples of the ideal output, and someone to maintain them? | Fine-tuning is feasible | Stay with prompting |
The example threshold is Google's: fine-tuning is most effective when you have a sizable dataset, think 100 examples or more, and its description of supervised fine-tuning speaks of hundreds of labeled examples. It also stresses that high-quality, well-labeled data is crucial and better than quantity.
The last question is the one teams skip. A retrieval index is maintained by whoever maintains the documents. A fine-tuned model is maintained by whoever can rebuild the training set, rerun the job and re-test the result, every time the base model is retired.
Should you try prompting or long context first?
Yes. Both providers put prompting first. Google recommends starting with prompting to find the optimal prompt, then moving on to fine-tuning if required. OpenAI describes optimization as a loop of evals, prompt engineering and fine-tuning, and says the prompt engineering process may be all you need.
Sometimes you do not even need retrieval. Anthropic wrote in September 2024 that if your knowledge base is smaller than 200,000 tokens, about 500 pages, you can include the entire knowledge base in the prompt with no need for RAG, and that prompt caching makes this significantly faster and more cost-effective. Our explainer on prompt caching covers how the cached prefix is billed. Once the knowledge base outgrows that, retrieval is the scalable answer.
How reliable is retrieval, and how do you make it better?
Retrieval fails when the right passage is not found, and the model then answers from whatever it did find. Anthropic measured this directly. With its best embedding setup, the share of relevant documents missing from the top 20 retrieved chunks was 5.7%. Adding a short, generated context to each chunk before indexing, which it calls Contextual Retrieval, cut that failure rate to 3.7% with contextual embeddings, 2.9% when combined with contextual keyword search, and 1.9% with a reranking step added.
The same write-up gives practical settings: passing the top 20 chunks to the model worked better than the top 10 or top 5, chunk size and overlap affect performance, and it advises you to always run evals. Its 2024 estimate of the one-time cost to contextualize documents was $1.02 per million document tokens with prompt caching, under the assumptions stated in the post.
The lesson for a buyer is that "we use RAG" says little. Ask how retrieval quality is measured, on which questions, and what the system does when nothing relevant is found.
Can you combine RAG and fine-tuning?
Yes, and when both are justified they do different jobs. Retrieval supplies the facts for each question. Fine-tuning shapes how the model uses them: the output structure, the labels, the tone. A support assistant might retrieve from the help centre and policy documents, while a tuned model classifies each ticket into your internal categories. OpenAI notes that fine-tuning can enable a model to consistently format responses in a certain way, which is exactly the part retrieval cannot fix.
What you should not do is fine-tune to avoid building retrieval. The Ovadia results suggest the facts will not stick reliably, and you will have given up the ability to show where an answer came from.
Is fine-tuning still available in October 2026?
It depends on the provider, and this changed during 2026. OpenAI's model optimization guide now states that OpenAI is winding down the fine-tuning platform and it is no longer accessible to new users. Its deprecations page gives the dates: from May 7, 2026, fine-tuning is not available to organizations that have not previously run it; from July 2, 2026, not to organizations that have not run inference on a fine-tuned model in the past 60 days; and on January 6, 2027, active existing customers can no longer create new fine-tuning jobs. Existing fine-tuned models keep running until the underlying base model is deprecated. On October 14, 2026, OpenAI announced that GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini leave the API on January 14, 2027, and that fine-tuned versions of these models follow the same shutdown schedule as their underlying base models.
Google still offers supervised tuning on Gemini. As of October 2026 its documentation lists Gemini 3.5 Flash, Gemini 3.1 Flash-Lite, Gemini 2.5 Pro, Gemini 2.5 Flash-Lite and Gemini 2.5 Flash as supported models. A fine-tuned model is tied to its base model, so at OpenAI its lifespan is the base model's lifespan. A RAG system can usually move to a newer model without retraining anything, provided you re-run its tests.
What does each approach cost to run and maintain?
The running cost of RAG is mostly input tokens: every retrieved passage is sent with every question. Our guide to input vs output tokens shows how that adds up, and caching the stable parts of the prompt reduces it. You also pay for embeddings and a search index, and someone keeps the documents current.
Fine-tuning moves cost to the front. You pay for building and cleaning the dataset, for training, and for testing the tuned model against the base model. The saving comes later, in shorter prompts and lower latency or the ability to use a smaller model. That trade usually pays off only on high volumes of a stable task.
At Qubba we build document processing systems, internal tools and AI agents on a fixed written quote after a 30 minute call.
Frequently asked questions
Is fine-tuning dead?
No, but its role is narrower than it was.
Google still documents supervised and preference tuning for Gemini, and recommends it for tasks like classification and entity extraction.
OpenAI, however, has been winding down its self-serve fine-tuning platform since May 7, 2026.
Treat fine-tuning as a tool for behavior and cost, not for knowledge.
Does RAG stop hallucinations?
It reduces the room for them but does not remove them.
The original RAG paper found retrieval produced more factual language than a parametric-only baseline, and citations let a reader check the claim.
The model can still misread a passage or answer when retrieval found nothing useful, so the system needs a rule for what to say when the sources do not contain the answer.
What is the difference between RAG and prompt engineering?
Prompt engineering is how you instruct the model.
RAG is how you fill part of that prompt with the right documents automatically.
OpenAI's own advice for prompts is to include relevant context the model needs from outside its training data, such as private databases or current information; RAG is the machinery that does that at scale.
How much data do I need to fine-tune a model?
Google's guidance is 100 examples or more for fine-tuning to be effective, and quality matters more than volume.
The examples should reflect the prompt distribution, format and context the model will see in production.