RAG or fine-tuning? A practical way to decide
Retrieval and fine-tuning solve different problems. Here is how to tell which one yours is.
, 7 minute read
Teams adding language models to a product usually reach the same fork early: should we give the model our knowledge through retrieval, or teach it through fine-tuning? The two are often discussed as rivals. In practice they answer different questions, and confusing them is one of the most common reasons early projects stall.
What each technique actually changes
Retrieval-augmented generation (RAG) leaves the model untouched. At request time you search your own documents, place the most relevant passages in the prompt, and ask the model to answer from them. The model's knowledge is supplied fresh on every call.
Fine-tuning changes the model's weights by training it further on examples. It is good at shifting behaviour: tone, format, the way a task is performed, or a narrow skill like classifying your support tickets. It is a poor way to store facts that change, because updating them means training again.
Four questions that usually settle it
- Does the answer depend on information that changes weekly or monthly? Prefer retrieval.
- Do users need to see where an answer came from? Prefer retrieval, which can cite its sources.
- Is the problem the model's behaviour, not its knowledge (format, style, a specialised classification)? Consider fine-tuning.
- Do different users have permission to see different documents? Retrieval can filter per user; a fine-tuned model cannot forget what it learned for one user when serving another.
Start with measurement, not architecture
Before choosing, build a small evaluation set: fifty to a hundred real questions with known good answers. Run a plain prompt against it, then a basic retrieval pipeline. The gap between them tells you how much of your problem is missing knowledge. If retrieval fixes most failures, you probably don't need fine-tuning yet. If the remaining failures are about format or reasoning style, that is where a fine-tuning experiment earns its cost.
A useful rule of thumb: retrieve what the model needs to know; fine-tune how the model needs to behave.
When to combine them
Mature systems often use both. A model fine-tuned to read retrieved passages carefully, cite them and refuse when evidence is missing can outperform either technique alone. But that combination is a second step. Teams that start there tend to spend months on training infrastructure before learning whether retrieval alone would have been enough.
Our LLMs and RAG Systems program covers this decision in depth, including how to measure retrieval quality separately from answer quality.