Most comparisons of fine-tuning, RAG and prompt engineering treat them as three competing ways to do the same job, then hand you a table of pros and cons.
That framing is the reason teams pick wrong. They are not alternatives. They fix three different failures, and choosing between them starts with diagnosing which failure you actually have.
Get the diagnosis right and the choice is usually obvious. Get it wrong and you can spend months fine-tuning a model to fix a problem fine-tuning cannot touch.
The Quick Answer
- The model doesn't know something → retrieval (RAG)
- The model knows, but behaves wrong → prompting first, fine-tuning if it persists
- You haven't tried yet → prompting, always, before anything else
That is the whole decision. The rest of this article is why, and where it gets complicated.
Diagnose the Failure First
Start from the symptom, not the solution.
The distinction that does the most work is knowledge versus behaviour.
A model that confidently invents a policy your company does not have is missing knowledge. No amount of fine-tuning on writing style fixes that — you will get a fluent, on-brand, wrong answer.
A model that knows the right answer but buries it in three paragraphs of preamble has a behaviour problem. Retrieval will not help; you are feeding it more material to be verbose about.
Prompting: Start Here, Always
Prompt engineering means changing what you send, not what the model is: clearer instructions, worked examples, a system prompt that states the format, explicit constraints.
It is free, takes hours rather than months, and has a benefit teams consistently undervalue: it tells you what the model can already do. You cannot know whether you need the expensive options until you have established the baseline.
Two things made prompting substantially more powerful by 2026:
Context windows got enormous. Frontier models now carry roughly a million tokens, so "just include the reference material" is viable for volumes that once demanded a retrieval pipeline. Our guide to context windows covers what that does and does not buy you.
Caching made repetition cheap. Prompt caching bills repeated prefixes at roughly a tenth of the normal input rate. A large fixed instruction block or reference document that once looked unaffordable at scale is now routine.
Together these moved the threshold. Work that justified a project in 2023 is now a long prompt.
RAG: When the Model Doesn't Know
Retrieval-augmented generation keeps your knowledge outside the model and pulls in only the relevant parts at question time.
The RAG survey by Gao and colleagues frames the purpose precisely: RAG addresses hallucination, outdated knowledge, and non-transparent, untraceable reasoning, and it allows "continuous knowledge updates and integration of domain-specific information."
That last phrase is the practical argument. Your documentation changes weekly. A retrieval index can be updated in minutes; a fine-tuned model has to be retrained. If the knowledge moves, it does not belong in the weights.
RAG also gives you something fine-tuning structurally cannot: traceability. You can show which document produced an answer. In regulated settings that is often the requirement that decides the architecture.
The mechanics are in our RAG explainer, and if your questions are about relationships across a whole corpus rather than passages, GraphRAG vs vector RAG covers where plain retrieval runs out.
Fine-Tuning: When Behaviour Is Wrong
Fine-tuning continues training on your examples, adjusting the model's weights. It is the only one of the three that changes the model itself.
It genuinely wins in four situations:
- Persistent format or style requirements that prompting keeps drifting from
- Classification with a fixed label set and plenty of labelled examples
- Latency or cost pressure, where a fine-tuned small model replaces a large general one
- Deeply idiosyncratic tasks with no natural language description
Notice what is absent: teaching the model facts. Fine-tuning can shift what a model tends to say, but using it as a knowledge store is the most common and most expensive mistake in this area. Facts go stale, cannot be cited, and cannot be updated without retraining.
The Study Everyone Cites — and What It Actually Shows
There is one head-to-head comparison that circulates constantly, and it deserves to be read properly.
Kermani, Perez-Rosas and Metsis ran a systematic evaluation of all three approaches on mental-health text analysis using LLaMA 3, across two datasets and two tasks: emotion classification and mental health condition detection.
Fine-tuning won decisively: 91% accuracy on emotion classification and 80% on condition detection, against 40–68% for prompt engineering and RAG. The authors also note it demanded significant computational resources and large training datasets, while the other two offered more flexible deployment.
Now read the task. This was classification — a fixed set of labels, with ample labelled examples. That is precisely the scenario fine-tuning is built for, and precisely where RAG has least to contribute, because there is no external knowledge to retrieve. The comparison is fair; the generalisation people draw from it is not.
If your problem is "sort these into five categories," this study is directly relevant. If your problem is "answer questions about our internal documentation," it tells you almost nothing.
The Hidden Cost: Catastrophic Forgetting
Fine-tuning carries a risk that rarely appears in vendor documentation.
Pre-trained knowledge and task-specific knowledge live in the same weight matrices. There is no separate compartment for general ability. When gradient descent adjusts a weight to fit your new data, it may be degrading a capability that weight was also serving. This is catastrophic forgetting, and it is why a model that becomes excellent at your narrow task can get noticeably worse at everything around it.
The usual reassurance is that parameter-efficient methods such as LoRA avoid this. The research is less comfortable. As Xiong and Xie put it, LoRA "suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge." LoRA constrains how much the weights move; it does not guarantee that what mattered is preserved.
Practical implication: evaluate the capabilities you did not fine-tune for. A benchmark that only measures your target task will report success while the model quietly gets worse at reasoning, instruction-following, or formatting.
Climb the Ladder in Order
The order matters more than the options. Escalate only when the rung below has demonstrably failed — not when it feels unsophisticated.
The failure mode worth naming: a team skips to fine-tuning because it sounds like the serious engineering answer, spends months and considerable GPU budget, and ends up with a model that produces wrong answers in the correct house style. The knowledge gap was never addressed, and it is now harder to see.
They Combine, and That Is the Common Endpoint
The three are not mutually exclusive, and mature systems usually run at least two.
The standard combination is fine-tuning for behaviour plus RAG for knowledge: a model tuned to follow your output format and domain conventions, retrieving current facts at question time. Each does what it is actually good at, and the tuned model tends to use the retrieved context more reliably because handling it is exactly what it was trained on.
That pairing is the architecture behind most serious enterprise deployments — but note the order in which it arrives. Teams reach it after establishing that prompting alone was insufficient, not instead of trying.
The Bottom Line
Ask what is actually broken before choosing a tool.
Missing or stale knowledge is a retrieval problem. RAG updates in minutes, cites its sources, and never needs retraining. Wrong behaviour is a prompting problem first and a fine-tuning problem only when prompting has genuinely and repeatedly failed.
The 91% figure from the head-to-head study is real, and it is about classification. Fine-tuning earns that result on tasks with fixed labels and abundant examples. It does not earn it on questions whose answers live in documents.
And in 2026, the bar for escalating moved. Million-token context windows and caching at roughly a tenth of the input rate mean a surprising amount of what once required infrastructure is now a well-constructed prompt. Try that first — it is the cheapest experiment you will run, and it tells you which of the other two you actually need. More in our LLM hub.
Frequently Asked Questions
What is the difference between fine-tuning and RAG?
Fine-tuning changes the model by continuing training on your examples, which adjusts how it behaves. RAG leaves the model untouched and supplies relevant external information at question time, which changes what it knows for that request. The short version: fine-tuning changes behaviour, retrieval changes knowledge. Choosing between them starts by identifying which of those two is actually failing.
Can I fine-tune a model to teach it my company's information?
You can, but it is usually the wrong tool and the most expensive mistake in this area. Facts embedded in weights cannot be cited, go stale the moment your documentation changes, and require retraining to update. Retrieval keeps the knowledge outside the model, updates in minutes, and can show which document produced an answer — which is often a hard requirement in regulated settings.
Does fine-tuning make a model worse at other things?
It can, and this is under-discussed. Pre-trained and task-specific knowledge share the same weight matrices, so adjusting weights for your task can degrade capabilities those weights also served — catastrophic forgetting. Parameter-efficient methods like LoRA reduce but do not eliminate it; research shows LoRA updates can still interfere with the directions encoding essential pre-trained knowledge. Always evaluate capabilities you did not tune for.
Which performs best: fine-tuning, RAG or prompting?
It depends entirely on the task, which is why the widely-cited study needs careful reading. Kermani et al. found fine-tuning reached 91% accuracy against 40–68% for prompting and RAG — on classification, with fixed labels and ample labelled examples, which is fine-tuning's strongest scenario. For knowledge-intensive question answering, where facts must be current and traceable, retrieval is generally the better fit.
Has 2026 changed when fine-tuning is worth it?
Yes, the threshold moved upward. Frontier context windows now hold around a million tokens, so reference material that once needed a retrieval pipeline can often simply be included. Prompt caching bills repeated prefixes at roughly a tenth of the normal rate, making large fixed instructions economical at scale. Both narrow the range of problems where fine-tuning is the cheapest adequate answer.
Should I use more than one of these together?
Often, and it is the common endpoint for mature systems. The standard pairing is fine-tuning for behaviour plus RAG for knowledge: a model trained to follow your format and conventions, retrieving current facts at question time. A tuned model also tends to use retrieved context more reliably, since handling that context is what it was trained to do. Arrive there after prompting proves insufficient, not instead of trying it.
How long does each approach take to implement?
Prompting is measured in hours and needs no infrastructure. RAG typically takes weeks and leaves you a retrieval pipeline to maintain — ingestion, chunking, embeddings and an index. Fine-tuning is measured in months and requires labelled training data plus GPU capacity, and the resulting model needs re-evaluating whenever the base model updates. The gap between rungs is the main argument for climbing them in order.
Sources
- Kermani, Perez-Rosas & Metsis — A systematic evaluation of LLM strategies for mental health text analysis: fine-tuning vs. prompt engineering vs. RAG (arXiv:2503.24307)
- Gao et al. — Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv:2312.10997)
- Xiong & Xie — OPLoRA: Orthogonal Projection LoRA prevents catastrophic forgetting during parameter-efficient fine-tuning (arXiv:2510.13003)
- Databricks — Efficient fine-tuning with LoRA: a guide to optimal parameter selection
- Anthropic — Prompt caching
