LLM Fine-Tuning vs. RAG: When to Use Each

The question is usually posed as a fork — fine-tune or use RAG — and posed that way it has no good answer, because the two techniques address different problems. One is about knowledge, the other about behaviour. Deciding between them requires knowing which of those two things your system is currently getting wrong.

Definitions, precisely

Retrieval-augmented generation (RAG): Fetch relevant passages from an external store at query time and include them in the prompt. Changes what the model knows for that one answer. Knowledge updates by re-indexing a document.
Fine-tuning: Continue training a base model on your own examples so its weights shift. Changes how the model behaves by default — format, tone, task convention, domain vocabulary. Knowledge updates by retraining.
Parameter-efficient fine-tuning (LoRA/QLoRA): Train a small set of adapter weights instead of the whole model. Most of the behavioural benefit at a fraction of the compute and storage, and adapters can be swapped per customer or task.
Prompt engineering: Neither of the above, and the correct first attempt. Free, instant to iterate, and frequently sufficient. Exhaust it before you spend money.

The decision framework

Ask three questions in order. The first that yields a clear answer determines the approach.

1. Does the failure involve facts the model does not have?

If the model is wrong because it does not know your policies, your product catalogue, your customers or anything that happened after its training cutoff, that is a knowledge gap, and knowledge gaps are retrieval problems. Fine-tuning facts into weights is possible and almost always the wrong trade: the facts become unattributable, un-updatable without retraining, and impossible to cite.

2. Does the failure involve format, tone or task convention?

If the model knows the answer but presents it wrongly — ignores your schema, adopts the wrong register, mislabels categories your domain defines idiosyncratically — that is behavioural, and behaviour is what fine-tuning changes. The tell is that a long prompt full of examples fixes it, but the prompt is now so long it is expensive and still occasionally ignored.

3. Is the problem cost or latency at scale?

A fine-tuned small model can match a much larger model on one narrow task, at a fraction of the per-token cost and latency. If a working system is simply too expensive or too slow, distilling it into a fine-tuned smaller model is a legitimate and underused move — but only once you have a working system and its outputs to train on.

Choose RAG when

The knowledge changes — documentation, policies, prices, inventory, tickets, anything with a version.
Answers must cite their source. Retrieval gives you provenance for free; fine-tuning destroys it.
Access control matters. Retrieval can be filtered per user; weights cannot.
The corpus is large. You cannot fine-tune a hundred thousand documents into a model usefully, and you do not need to.
You need auditability — the ability to show exactly which passage produced a given answer.
Requirements are still moving. Re-indexing takes minutes; retraining takes a cycle.

Choose fine-tuning when

Output format must be exact and prompt instructions are inconsistently followed.
The domain has vocabulary or conventions the base model handles poorly — specialised clinical, legal or industrial language.
Tone or house style must be consistent across thousands of generations.
The task is narrow, repetitive and high-volume, and a smaller fine-tuned model would cut cost materially.
Few-shot examples work but consume so much context that they have become the dominant cost.
Latency budgets rule out the large model that currently gets it right.

Use both when the system is mature

The strongest production systems fine-tune for behaviour and retrieve for facts. A support assistant might be fine-tuned to follow your escalation taxonomy, adopt your tone and always emit a structured resolution object — while retrieving the current product documentation at query time. Neither technique substitutes for the other; each is doing the job it is suited to.

What each actually costs

Comparing costs honestly means separating one-off from recurring, and engineering time from compute. Compute is rarely the dominant term.

RAG

One-off: ingestion connectors, chunking strategy, index setup, evaluation set. Days to weeks of engineering.
Recurring: vector database hosting, re-embedding on document change, and a larger prompt on every query because retrieved context is included.
Iteration: fast. A chunking or reranking change is testable in minutes.

Fine-tuning

One-off: dataset construction — realistically hundreds to low thousands of high-quality examples, and this is the expensive part, not the GPU time.
Recurring: cheaper inference per token if you moved to a smaller model, plus retraining whenever behaviour or the base model changes.
Iteration: slow. Each experiment is a training run, and dataset problems only surface after it.

The asymmetry that matters: RAG iterations are cheap and reversible, fine-tuning iterations are neither. That alone justifies trying retrieval first in ambiguous cases.

Failure modes to expect

RAG: retrieval misses: The answer exists in the corpus but never reached the prompt. Almost always chunking or the absence of hybrid search. Measure recall@k before blaming the model.
RAG: context dilution: Too many passages retrieved, the relevant one buried among mediocre neighbours. Retrieve broadly, rerank hard, pass few.
Fine-tuning: catastrophic forgetting: The model gets better at your task and worse at everything else. Mitigated by parameter-efficient methods and by keeping general examples in the training mix.
Fine-tuning: baked-in staleness: Facts trained into weights are now wrong and cannot be corrected without retraining. This is the failure that makes people regret fine-tuning knowledge.
Both: no evaluation set: Without a golden set, every change is judged by whichever example someone tried last. This is the most common failure of all, and it is not technical.

The short version

Facts that change, need citations, or need per-user access control: retrieval.
Format, tone, domain convention, or cost-driven model shrinking: fine-tuning.
Try prompt engineering properly before either — it is free and often enough.
Build the evaluation set before choosing, or you will not be able to tell whether the choice worked.
Mature systems use both, for the different jobs each is good at.