RAG vs Fine-Tuning: The Decision Guide Engineers Actually Use

The RAG-versus-fine-tuning conversation has been recycled in AI blogs for three years. The conclusion is usually "it depends." That is correct and useless. This is the decision framework that gets the answer right per workflow, with the engineering trade-offs both choices actually entail.

What each technique actually does

RAG (Retrieval-Augmented Generation): at inference time, retrieve relevant context from a knowledge base and pass it to the model as part of the prompt. The model reasons over the retrieved context plus your instruction. The knowledge stays in the retrieval store; the model stays generic.

Fine-tuning: take a base model and continue training it on examples relevant to your task. The model's weights update. The knowledge moves into the model's parameters. At inference time you do not need to provide the same context because the model has been adjusted.

These are not opposite ends of a spectrum. They solve different problems and often work best in combination.

When RAG is the right answer

RAG fits when:

• The knowledge is large: thousands or millions of documents, more than would fit in a fine-tuning dataset cleanly
• The knowledge changes frequently: new content arrives daily or weekly and you want it available immediately
• The knowledge needs to be auditable: you need to show the user (or the regulator) which source documents informed an answer
• Different users see different content: per-user, per-tenant, or per-role retrieval rules apply
• You want citations: the model should cite which document an assertion came from
• The task is question-answering or research: rather than a specific behavioural pattern

RAG is the right default for most enterprise AI agent workloads. The platform pattern of knowledge base + retrieval + grounded generation is mature and well-understood.

When fine-tuning is the right answer

Fine-tuning fits when:

• The task has a specific output format: you want the model to consistently produce JSON in a specific schema, write in a specific style, follow a specific reasoning pattern
• The task is well-defined and stable: a classification task with 100 categories that does not change weekly
• You have high-quality labeled examples: hundreds to thousands of curated input-output pairs
• Latency matters: fine-tuned models can be smaller and faster than equivalent quality from a larger model with RAG context
• Cost matters at high volume: smaller fine-tuned models can be dramatically cheaper than larger models doing the same task
• The behaviour pattern is the value: not the knowledge, the behaviour

For high-volume narrow tasks (extracting structured data from a specific document type, classifying support tickets into a stable taxonomy, generating compliance disclosures in a specific format), fine-tuning a smaller model often wins on cost and latency.

When the combination is the right answer

Most production AI agent workflows benefit from both:

• Fine-tune for behaviour: a smaller model fine-tuned to follow the agent's specific reasoning pattern, output format, or domain-specific vocabulary
• RAG for knowledge: the fine-tuned model receives retrieved context at inference time
• Result: cheaper, faster, more controllable than RAG on a frontier model, with the knowledge freshness RAG provides

This pattern is common in mature deployments. It does require fine-tuning capability you may or may not have, which is itself an engineering question.

The cost trade-off in numbers

For a representative workload of 100,000 inferences per month on a moderately complex task:

Frontier model + RAG:

• Model cost: EUR 500-3,000 per month
• Retrieval cost: EUR 50-200 per month
• Engineering cost: 0.2-0.5 FTE ongoing
• Time to first quality: 2-6 weeks

Fine-tuned smaller model + RAG:

• Fine-tuning cost: EUR 500-5,000 one-time per model version
• Model cost: EUR 100-500 per month
• Retrieval cost: EUR 50-200 per month
• Engineering cost: 0.5-1.5 FTE ongoing (including model retraining cycles)
• Time to first quality: 6-16 weeks
• Quality risk: fine-tuning can fail or produce regressions; budget for iteration

For 100,000 inferences per month the fine-tuning investment often pays back in 6-12 months on direct cost alone. For 10,000 inferences per month it usually does not. For 1,000,000 inferences per month it pays back in weeks.

The engineering trade-off in operations

RAG operations are about:

• Document ingestion pipelines
• Chunking strategies
• Embedding models and vector indexes
• Retrieval quality evaluation
• Access control on retrieval

Fine-tuning operations are about:

• Training data curation
• Training pipeline reliability
• Model versioning and deprecation
• Evaluation harnesses
• Production model serving

Each is real engineering work. RAG operations are typically more tractable for teams without ML engineering background. Fine-tuning operations require ML engineering capability that not all teams have.

What does not change either way

Both approaches require:

• Clear understanding of the task and success metrics
• Evaluation infrastructure to measure quality over time
• Production monitoring for drift and degradation
• Governance for the data flowing through the system

If your team does not have these for RAG, adding fine-tuning will not help. If your team has these for RAG, adding fine-tuning is tractable.

The decision questions

For each workflow, walk these:

1. Is the knowledge changing weekly or faster? If yes, RAG. If no, either is possible.

2. Do you need citations to source documents? If yes, RAG. If no, fine-tuning is possible.

3. Is the volume high enough to justify the fine-tuning investment? Above ~50,000 inferences per month for a stable task, yes. Below, probably not.

4. Is the behaviour pattern unusual enough that prompting cannot capture it? If yes, fine-tuning. If no, RAG with good prompting.

5. Does latency or per-call cost dominate the workload economics? If yes, fine-tuning a smaller model helps. If no, RAG on a capable model is fine.

6. Do you have the ML engineering capacity? If no, RAG only. If yes, evaluate both.

Common failure patterns

Fine-tuning when RAG would do: teams sometimes reach for fine-tuning because it sounds more sophisticated. The result is months of work and an opaque model when a well-built RAG pipeline would have shipped in weeks.

RAG with poor retrieval: bad chunking, weak embeddings, no evaluation. The model gets noisy context and produces mediocre answers. The fix is RAG engineering work, not switching to fine-tuning.

Fine-tuning on bad data: fine-tuning amplifies whatever pattern is in the training set. Bad examples in, bad model out. Data curation is the dominant engineering cost.

Skipping evaluation: both techniques require evaluation harnesses. Without them you cannot tell whether changes help or harm.

What AgentWorks supports

AgentWorks supports both patterns. The knowledge base and RAG features handle the retrieval side natively. For fine-tuned models (your own or from a fine-tuning provider), the platform routes to them through the same model interface as the frontier models. The audit log captures which model and which retrieved context informed each output, so the auditability story works for either pattern.

The recommendation we make most often: start with RAG on a frontier model. Measure quality and cost. If quality is good and cost is acceptable, ship. If cost is the constraint at scale, evaluate fine-tuning a smaller model with RAG on top. If quality is the constraint, the answer is usually better RAG engineering before fine-tuning is the right move.