Skip to content
Shahid Malla

Fine-Tuning vs RAG vs Prompt Engineering: Which to Use

The three levers fix different faults, so the question is which fault you have. What each one changes, the data and effort it needs, how fast you can update it, and what the published experiments actually measured.

· Published · 11 min read · 7 steps

Prompt engineering, RAG and fine-tuning fix different faults. A prompt fixes instructions the model misread. RAG fixes missing or changing facts, because it puts your own documents into the context at answer time. Fine-tuning fixes behavior: format, tone, and a task that has to come out the same way every time. Start with the prompt, add retrieval for knowledge, fine-tune last.

Fine-tuning continues a model's training on your own examples, so the change lives in the weights and applies to every request. RAG, retrieval-augmented generation, leaves the weights alone and searches your documents for passages to put in the prompt before the model answers. Prompt engineering changes only the instructions and examples you send, so it is the one lever you can undo in a minute.

Everything below is checked against sources I read on 14 November 2026: the OpenAI, Anthropic and Mistral documentation, and two papers that measured the comparison instead of asserting it. My guide on what a RAG chatbot is covers the retrieval machinery itself.

Five-step order for choosing an LLM method: write the eval, fix the prompt, add retrieval, fine-tune last, then re-score against the baseline and loop
The order I work in on a build: the eval set first, then the cheapest lever that fits the fault. Source: shahidmalla.com.

What does each method actually change?

One changes the instructions, one changes the context, one changes the weights. OpenAI's Optimizing LLM Accuracy guide frames this as two axes rather than a ladder. Context optimization is for a model that "lacks contextual knowledge because it wasn't in its training set" or needs proprietary information; it "maximizes response accuracy". LLM optimization is for one that "is producing inconsistent results with incorrect formatting" or has the wrong tone; it "maximizes consistency of behavior".

Here it is as faults you can see in a log.

The fault you seeThe lever that fixes itWhy
It ignores an instruction in some answersPrompt engineeringClearer wording and worked examples take minutes
The output format drifts between runsPrompt first, fine-tuning if it persistsA fixed template or structured output usually closes it
It does not know a fact only your company holdsRAGThe fact is not in the weights, so it must arrive in the context
The fact changes weekly: prices, stock, policyRAGEditing a document is instant, retraining is not
The answer needs a citation a customer can checkRAGRetrieval gives you the passage the answer came from
House tone and structure are wrong at scaleFine-tuningBehavior learned from many examples, not described once
The prompt is long and the token bill hurtsFine-tuningTraining replaces instructions and examples with weights
A small model must match a big one on one taskFine-tuningDistilling a larger model's outputs into a smaller one

The last two rows get forgotten. OpenAI's Model optimization page lists them plainly: fine-tuning lets you "use shorter prompts with fewer examples and context data, which saves on token costs at scale", and train "a smaller, cheaper, faster model" where a larger one is not cost-effective. Those are economic reasons, not accuracy ones.

How do you tell a knowledge problem from a behavior problem?

You read the failures one by one and ask whether the model lacked a fact or lacked a habit. OpenAI frames every failed question as an in-context or a learned memory problem. In-context is "you have the textbook with you, and can look up the right information". Learned is "you attend class for the last 6 months". The first is retrieval, the second is training.

Four questions over a set of real failures, before spending anything:

  • Did a clearer prompt fix it? If yes, stop. Anthropic's prompt engineering overview still says plainly that "not every success criterion or failing eval is best solved by prompt engineering".
  • Is the missing piece a fact in a document you own? That is retrieval, and no amount of training will keep it current.
  • Is the fault the shape of the answer rather than its content? Format, tone, length, structure. That is the behavior axis.
  • Do you have fifty or more real examples of the right answer, plus a held-out set? If not, you are not ready to fine-tune, whatever the diagnosis.
Decision flow: a better prompt ships, a fact in your own documents means retrieval, a format fault with fifty examples means fine-tuning, otherwise write the eval set first
The four questions above as a flow. A no at question three usually means the eval set is too vague to act on. Source: shahidmalla.com.

Question three catches a trap. If you cannot say whether a failure is knowledge or behavior, the problem is the test. Anthropic's guide to success criteria sets the standard: not "The model should classify sentiments well", but a scorable target such as an F1 score of at least 0.85 on a held-out set of 10,000 diverse Twitter posts.

What does each option need from you?

They differ in the setup each one needs and in how fast you can change your mind afterward. The data sizes below come from OpenAI's fine-tuning documentation, read on 14 November 2026; the timings are my own estimates, not vendor figures.

 Prompt engineeringRAGFine-tuning
What you prepareInstructions, worked examples, an output templateDocuments, a chunking and embedding pipeline, a vector storeA JSONL file of prompt and ideal-answer pairs, plus a held-out set
How much dataA few worked examples; OpenAI's case used threeAs many documents as you have, cleanedMinimum 10; OpenAI reports gains from 50 to 100 and advises starting at 50
Time to first resultMinutesDays, mostly on the documentsDays to weeks, mostly on the dataset
Time to change an answerEdit the promptEdit the document, re-indexRebuild the dataset, train again
Running costTokens for a longer prompt on every callEmbedding plus retrieved tokens on every callTraining once, then a shorter prompt
What it cannot doScale past what fits in contextChange how the model writes or reasonsKeep a changing fact current

Those data numbers come from OpenAI's supervised fine-tuning page: the minimum is 10 examples, and the advice is to "start with 50 well-crafted demonstrations". The follow-up rule is the useful part: if 50 good examples make no difference, "rethink your task or prompt before adding training data".

One availability note changes the decision for new projects. That page says OpenAI "is winding down the fine-tuning platform", which is "no longer accessible to new users"; eligible existing customers can create training jobs until 6 January 2027, unless the base model shuts down earlier. Mistral's fine-tuning documentation, read the same day, says that feature "is deprecated and is no longer actively supported". Check that your provider still sells fine-tuning before you plan around it.

What does the research say about teaching a model new facts?

It says retrieval wins, and not by a little. In Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, Oded Ovadia, Menachem Brief, Moshik Mishaeli and Oren Elisha of Microsoft tested three 7B models with unsupervised fine-tuning, RAG, and the two combined. Their finding: "while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge".

Their sharpest test is a current events set: 910 four-option questions built from United States events of August to November 2023, after the models' training cutoffs, so guessing scores 25 percent.

Bar chart of Mistral 7B on current-events questions: base model 48.1 percent, fine-tuned 50.4, fine-tuned on paraphrases 58.8, retrieval 87.5, fine-tuning plus retrieval 81.0
Table 2 of Fine-Tuning or Retrieval?, Mistral 7B row, read 14 November 2026. Source: arXiv:2312.05934.
Current events accuracyBaseFine-tunedFine-tuned on paraphrasesBase + RAGFine-tuned + RAG
Mistral 7B0.4810.5040.5880.8750.810
Llama2 7B0.3530.2190.3920.5850.326
Orca2 7B0.4560.5110.5660.8760.820

Three things there matter. Fine-tuning on the source text barely moved Mistral and Orca2, and damaged Llama2, dropping it below its own base score. Fine-tuning on ten paraphrases of each chunk did much better, which is the paper's hypothesis: "in order to teach pre-trained LLMs new knowledge, the knowledge must be repeated in numerous ways". And RAG on top of a fine-tuned model scored below RAG alone.

A second paper reaches the same place from the other end. Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge, by Heydar Soudani, Evangelos Kanoulas and Faegheh Hasibi, tested twelve models on low-frequency entities, the long tail where base models are weakest. Fine-tuning helped "across entities of varying popularity", but "RAG surpasses FT by a large margin particularly for least popular factual knowledge".

Read the limits too. The first paper used unsupervised fine-tuning, not instruction tuning, every corpus was scraped from Wikipedia, and the authors note that the choice of hyperparameters "significantly impacts the results". The direction holds; the numbers belong to their setup.

How do you prove which one worked?

With a scored test set written before you touch anything, run again after each change. OpenAI's accuracy guide puts the bar at "a set of 20+ questions and answers" with the failures already inspected. Anthropic's success criteria page adds a rule most teams get backwards: "Prioritize volume over quality", because automated grading scales and hand-grading does not.

OpenAI's own worked example corrects Icelandic sentences, scored by BLEU out of 100, and the ranking is not what the usual advice predicts.

MethodBLEUWhat it tells you
GPT-4, zero-shot62The baseline you must beat
GPT-4, three few-shot examples70Examples bought eight points for a longer prompt
GPT-3.5-turbo fine-tuned on 1,000 examples78A smaller fine-tuned model beat the bigger prompted one
GPT-4 fine-tuned on 1,000 examples87The winner, on a task that was behavior all along
GPT-4 fine-tuned, with RAG added83Retrieval cost four points by adding noise

That last row is the argument for measuring. Retrieval is the right answer to a knowledge problem and the wrong answer to a behavior problem, and only the score tells you which you had. The guide's summary is blunt: "squeeze as much accuracy out of basic methods as you can before reaching for more complex RAG or fine-tuning".

Then decide what score is enough, which is a business number. The same guide works a customer service case to a break-even accuracy of 81.5 percent, and says that where a mistake costs a lot, "the human would drive and the AI would function as an assistant". I set that threshold with the client first.

When is it worth combining them?

When your evals show two faults at once. OpenAI calls the methods "additive, not exclusive". The combined build is fine-tuning for behavior plus RAG for context, with one condition: train on examples that already contain retrieved context, "so it isn't learning how to use the context zero-shot". Non-representative training data is one of the most common pitfalls they name.

There is a cheaper middle step people skip. Anthropic's increase output consistency page treats format drift as a prompting problem first: specify the exact format, constrain with examples, ground answers in a fixed information set, and use structured outputs where JSON must match a schema. Much of what gets diagnosed as "we need to fine-tune" is a schema and three examples.

For what it is worth, the AI agents I build for WHMCS, WordPress, SEO and servers, and the chatbots I ship, run on prompt design, retrieval over the client's own content and tool calls, the shape I describe in how to build an AI agent. I sell that work, so treat it as the pattern I reach for, not a rule about your project.

What can go wrong with each one?

Each lever has a failure mode that shows up after launch.

  • Fine-tuning can forget. The Microsoft paper lists catastrophic forgetting among the causes of factual errors, and notes "the inherent instability of fine-tuning". Keep the base model as a comparison.
  • Fine-tuning can expire. OpenAI states that fine-tuned models stay available only until their base model shuts down. Budget for a retrain you did not plan.
  • RAG can retrieve the wrong thing. Failures split in two: wrong context, so the model "can't possibly answer", or too much irrelevant context, which "drowns out the real information and causes hallucinations".
  • A long prompt is not a free substitute. Models "can struggle to maintain attention across very large prompts with complex instructions", so evaluate at several context sizes rather than trusting a big window.

What I will not do is fine-tune a model so it can recite a price list. That is a knowledge problem with a document behind it, and the document will change before the training job ends.

What else do people ask about fine-tuning, RAG and prompt engineering?

Which is better, fine-tuning or RAG?

Neither, until you say what is broken. For getting facts into answers the comparisons favor RAG: in the Microsoft experiments retrieval was consistently better than fine-tuning alone, on known and new knowledge alike. For a consistent format, tone or task behavior, fine-tuning won OpenAI's Icelandic example outright, and adding retrieval there made it worse.

Can you use RAG and fine-tuning together?

Yes, and that is normal when you have both a knowledge gap and a behavior gap. The condition is that training examples must look like production, so a model headed for a RAG pipeline should be trained on examples that already include retrieved passages. Without that step it can score below RAG alone.

The published answer is no. Soudani, Kanoulas and Hasibi found RAG ahead "by a large margin particularly for least popular factual knowledge", while fine-tuning helped but demanded far more compute and synthetic data. Rare entities are the worst case for weights and the best case for a search index.

Is prompt engineering enough on its own?

Often, yes. OpenAI says prompt engineering "is often the only method needed" for cases like summarization, translation and code generation, and that many of its largest deployments used only prompt engineering and RAG. The limit is scale: wider knowledge than fits in context, or more consistency than a few examples can hold.

To decide this on your data rather than in the abstract, I take it as AI agent development work, or you can hire me as an AI developer by the hour, with a free scoping call of about 30 minutes and a written quote within one business day.

Last reviewed 14 November 2026. Disclosure: I sell AI development work, including chatbots and agents, so I am not neutral. I am CTO at ElySpace, a freelance developer for Marmaids LLC, and founder of WHMCSPilot and MagizAI, an AI chat product of my own. All quotations and figures come from the pages linked above, read on 14 November 2026.

Share this article

Shahid Malla

Written by

Shahid Malla

Freelance WHMCS developer and web designer with 13+ years of client work. CTO at ElySpace, freelance developer for Marmaids LLC, and founder of WHMCSPilot and MagizAI.

Got a project like this?

Tell me what you need and I will send a written quote within one business day.