Prompting, RAG or fine-tuning: which one does your AI project need?

Your AI pilot works, mostly. The answers are too long, the tone is off, and it doesn’t know your products or your prices. Then someone in the meeting says: “We need to train it on our data.”

Usually you don’t. There are three ways to adapt a general AI model to your company, and each fixes a different problem:

• Prompting: better instructions and examples, sent with every request.
• RAG (retrieval-augmented generation): looking up your documents at question time and giving the model the relevant parts.
• Fine-tuning: training the model further on your own examples, so its habits change.

Picking the wrong one is expensive. Fine-tuning a model so it learns your price list, for example, takes weeks and still gives you wrong prices. This post explains what each option changes, what it costs and how to decide.

Three ways to adapt a model

Prompting: say it clearly, show examples

The prompt is everything you send the model along with the user’s request: who it works for, what the task is, what format you want, what to avoid, and a few examples of good output. Many disappointing pilots run on a one-line prompt. A page of clear instructions and three good examples often fixes tone, length and format in an afternoon.

Prompting changes behaviour, one request at a time. It costs hours rather than weeks, and you can change it whenever you like. The limit: you can’t paste your whole company into every request, and long prompts cost more on every call. Our guide to writing prompts that work covers the practical side.

RAG: give it the right facts at the right moment

With RAG, the system searches your documents for passages relevant to the question and puts them into the prompt. The model answers from those passages and can cite them. When a document changes, the next answer uses the new version. RAG adds knowledge; it doesn’t change how the model writes. We explain it step by step in RAG explained.

Fine-tuning: change its habits

Fine-tuning means training an existing model further on hundreds or thousands of examples: an input, and the output you want for it. The result is a variant of the model that does one job your way without long instructions. It changes behaviour, not knowledge. The model gets better at a format, a style or a narrow decision. It is a poor way to teach facts: it can’t tell you where a fact came from, and the facts start going stale the day you train it.

Prompting, RAG and fine-tuning

	Prompting	RAG	Fine-tuning
What it changes	How it answers, per request	What it knows, per question	How it answers, built in
Adds your facts	Partly	Yes	No
Stays up to date	Yes	Yes	No
Can show sources	Partly	Yes	No
Time to first result	Hours to days	Weeks	Weeks to months
What you need	Clear instructions, a few examples	Documents someone maintains	Hundreds or thousands of good examples
Easy to change later	Yes	Yes	No

Knowledge or behaviour?

The most useful question is whether your problem is about what the model knows or how it behaves.

• “It doesn’t know our return policy” is a knowledge problem. Use RAG.
• “It writes three paragraphs when we want three bullet points” is a behaviour problem. Fix the prompt first.
• “It has to sort 40,000 tickets a month into 30 categories, quickly and cheaply” is a behaviour problem at volume. Start with the prompt; fine-tune a smaller model only if the prompt isn’t accurate enough or costs too much at that volume.

Which problem do you have?

Needs special behaviour →

Prompting; fine-tune only at volume

RAG plus a strong prompt

A good prompt is enough

RAG

• 1 Summarise meeting notes
• 2 Rewrite emails in house style
• 3 Sort 40,000 tickets a month
• 4 Answer HR policy questions
• 5 Draft quotes from the price list
• 6 Support replies from the help centre

Needs your company’s knowledge →

1. 1 Summarise meeting notes
2. 2 Rewrite emails in house style
3. 3 Sort 40,000 tickets a month
4. 4 Answer HR policy questions
5. 5 Draft quotes from the price list
6. 6 Support replies from the help centre

What each one really costs

We won’t quote per-token prices here; they change too often. The shape of the costs is stable:

• Prompting costs a few hours of someone who knows the work, plus a little more per request because the prompt is longer. At very high volume, that extra per-request cost starts to matter.
• RAG costs a build: connecting the sources, cleaning documents, setting up search, permissions and tests, typically a few weeks. Then it has running costs: a search index, re-indexing when documents change, and someone who owns the sources.
• Fine-tuning costs mostly in data. Collecting and checking hundreds or thousands of correct examples takes far longer than the training itself. Then come evaluation, hosting, and doing it all again when the base model is retired or your process changes.

The vendor side moves too. OpenAI has announced that it is winding down its self-serve fine-tuning platform: it no longer accepts new users, and existing fine-tuned models keep running until their base models are retired. Fine-tuning remains possible with open-weight models such as Llama, Mistral or Qwen, usually with techniques like LoRA that train a small add-on instead of the whole model, and on some cloud platforms. If you go that way, you also take on hosting; see self-hosted LLMs or API models. For the full picture, read what an AI project really costs.

Effort and commitment

1. Rewrite the prompt Instructions, format, tone 1
2. Add examples Three to ten good outputs 2
3. RAG on your documents Search, permissions, tests 3
4. Fine-tune and host a model Data, training, evaluation, hosting, repeat 4

← An afternoon An ongoing project →

1. 1 Rewrite the prompt Instructions, format, tone
2. 2 Add examples Three to ten good outputs
3. 3 RAG on your documents Search, permissions, tests
4. 4 Fine-tune and host a model Data, training, evaluation, hosting, repeat

When fine-tuning is the right call

Fine-tuning earns its place in a few situations:

• High-volume, narrow tasks. Sorting tickets, tagging documents or pulling the same fields from the same kind of form, tens of thousands of times a month. A small fine-tuned model can often come close to a large model on that one task, at a lower cost per request and with faster answers.
• A strict format or style that prompts don’t hold. When even long instructions and good examples leave too many outputs off format.
• Private or offline setups. When a small open-weight model has to run on your own servers, fine-tuning can make it good enough at its one job.

It is the wrong call when you want the model to know facts, when you have fewer than a few hundred good examples, when the task changes every few months, or when nobody has yet tried a careful prompt.

A simple decision path

Which one, in which order

1. 01 Test set first 30–100 real cases with the right answer
2. 02 Better prompt Clear instructions and examples
3. 03 Missing facts? Add RAG over your documents
4. 04 Still off at volume? Consider fine-tuning a smaller model
5. 05 Measure The same test set after every change

The test set is the step most teams skip, and it is the one that makes everything else work. Without it, you can’t tell whether a new prompt, RAG or a fine-tuned model actually did better. OpenAI’s own guide to model optimisation also starts by writing evaluations before changing anything.

Combining them

In production, most systems use at least two of the three. A typical customer support setup uses a careful prompt for tone and format, and RAG over the help centre and past tickets for facts. Only if the volume justifies it does a small fine-tuned model take over the first step, sorting incoming tickets. Each part does the job it is good at.

A worked example

Here is an illustrative case. A building-materials distributor receives a few hundred order emails a day. It wants AI to read each one and prepare a draft order in the ERP for a person to confirm.

1. First attempt. A one-line prompt: “Extract the order from this email.” The model returns free text, mixes up boxes and square metres, and often misses the customer number.
2. Better prompt. A fixed output format with named fields, rules for units, and five real emails with the correct output as examples. Most format problems go away. That is a behaviour fix, done in a day.
3. Missing knowledge. Product codes are still wrong, because the model has never seen the catalogue. Customers write “the grey 30×60 tiles”, not “TL-3060-GR”. This is a knowledge problem, so the system looks up likely matches in the product list and gives them to the model with the email. Now it picks from real codes instead of inventing them.
4. Fine-tuning? With a good prompt and the catalogue lookup, the errors that remain are unusual emails that a person should check anyway. At a few hundred emails a day, fine-tuning wouldn’t pay for the data work. It would be worth another look only if volume grew many times over, or if the company had to move to a small model on its own servers.

Common mistakes

• Fine-tuning to add knowledge. The model picks up the style of your documents and still invents the details.
• No test set. Every change is judged on three examples someone tried on a Friday afternoon.
• Skipping the prompt work. Comparing a one-line prompt with a fine-tuned model isn’t a fair test.
• Training on messy examples. The model learns your inconsistencies along with your style.
• Forgetting the upkeep. Fine-tuned models need retraining when the base model is retired; RAG indexes need an owner.

What to do next

1. Write down the three things the current output gets wrong.
2. Sort each one: knowledge (facts it doesn’t have) or behaviour (how it answers).
3. Collect 30–100 real examples with the right answers.
4. Fix behaviour with the prompt and knowledge with RAG, and measure on your examples.
5. Talk about fine-tuning only if a high-volume task still falls short.

If you are not sure which of these your project needs, we can look at it together in a 30-minute call.