The Same Conversation, Three Times a Month
A business owner tells us the AI does not know anything about their company, and asks what it would cost to train it on their data. It is a completely reasonable question. It is also, nine times out of ten, the wrong thing to want — and the version they actually need is cheaper, faster, and correctable by editing a document.
There are three ways to make an AI system work with information it did not learn during training. They get talked about as though they were competing options at different price points. They are not. They solve different problems, and picking the wrong one is expensive in a way that is not obvious until several months in.
The Three Options, Briefly
Better prompting
Give the model the information and the instructions it needs, in the message itself. No system to build, no infrastructure. You paste the context, or you write a reusable template that includes it.
Retrieval (RAG)
Index your documents. When a question comes in, find the passages that relate to it and put those into the prompt automatically before the model answers. The model still learns nothing — it just gets handed the right page each time.
Fine-tuning
Further train the model on your own examples so that it internalises a style, a format, or a way of responding. This changes the model itself, which is both the appeal and the problem.
Why Fine-Tuning Is Almost Never the Answer for Facts
Say you fine-tune a model on your product catalogue. It will get very good at sounding like your catalogue: the phrasing, the structure, the way you describe things. What it will not do is reliably tell a customer the current price of item 4471, because the information got smeared into the weights along with everything else rather than stored anywhere you can point at.
Worse, when the price changes, there is nothing to edit. You retrain, or you live with the wrong answer. Every fact you bake in is a fact you have committed to maintaining through a process that costs money and takes time. For a business whose information changes — which is every business — that compounds badly.
Where each one genuinely earns its place:
Retrieval is right when
- The answer lives in documents you can point at
- Those documents change, and corrections must be immediate
- You need to know which source an answer came from
- Being wrong about a fact has consequences
- The volume of material exceeds what fits in one prompt
Fine-tuning is right when
- You need a consistent style or format, not facts
- The task is narrow and you have hundreds of good examples
- Prompting gets you close but not reliably enough
- You are trying to shrink cost by making a small model behave like a big one
- The behaviour you want is hard to describe but easy to demonstrate
Start With Prompting, Genuinely
This is the recommendation people skip past, because it sounds like being told to try turning it off and on again. But a startling share of "we need a custom AI system" conversations end when someone writes a proper instruction with an example attached.
Most disappointing AI output comes from a one-line prompt with no context, no example of what good looks like, and no statement of who the output is for. Giving the model your best previous version of the same document changes results more than any other single adjustment — more than model choice, more than clever phrasing, considerably more than infrastructure.
Prompting stops being enough at a specific and recognisable point: when the material you would need to paste is too large to paste, or when it changes too often for anyone to keep pasting the current version. That is the moment retrieval starts to make sense, and not really before.
“How much do you charge if I live outside town?”
- 1
Retrieve
Search the knowledge base for passages that look related to the question.
- 2
Augment
Paste those passages into the prompt, above the original question.
- 3
Generate
The model answers using the passages, not its training data.
The Decision, as a Table
| If your problem is | Reach for | Because |
|---|---|---|
| The output is the wrong shape, tone, or length | Prompting | You have not told it what you want. Add an example of good output. |
| It does not know a fact about your business | Retrieval | The fact needs to be looked up at question time, and corrected by editing a file. |
| It knows the fact but keeps phrasing it badly | Prompting, then fine-tuning | Style is what fine-tuning is actually for — but exhaust prompting first. |
| Answers are inconsistent across runs | Prompting | Usually an under-specified instruction. Tighten it before blaming the model. |
| The material is far too big to send | Retrieval | This is the problem retrieval was invented to solve. |
| It works, but costs too much at volume | Fine-tuning a smaller model | A legitimate and under-used case: teach a cheap model one narrow job well. |
What Retrieval Costs, Honestly
Retrieval is the right answer often enough that it is worth being straight about the effort. The technology is not the hard part and has not been for a while. The hard part is your documents.
What actually determines the cost of a retrieval project:
- Whether your policies and procedures exist as current, written files, or live in three people's heads
- Whether there is one version of each document or four, and whether anyone knows which is current
- Whether somebody will own keeping them up to date after launch
- How much of the material is scanned PDFs that need extracting first
- Whether the questions people ask actually have documented answers
If the answers are good, this is a modest project. If the knowledge is undocumented, the real work is writing it down — and that is worth doing regardless of whether any AI ever touches it. Several times we have started a retrieval scoping conversation and finished it by recommending a documentation exercise instead, because there was nothing to retrieve.
The Short Version
The decision rule
- Fine-tuning teaches behaviour. Retrieval supplies facts. Most people want facts and ask for fine-tuning.
- Start with a better prompt and a worked example — it resolves more cases than anyone expects.
- Move to retrieval when the material is too large to paste or changes too fast to keep pasting.
- Consider fine-tuning for style, for narrow repeated tasks, or to make a cheap model do one job well.
- A retrieval project is a documentation project wearing a technical hat. Judge it on your documents, not the tooling.
What is the difference between RAG and fine-tuning?
RAG looks information up at question time and puts it into the prompt, leaving the model unchanged. Fine-tuning further trains the model itself on your examples, changing how it behaves. The practical difference is correction: with RAG you fix a wrong answer by editing a document, and with fine-tuning you fix it by retraining.
Can I train ChatGPT on my company data?
You can fine-tune some models on your examples, but it is usually not what people mean or need. If the goal is for AI to answer questions using your policies, prices, or procedures, retrieval does that job better — it is immediate, correctable, and can tell you which document an answer came from. Fine-tuning is for teaching a style or a narrow repeated behaviour.
How much does it cost to set up RAG for a small business?
It depends far more on the state of your documents than on the technology. If your material already exists as clean, current files, this is a modest project. If the knowledge lives in people's heads and a folder of scanned PDFs, the real work is writing it down — and that is the part worth budgeting for, and worth doing anyway.
Do I need RAG or is prompting enough?
Prompting is enough until one of two things is true: the material you would need to paste is too large to paste, or it changes too often for anyone to keep pasting the current version. Until then, a well-written template with an example attached does the job at zero infrastructure cost.
Is fine-tuning ever the right choice for a small business?
Occasionally, and the strongest case is cost rather than capability: teaching a small, cheap model to do one narrow job as well as an expensive one, where you run that job at high volume. Outside of that, style consistency on a repeated document type is the other real case. Both assume you already have hundreds of good examples of the output you want.