Prompting, RAG, and fine-tuning

“We’ll train it on our data” covers three different pieces of engineering, and only one of them is training. The first is rewriting the text the model reads before it answers. The second is wiring a document search to the model, so the passages that bear on the question arrive alongside it. The third is retraining the model itself on curated examples.

Prompting, retrieval and fine-tuning. They cost different amounts, take different amounts of time, and fix different problems. Bundled under one phrase they get picked by accident, and picking the right combination is much of the real engineering work in any business AI system.

Prompting

Prompting shapes what the model does by changing the text put in front of it. The prompt is everything the model sees before it produces a response, and better prompts produce better output from the same model. Why prompts matter covers what a good one carries and how careful prompters work.

The prompt sets the tone and style of the output, the format (a list, a paragraph, JSON, a table), the level of detail, whether the model follows a set procedure or a worked example, what it should and should not do, and the role it adopts for the response.

It costs almost nothing. Rewriting a prompt is text editing, and iterating is immediate: change the wording, see the effect. A longer prompt does add to the bill, since models are charged by the token, the small chunk of text they read and write in, and next to the engineering behind the other two that barely registers. The real cost is the time a team spends crafting prompts that work, and the more often a prompt gets reused, the better that time pays back.

Prompting is the natural first move, and for most everyday business needs a well-crafted prompt gets most of the way there. It suits the tasks a model already handles, drafting, summarising, classifying, pulling fields out of text, and cases where the inputs change every time but the shape of the task does not.

The prompt has to fit inside the context window. A prompt cannot add knowledge the model does not have; it steers what is already there, and no rewording will make a model know last quarter’s revenue unless the number is in the prompt. A prompt also cannot push the model past what it is capable of. If it cannot reliably produce a structure, better wording will not create the ability.

Prompt engineering is mostly dull craftsmanship: be specific, show an example, name the format, run it again. The mystique of it as a high-skill profession was always overblown. It is a useful skill, learnable in a few days, that compounds over months as a team builds up a library of prompts that reliably work.

Retrieval (RAG)

Retrieval fetches the passages that bear on the question and pastes them into the prompt, so the model answers from what is in front of it rather than from what it absorbed in training. The name is RAG, retrieval-augmented generation. Tools and memory walks the 5 steps of the mechanism and shows the whole path running; what matters here is when to reach for it.

Retrieval decides which facts the model has in hand for a specific answer, whether the answer can cite the documents it came from, and how current the information is. The document store sits under the business’s own control and can be updated daily, hourly, or as changes land.

The costs are engineering and upkeep. Someone builds the pipeline that indexes documents, the search index itself, the retrieval logic and the citation handling. As documents change the index has to keep up, and stale retrieval is worse than no retrieval. Retrieved passages also sit in the prompt and are paid for on every question, so more material retrieved means more cost per answer.

Reach for retrieval when the model needs to know things specific to one business: its documents, customer history, product specs, policies. It fits information that changes often, and answers that have to be checkable against a source, which covers compliance, support, and anywhere someone will ask where that came from. Most “ask questions of our knowledge base” products are this shape.

Example

Let’s say a support team wants answers taken from its own policy pages, and those pages change most weeks. Retraining a model on every change is out of the question. Reindexing the pages that changed takes one overnight run, and the next morning’s answers quote the new wording.

The quality of the answer is the quality of the retrieval. When the wrong passages come back the model writes a confident wrong answer that looks exactly like a right one, which is where most production systems actually fail. The model still has to interpret what it gets, so contradictory documents produce contradictory answers. And retrieval does not raise the model’s capability. It hands the model better material. A weak model with good retrieval is still a weak model.

Fine-tuning

Fine-tuning retrains the model on examples specific to a domain, so the model itself behaves differently from then on. The loop is the same one described in how AI models are trained, show examples and adjust weights, run on top of an already-trained base model with domain-specific examples.

It changes how the model behaves on every call rather than per prompt: its tone, structure, style, how closely it holds to a format. It teaches patterns specific to one domain, such as house jargon, industry abbreviations, or the way a particular customer base phrases things. And it makes that behaviour compact and repeatable where the alternative is a very long prompt every time.

It is expensive. It needs curated training data, typically hundreds to thousands of good input and output pairs, and producing that data is a project in itself. Training costs real money, though far less than pre-training a base model. Someone has to run the pipeline, evaluate the result and decide when to do it again. And a fine-tune is tied to one base model, so when a meaningfully better base ships, the fine-tune is sitting on an outdated foundation and has to be redone.

Fine-tuning earns its place on the same task run thousands or millions of times, where paying for long prompted examples on every call stops being absorbable, on a specific output format that is awkward to enforce with prompts alone, and on behaviour the base model genuinely cannot produce with prompting and retrieval, after those two have actually been exhausted rather than assumed.

It does not fit low-volume or one-off work, where the setup cost never gets recovered, or situations without curated data, where fine-tuning on noisy examples produces a model worse than the base it started from. And it does not fit adding knowledge. That is retrieval’s job.

Example

Let’s say a firm pays to fine-tune a model on 5 years of its own project documents, expecting the model to know its clients. What comes back writes in the house style and still invents client details. Fine-tuning changed how the model writes, not what it can look up. The gap was knowledge, so the answer was retrieval.

A model tuned narrowly sometimes gets worse at general tasks it used to handle well. And the fine-tune needs redoing as the data evolves and as base models improve, which is an ongoing cost rather than a one-time expense.

How the three combine in real systems

Production systems typically run a mix.

• Prompting is always there. Even a fine-tuned model with retrieval still needs a prompt. Prompting steers every individual request.
• Retrieval is in roughly every system that answers questions about a company’s own information. Without it there is no business-specific knowledge in the answer.
• Fine-tuning is the rarest. Most production systems never use it. Where they do, it is usually a high-volume specific task where the other two ran out of room.

A base model with good prompting and retrieval covers what most mid-sized companies need from a business AI system. Fine-tuning is an optimisation that earns its place at volume, once what is being optimised is clear. Two of the 3 change what the model is given rather than the model itself, which is where most of an AI product is built.

The natural sequence

Most well-built systems converge on the same order, for the same reason: each step is much cheaper than the next.

1. Prompting alone. Iterate on a genuinely good prompt for a week. The ceiling here is higher than most people expect.
2. Add retrieval when there is a knowledge gap. The model needs facts specific to one business that it was never shown.
3. Consider fine-tuning only after the first two. Specifically when the gap is in behaviour: a format, tone or structure that prompting and retrieval cannot reliably produce.

The order is not a rule, but it is robust. Skipping to fine-tuning before exhausting prompting and retrieval is the single most common expensive mistake in business AI projects.

How these get confused

“Custom AI trained on the business’s data.” The phrase covers all three indiscriminately. In most real implementations the “training” is retrieval: documents indexed and fetched at the moment of the question, not weights being updated. Which mechanism is underneath changes the cost, the timeline, and what the system can and cannot do.

“Fine-tuning to teach the model our knowledge.” Fine-tuning is poor at adding specific knowledge. Retrieval is the tool for that. The two get conflated constantly, including by people building these systems.

“It’s not RAG, it’s something better.” Sometimes true. There are newer retrieval architectures, graph-based, hybrid and agentic, that genuinely outperform classic vector RAG on certain workloads. Often, though, it is the same mechanism with a rebrand. The honest test: if it fetches relevant context at the moment of the question, it is in the RAG family, whatever the marketing says.

“We’ve built our own model.” Almost always means a layer of prompting, retrieval and possibly fine-tuning on top of an existing foundation model from one of the big labs or open-weights families. Building a frontier model from scratch costs hundreds of millions of dollars and is done by a handful of labs. Everyone else is composing on top.

Comparison at a glance

	Prompting	Retrieval (RAG)	Fine-tuning
What it changes	One request’s output	What the model sees per request	The model itself
Cost	Near zero	Moderate: engineering plus upkeep	High: data, compute, maintenance
Speed to iterate	Seconds	Hours to days	Days to weeks
Best for	Steering general behaviour	Adding business-specific knowledge	High-volume specific behaviour
Worst for	Facts the model was never shown	Changing how the model behaves	Adding facts
First move?	Yes, always	Yes, when there is a knowledge gap	Almost never