Supervised Fine-Tuning

You’ve already seen how a model is pretrained on vast amounts of text, but a pretrained model is like a brilliant generalist who has read everything but never learned how to behave. Supervised fine-tuning is how you turn that raw knowledge into something that can actually answer questions, follow instructions, and speak in a consistent style. It is the first and most important stage of post-training — every other alignment technique builds on what you do here. Master this and you understand why ChatGPT feels like a helpful assistant rather than a text-completion machine; understand that, and you can read about preference optimisation, RLHF, and DPO with an actual foundation. Without SFT, those later stages have nothing to work with: you’re teaching manners to a student who hasn’t yet learned to speak.

Think of a pretrained language model as someone who has devoured every book, article, and conversation in the library but has never had a job. They know how language works, but they don’t know when to say what, how to format a list, or how to be concise when asked for a short answer. They might respond to “What is the capital of France?” with an essay on French history, because that’s how they’ve seen text flow. Supervised fine-tuning is like an apprenticeship: you show them thousands of examples of the exact behaviour you want — a question followed by a good answer, a prompt followed by a well-formatted response — and they learn to mimic that pattern. The key word is “demonstrations”: you aren’t giving them rules, you’re giving them examples. The model adjusts its internal numbers so that when it sees a similar prompt, it produces a similar response. You’re not teaching it new facts; you’re teaching it how to behave with the facts it already knows. It learns format, style, tone, and the rhythm of a good answer — as much as the content itself. Crucially, the quality of those demonstrations matters more than their quantity: a thousand excellent examples beat a million sloppy ones, because the model will copy your mistakes along with your successes.

Imagine you’re training a new employee at a company. You don’t give them a manual of rules — that would be too rigid and wouldn’t cover every situation. Instead, you sit them down and walk them through dozens of past customer interactions: here’s a customer asking about a refund, here’s the response that worked, here’s one that didn’t. The employee learns not just what to say, but how to say it: the tone, the structure, the way to ask clarifying questions. That’s supervised fine-tuning. The employee starts with a general knowledge of language and the world from their education (that’s pretraining), and you shape their behaviour with examples. But the analogy has limits: an employee can actively reason about why a response worked and adapt to new situations; a fine-tuned model doesn’t ‘understand’ the demonstrations — it just adjusts its probability of producing certain words. Also, an employee can ask follow-up questions; a model can only learn from what you show it. If your examples are all for one type of customer, your employee will be great with that type and useless with others. The quality and diversity of your demonstrations directly determine the model’s behaviour, for better or worse.

Supervised Fine-Tuning (SFT) is the process of taking a pretrained language model and further training it on a curated dataset of prompt–response pairs — demonstrations of desired behaviour — so that it learns to produce similar responses to similar prompts.

You haven’t learned the formal concepts yet, but you know that pretraining gives a model general language ability, and that a model’s output is a probability over words. SFT sits between pretraining and preference optimisation: it’s the first stage of post-training. Your library notes call it “the first and most important stage of post-training,” and that the stack is layered: SFT first, then preference methods. SFT also connects to fine-tuning more generally — the idea of taking a general model and adapting it to a specific task — except that ‘supervised’ means the training data includes the correct answer for each prompt. The demonstrations you use are like the preference data in later stages, but here you only show the model what good looks like, not what bad looks like.