The Steering Test

Before you spend days (or weeks) fine-tuning a model, you need to know whether fine-tuning is even necessary. The Steering Test is the decision rule that separates two very different situations: when you can get the behaviour you need by changing what you say to the model (a prompt, some extra context, or pulling in the right documents), and when you genuinely need to change the model itself. Mastering this saves you time, money, and the hidden maintenance cost of a tuned model. It also connects directly to neighbouring ideas you'll meet soon: catastrophic forgetting (why you don't want to tune unless you must), chat format templates (why steering often starts with the right system prompt), retrieval-augmented generation (steering by giving the model the facts in the prompt), and PEFT/LoRA (the cheaper fine-tuning options if the test says you do need to tune).

Think of a large language model as a very knowledgeable, very literal assistant who has read a lot but has no memory of your specific situation. You can get it to do a lot by what you say to it — the prompt, any background you give, and any documents you hand over. The Steering Test is the question you ask yourself: can I get the behaviour I need by steering the model's attention with text, or do I need to change the model's underlying knowledge or habits? The test is a practical heuristic, not a hard theorem. It's like deciding whether to hire a new employee or just give your current one clearer instructions. If the issue is that the employee doesn't know the rules of your office, a clear briefing sheet (a prompt) might be enough. If the employee fundamentally doesn't understand your domain's language, you need training. The test forces you to name which kind of problem you have.

You're a manager at a busy customer service desk. A new assistant has arrived, smart but fresh from a different company. The assistant knows general customer service (how to greet people, how to handle returns, the tone of voice), but doesn't know your company's specific products, policies, or the quirks of your inventory. You could either: (a) write a detailed briefing document that the assistant reads before each call — that's your prompt and context; (b) give the assistant a searchable database of product specs and policies to look up during a call — that's retrieval; or (c) spend weeks retraining the assistant to think like your company — that's fine-tuning. The Steering Test asks: does the behaviour I need come from better instructions or from new knowledge? If the assistant already knows the general rules and just needs the specifics — steering works. If the assistant doesn't know how to handle a situation that isn't in any document — you need to change the assistant, and that's fine-tuning. But the analogy breaks down in one important way: a human assistant can remember your briefing document and apply it implicitly, while a model's context window has a hard limit — you can't give it the entire internet in a prompt. Fine-tuning also changes the model's default behaviour in a way that a prompt never can — it becomes the model's 'personality'. And fine-tuning carries a real risk: catastrophic forgetting, where the model loses some of its general abilities because it has focused so much on your narrow task. Steering never risks that.

The Steering Test is a decision heuristic: if you can achieve the behaviour you need by steering with prompts, context, or retrieval, then you do not need to fine-tune.

This concept sits at the very start of your journey into fine-tuning. It's the entry point to the broader topic 'Fine-Tuning' in your library. You haven't studied any prerequisites yet, but this concept builds on the basic idea of a large language model that you can talk to. It connects forward to several neighbours: 'Catastrophic Forgetting' — the main risk you avoid by not fine-tuning unless forced; 'Chat Format Templates' — the system prompt is often your first steering tool, and it must be in the model's expected format to work; 'Retrieval-Augmented Generation' — giving the model facts via retrieval is a form of steering; and 'PEFT' / 'LoRA and QLoRA' — the cheaper fine-tuning methods you'd consider only if the test says you must. It also relates to 'Classification Fine-Tuning' and 'Instruction Tuning' — both are fine-tuning, but the test helps you see that instruction tuning (teaching the model to follow directions) is often more about behaviour than knowledge, which you might be able to steer instead.