Reasoning Models

This concept is the gateway to understanding how modern AI solves hard problems — problems that need multiple steps of thinking rather than a single guess. Once you master this, you'll be able to follow how models like ChatGPT can solve math word problems, write multi-step code, or plan a trip, and you'll know why sometimes it's better to ask a cheap model quickly than to pay for a thinking model. It also sets you up to understand the next topics in your library: prompt engineering (how to ask the model to think step by step), retrieval-augmented generation (when to look up facts instead of reasoning), and AI agents that plan and call tools in loops. Without this, those topics will feel like magic. With it, they'll feel like engineering.

Think of two kinds of people you ask for advice. The first gives you an answer instantly, based on a gut feeling. If you ask them a simple question — 'what's 2+2?' — they're great. But if you ask them a complex one — 'should I buy a house now or wait five years?' — they might give you a confident but poorly thought-out answer. The second person doesn't answer right away. They say 'let me think', then they work through the problem step by step: income, savings, interest rates, inflation, job stability, and so on. Only after they've reasoned through all the steps do they give you an answer. The second person is like a reasoning model. It has been trained to spend extra time (and extra computational effort, called 'compute') to produce a chain of intermediate thoughts before arriving at the final answer. This extra 'thinking' dramatically improves performance on problems that require multiple logical steps, like solving a math puzzle, writing a piece of code that has to coordinate several functions, or planning a route with many constraints. The trade-off is that it costs more — more time and more money (because you pay for the tokens it uses to write its thoughts). So the key skill in using these models is knowing when to use them. For simple questions, a quick model is cheaper and just as good. For hard ones, a reasoning model is worth the cost. This concept teaches you how that trade-off works and how to make that decision.

Here's an analogy carried as far as it goes: imagine a mathematics teacher who has two ways of marking an exam. In the first way, she reads the final answer and gives a mark based only on whether it's right or wrong. In the second way, she requires every student to show their working — every step of the calculation — and she gives partial credit for correct intermediate steps, even if the final answer is wrong. The second system is more expensive (it takes her longer to mark), but it does two things: first, it encourages students to write down their steps, which forces them to think more carefully; second, it gives the teacher much more information about where the student went wrong. Now, imagine a model trained the same way. Instead of just predicting the final answer, it's trained to produce a chain of reasoning tokens first, and that chain is rewarded during training if it leads to the correct final answer. The model learns that writing out 'let's think step by step' actually improves its chance of getting the answer right. Where the analogy breaks down: a human teacher can check the logic of each step, but an AI reasoning model has no such external check — it's self-taught, so its 'reasoning' can sometimes be a chain of confabulations that look plausible but lead to a wrong answer. Also, the extra tokens are not just a scaffold — they are the very substance of the model's output, and they carry a real monetary cost. So unlike a teacher, the model doesn't care about showing its work for the sake of the reader; it only cares about producing a chain that statistically improves its final prediction.

A reasoning model is an AI system trained to produce a series of intermediate reasoning tokens (a 'chain of thought') before giving its final answer, trading extra computational cost and latency for higher accuracy on problems that require multiple logical steps.

This concept sits inside your library's parent concept, Large Language Models (LLMs). LLMs are next-token predictors — they just guess the most likely next word. Reasoning models are a special kind of LLM, one that has been trained to use its next-token predictions to generate a 'thinking' process before the final answer. The key idea from your notes on LLMs that matters here is the 'context window'. The model's entire working memory for a call is the context window — everything it sees, including the chain of thought it generates, must fit inside that window. If the chain gets too long, it runs out of room and may forget the original question. Also, your notes on temperature and sampling are relevant: reasoning models are often run at low temperature to make the chain of thought more deterministic, but the sampling process is still stochastic, so even the same prompt can give a different chain each time. Finally, this connects to the quality-cost-latency tradeoff from your notes: reasoning models are deliberately trading cost and latency for quality on hard tasks. The neighbouring notes you have on 'LLM Nondeterminism' and 'Context Window' are both constraining factors you'll need to manage when using reasoning models.