Temperature and Sampling

This is the dial that decides whether a chatbot sounds like a careful librarian or a creative storyteller. It directly controls the unpredictability of every AI system that generates text — from ChatGPT to image-captioning tools. Understanding temperature and sampling lets you: make a model give the same answer every time (important for customer support or medical advice), or make it give fresh, varied answers (important for creative writing or brainstorming). It also unlocks your next topics: LLM Nondeterminism (why you can't always get the same output) and Multi-Token Decoding (how models generate faster). Without this, you're just pressing a button and hoping; with it, you're the one choosing the behaviour.

Every time a language model writes the next word, it doesn't just pick the most likely word. It has a list of all possible next words, each with a probability — like a weather forecast: 'sunny 80%, cloudy 15%, rain 5%'. The model then 'rolls a dice' weighted by those percentages. If it rolls sunny most of the time, the text is predictable and safe. But you can change the dice. Temperature is a single number that reshapes those percentages. Low temperature (like 0.1) makes the most likely word almost certain to be chosen — the text becomes repetitive and safe. High temperature (like 1.5) flattens the percentages — even the 'rain' word gets a real chance, so the text becomes surprising, creative, and sometimes nonsense. Top-k and top-p are different knobs: they don't change the percentages, they cut off the list. Top-k says 'only consider the top 10 words, ignore the rest'. Top-p says 'keep adding words from the top until their combined probability reaches 0.9 — so if the top 3 words have 90% probability, ignore all the rest'. Temperature controls the shape of the dice, while top-k and top-p decide how many sides the dice even has.

Think of a language model as a street-food vendor who has to choose one dish to cook next from a menu of a hundred possible dishes. Each dish has a popularity score (probability). A 'greedy' vendor always picks the most popular dish (the most likely word) — that's temperature 0. A creative vendor wants to surprise customers, so they sometimes pick a less popular dish — that's high temperature. But they don't want to serve something nobody likes (like a word that makes no sense), so they limit the menu: top-k says 'only consider the 10 most popular dishes, pick randomly among those'. Top-p says 'keep the dishes that together cover 90% of customer demand — maybe that's the top 3, maybe the top 20, depending on how popularity is spread'. Temperature is how much you let the vendor deviate from the crowd; top-k and top-p are how many dishes are even on the table. Where the analogy breaks down: a real vendor has taste and experience, but the model has no understanding of the meaning of the words — it's purely a statistical game. Also, temperature doesn't add new knowledge; it just changes how often the model picks rare words, not what it knows.

Temperature and sampling are decoding controls that shape how the next token is drawn from the predicted probability distribution — temperature sharpens or flattens the probabilities, while top-k and top-p truncate the list of candidates before sampling from it.

You already know that a language model predicts the next token — that's the core of Large Language Models in your notes. You also know that LLM Nondeterminism means the same prompt can give different outputs — temperature is the main knob that controls how different those outputs can be. Your notes on Context Window tell you that the model's memory is only the tokens it sees; temperature decides how it picks the next one from that memory. This topic sits right after Sliding Window Sampling (how training examples are made) and Byte-Pair Encoding (how words become tokens) — those are about the input side, while temperature is about the output side. It also connects to Reasoning Models (they trade latency for accuracy) — and temperature affects how 'careful' the reasoning looks. In your book 'A Common-Sense Guide to AI Engineering', temperature is described as the 'creativity dial' on the API — and top-p as 'nucleus sampling' because it packs the probabilities into a nucleus of words.