LLM Nondeterminism

When you use ChatGPT or any other large language model in a real product — a chatbot, an email assistant, a code generator — you will notice that asking the same question twice sometimes gives you different answers. This unpredictability is called nondeterminism. Understanding it is essential for three things you will do as an AI builder: (1) testing your application — you cannot write a test that expects a single exact answer if the model can return several, so you need testing strategies that tolerate variability; (2) designing downstream systems — if your app passes the model's output to another function or a database, that function must be able to handle small variations in wording or structure without breaking; and (3) setting user expectations — users will be confused or even angry if they ask twice and get different answers, and part of your job is to design the product so that variation feels natural or is hidden. This concept unlocks the practical skills of writing robust tests, designing prompts that reduce variability when needed, and choosing decoding settings like temperature to control how much variation you get.

A large language model, when you give it a prompt, does not have a single fixed answer in mind. Instead, it works like this: it looks at the words you typed, thinks about all the possible next words that could follow, and assigns a chance (a probability) to each of them. For example, if you type "The capital of France is", the model might say there is a 95% chance the next word is "Paris", a 2% chance it's "a", a 1% chance it's "the", and tiny chances for many other words. Then, when it actually has to produce the next word, it draws one of those words at random, using those chances as weights — so "Paris" comes out most of the time, but occasionally another word slips through. This randomness is not a bug; it is a core feature of how these models are built and trained. Because the model is drawing at random, asking the same question twice can yield different words, and those differences can cascade — a slightly different word at position 5 changes everything the model considers when choosing position 6, and so on. Even if you try to make the model deterministic by setting a temperature of zero (which we'll discuss later), the model's internal operations on a computer involve floating-point arithmetic and parallel processing, which can produce tiny numerical differences from one run to the next, especially when the model is served on a GPU or across multiple devices. So, complete determinism is almost never achievable in practice. This means that when you build anything on top of an LLM, you must design it to handle the fact that the output is a distribution, not a single fixed string.

Think of asking a question to a group of experts. You have a panel of five experts in front of you, and you ask each one the same question: "What is the best way to learn French?" Each expert has seen different books, has different experiences, and weighs things differently. One says "immersion in France", another says "daily practice with an app", a third says "take a structured course". They are all reasonable, but they give different answers. If you ask the same group the same question the next day, they might give the same answers again — or they might change their minds slightly based on what they read last night. This group of experts is a loose metaphor for a language model: it has many internal 'parts' that have been trained on different data, and the final answer is an aggregation of many possible routes. When you ask twice, the model 'draws' a route at random each time, so you get different yet reasonable answers. The metaphor breaks down in an important way: the experts have fixed knowledge and preferences, whereas the model's randomness is not because it changes its mind but because it literally samples from a probability distribution over words. The experts are deterministic (they have a fixed answer if you ask them identically), but the model is stochastic. So, the analogy helps you understand that multiple answers can be valid, but the cause is not changing opinions — it's random sampling. Another break: the group of experts can be asked the same question and give the same answer twice if you force them to 'pick the same as before', whereas the model cannot be forced to do that reliably.

LLM nondeterminism is the property that a language model, given the same input prompt, can produce different output sequences on different runs, because the model's decoding process involves stochastic sampling from a probability distribution and, even when sampling is made deterministic, the underlying computation is subject to floating-point and hardware non-determinism.

You are learning this before any other topic, which is perfect. This concept is the foundation for everything else you will study about large language models. It directly connects to 'Temperature and Sampling' — that is the knob you turn to control how much randomness you get. It also relates to 'Tokenization' and 'Context Window' because the model's randomness operates at the level of tokens (pieces of words) inside the context window. And later, when you learn about 'Retrieval-Augmented Generation' and 'AI Agents', you will see that these systems are built to cope with the fact that the model is nondeterministic — they often need to call the model multiple times and compare or aggregate results because a single call might not be reliable. Understanding nondeterminism now will make those topics much easier to grasp. Your library notes say: 'Deterministic output is never fully guaranteed, even at temperature zero' and 'Testing must accommodate variability rather than assume exact matches' — these are the practical takeaways you will carry into every later project.