In words
What it is, why it matters, and what it is like.
Why am I learning this?
This idea is the rock under your feet for three things you will meet again and again. First, it defines the absolute floor on lossless compression — no matter how clever your file-zipping or model-weight storage gets, you cannot store a stream of samples in fewer bits than its entropy. Second, it is the anchor from which cross-entropy loss is built; that loss is what every classification model and every language model minimises when it learns to predict the next token. Third, it is the starting point for mutual information, which measures how much one variable tells you about another — the same logic that powers retrieval, embeddings, and feature selection. Learn entropy once, and cross-entropy and mutual information become small steps from something you already own.
The idea, in plain terms
Think of a coin. If it is fair — heads and tails equally likely — then every toss genuinely surprises you: you had no way to guess better than a coin flip. If the coin is so biased that it lands heads 99 times out of 100, then a head is hardly a surprise at all; a tail, on the other hand, is a big surprise. Entropy is a way of putting a number on 'how surprising, on average, are the outcomes in this situation?' The more evenly spread the probabilities, the higher the surprise on average, and the higher the entropy. If one outcome is near-certain, the average surprise is low — you almost always know what is coming. That is the whole idea, before any maths: entropy measures how uncertain you are, on average, before you see the next outcome. Uniform distributions (all outcomes equally likely) have the maximum entropy; concentrated distributions (one outcome dominates) have low entropy.
An analogy
Imagine you are playing a guessing game with a friend who silently picks one of four colours: red, blue, green, yellow. If all four are equally likely, you have no clue which one it is — you need to ask 'is it this one?' questions, and on average it takes 2 questions to pin it down. That 2 is the entropy, in bits. Now suppose your friend is a fan of red and picks red 90% of the time. Most of the time you can guess red and be right; you rarely need many questions. The average number of questions drops, and so does the entropy. In fact, if red is 90%, blue 5%, green 3%, yellow 2%, the best strategy — ask 'is it red?' first — gives an average of about 0.6 questions. That 0.6 is the entropy in bits. This analogy works beautifully for a while: entropy really is the average number of yes/no questions needed to identify an outcome, and it is exactly the number of bits needed to store the outcome in an optimal code. But it breaks down if you push it too far. Real codes work on long streams of symbols, not one symbol at a time, and the 'questions' in the analogy assume you can design the best possible questioning strategy. Also, entropy can be a fractional number of bits (like 0.6), even though you can only ask a whole number of questions for a single outcome — the fractional bits come from amortising over many outcomes. So treat the guessing game as a good ladder, not the destination.
Definition
Shannon entropy is the average number of bits needed to represent each symbol from a distribution, and it is the theoretical minimum average number of bits for lossless compression of a stream of samples from that distribution.
Where this sits
You are starting from a clean slate — no prior knowledge of probability, functions, or algebra is assumed here. The only branch of mathematics you need now is arithmetic: adding, multiplying, and logarithms (which we will teach from scratch when we need them). Shannon entropy is the first concept in information theory. From this seed, three ideas grow: cross-entropy loss (the cost of using the wrong distribution — the standard loss for classification and next-token prediction), KL divergence (the difference between two distributions, which is essentially cross-entropy minus entropy), and mutual information (shared structure between variables). Your notes record that entropy is 'the average surprise in a distribution' — that is exactly the intuition you just built. You will also meet it again when you study data compression (Huffman codes are built to approach the entropy limit) and the ELBO (which contains a KL term that itself contains an entropy).
