In words
What it is, why it matters, and what it is like.
Why am I learning this?
Imagine you want to send a message to a friend, or save a photo on your phone. Every piece of information takes up space — bits in a file, or time on a call. Entropy tells you the absolute minimum amount of space needed to store or transmit that information without losing anything. If you understand entropy, you understand why some data is easy to shrink and other data resists compression. It answers the question: what is the most efficient way to write down what I just saw? For example, if you are sending a long string of letters, and some letters appear very often (like 'e') while others rarely do, entropy helps you figure out how many bits each letter needs in an optimal system. If you don't account for this efficiency, you waste space and time.
The idea, in plain terms
Think of a coin. If it is fair — heads and tails equally likely — then every toss genuinely surprises you: you had no way to guess better than a coin flip. If the coin is so biased that it lands heads 99 times out of 100, then a head is hardly a surprise at all; a tail, on the other hand, is a big surprise. Entropy is a way of putting a number on 'how surprising, on average, are the outcomes in this situation?' The more evenly spread the probabilities, the higher the surprise on average, and the higher the entropy. If one outcome is near-certain, the average surprise is low — you almost always know what is coming. That is the whole idea, before any maths: entropy measures how uncertain you are, on average, before you see the next outcome. Uniform distributions (all outcomes equally likely) have the maximum entropy; concentrated distributions (one outcome dominates) have low entropy.
An analogy
Imagine you are playing a guessing game with a friend who silently picks one of four colours: red, blue, green, yellow. If all four are equally likely, you have no clue which one it is — you need to ask 'is it this one?' questions, and on average it takes 2 questions to pin it down. That 2 is the entropy, in bits. Now suppose your friend is a fan of red and picks red 90% of the time. Most of the time you can guess red and be right; you rarely need many questions. The average number of questions drops, and so does the entropy. In fact, if red is 90%, blue 5%, green 3%, yellow 2%, the best strategy — ask 'is it red?' first — gives an average of about 0.6 questions. That 0.6 is the entropy in bits. This analogy works beautifully for a while: entropy really is the average number of yes/no questions needed to identify an outcome, and it is exactly the number of bits needed to store the outcome in an optimal code. But it breaks down if you push it too far. Real codes work on long streams of symbols, not one symbol at a time, and the 'questions' in the analogy assume you can design the best possible questioning strategy. Also, entropy can be a fractional number of bits (like 0.6), even though you can only ask a whole number of questions for a single outcome — the fractional bits come from amortising over many outcomes. So treat the guessing game as a good ladder, not the destination.
Definition
Shannon entropy is the average uncertainty or 'surprise' in a set of possible outcomes. It represents the theoretical minimum average number of bits needed to store any item from that set in a perfectly efficient system.
Where this sits
You are starting from a clean slate — no prior knowledge of probability, functions, or algebra is assumed here. The only branch of mathematics you need now is arithmetic: adding, multiplying, and logarithms (which we will teach from scratch when we need them). Shannon entropy is the first concept in information theory. From this seed, you can understand how data compression works by assigning shorter codes to frequent items, and how prediction errors are measured by comparing what you expect with what actually happens. Your notes record that entropy is 'the average surprise in a distribution' — that is exactly the intuition you just built.
