Mutual Information

Mutual information is the tool that lets you answer a question no other single number can: does knowing one thing actually tell you something about another, in a way that matters? It appears everywhere in AI — in deciding which features to feed a model, in making sure a language model pays attention to the right words, in compressing data without losing meaning, and in detecting when two systems are genuinely independent. Mastering it unlocks the rest of information theory: how entropy measures uncertainty, how cross-entropy and KL divergence measure the cost of guessing wrong, and how the ELBO makes variational inference tractable. Those, in turn, unlock generative models and deep learning.

Imagine you are trying to guess whether it will rain in Mumbai tomorrow. Your uncertainty is high — you have no idea. Now I tell you the sky is dark and heavy with clouds. Your uncertainty collapses: it is probably going to rain. Knowing the cloud cover reduced your uncertainty about rain. That reduction is exactly what mutual information measures. If I told you instead that it is Tuesday, your uncertainty about rain would not change at all — Tuesday tells you nothing about rain. Mutual information between 'cloud cover' and 'rain' is high; between 'day of week' and 'rain' is near zero. The concept is symmetric: if knowing the clouds helps you guess the rain, then knowing it rained also helps you guess the sky was cloudy. It is non-negative: knowing something can never *increase* your uncertainty about another thing — it either helps or does nothing. The crucial point: mutual information captures *any* kind of dependence, not just a straight-line one. A perfect circle has zero linear correlation, but knowing the x-coordinate narrows down the y-coordinate to two possibilities — mutual information would catch that. Correlation misses it entirely.

Think of mutual information as a shared secret between two friends. Imagine Priya and Rahul each hold a piece of paper with a number on it. You are trying to guess Rahul's number. Without any information, you are completely in the dark — maximum uncertainty. Now suppose you learn Priya's number. If knowing Priya's number lets you guess Rahul's with high accuracy, they share a big secret — there is a lot of mutual information between the two numbers. If knowing Priya's number tells you nothing about Rahul's, they have no shared secret — mutual information is zero. The analogy carries to the arithmetic: the size of the shared secret is measured in bits — the number of yes/no questions you save on average. But the analogy stops working in one important way: mutual information is not about a single pair of numbers but about the *distribution* over all possible pairs. Two people could share a secret that only shows up in the pattern of many draws, not in any single pair. Also, it is symmetric — if knowing Priya's number helps you guess Rahul's, then knowing Rahul's helps you guess Priya's. The secret is shared, not owned by one person.

Mutual information is the reduction in uncertainty about one random variable that comes from observing another, measured in bits, and it is zero if and only if the two variables are independent.

This is your first topic in information theory, so there is nothing to build on yet — but this is where the whole subject starts. The key quantity that mutual information builds on is Shannon entropy, which measures the average surprise in a single variable. Mutual information is then the difference between the entropy of one variable and the entropy of that variable *after* you have observed the other. From here, you will connect to cross-entropy and KL divergence, which measure the cost of modelling one distribution with another, and ultimately to the ELBO, the objective behind variational inference. In AI, cross-entropy loss (which you will meet later, when you study classification) is the natural objective whenever the output is a distribution — and mutual information is the same idea applied to two variables instead of one. Mutual information also underlies data compression — entropy is the floor on how many bits you need, and mutual information tells you how much you can save by encoding two variables together.