Mutual Information

You want to know if two things in your professional life are genuinely connected or just coincidentally appearing together. For example, does a customer reading page A actually make them more likely to buy product B? Or is that just noise? Mutual information is the specific metric that answers this: it tells you exactly how much knowing one piece of data reduces your uncertainty about another. It matters now because it allows you to filter signal from noise. If you are trying to decide which sales metrics matter, mutual information shows you which variables actually carry information about your goal (like revenue) and which are just random chatter. It works for any kind of relationship—whether the connection is a straight line or a complex curve—and it gives you a single number that quantifies that strength in terms of saved uncertainty.

Imagine you are trying to guess whether it will rain in Mumbai tomorrow. Without any information, your uncertainty is high; you have no idea. Now I tell you the sky is dark and heavy with clouds. Your uncertainty collapses: it is probably going to rain. Knowing the cloud cover reduced your uncertainty about rain. That reduction in uncertainty is exactly what mutual information measures.

To see this numerically, imagine we track weather data for 100 days. On 50 of those days, it rained (rain happens half the time). Your initial 'uncertainty' is high because there are many possible outcomes. But if I tell you that on a specific day, the cloud cover was above 90%, and we know that in our historical data, it rained on all 20 days with such heavy clouds, your uncertainty drops to near zero. The mutual information between 'cloud cover' and 'rain' is high.

Contrast this with another variable: the day of the week. We look at the same 100 days. Whether it was Tuesday or Wednesday never changed the likelihood of rain; on average, rain happened 50% of the time regardless of the weekday. Here, knowing the day of the week tells you nothing new. The mutual information between 'day of week' and 'rain' is zero.

Crucially, this metric captures *any* pattern of dependence, not just a straight-line relationship. Imagine two variables that form a perfect circle: if you know the X-coordinate of a point on the circle, you know the Y-coordinate must be either positive or negative (two possibilities), drastically narrowing down the range of values. Linear correlation would see no relationship here because there is no slope, but mutual information would catch that strong dependency entirely.

Think of mutual information as a measure of shared insight between two colleagues reporting on a project. Imagine Alex and Sam each write a daily report with a number representing their progress. You are trying to guess Sam's number without seeing his paper. Initially, you have maximum uncertainty—you know nothing.

Now, suppose you are allowed to peek at Alex's number. If knowing Alex's progress allows you to predict Sam's progress with high accuracy (perhaps because they work on the same dependent tasks), then there is a high mutual information between their reports. You have 'saved' yourself from having to guess blindly.

If, however, Alex's numbers are totally random and unrelated to Sam's, peeking at Alex’s report changes nothing about your ability to guess Sam's. The mutual information is zero. The key feature is symmetry: if seeing Alex's number helps you guess Sam's, then seeing Sam's number equally helps you guess Alex's. It is a shared reduction in uncertainty for both parties.

This analogy holds up well for understanding the concept, though note that it focuses on single instances; mutual information actually looks at the aggregate pattern of how often numbers appear together across many days, not just one lucky match.

Mutual information is the amount by which knowing the value of one variable reduces your uncertainty about another, measured in units called bits, and it equals zero if and only if the two variables are completely unrelated.

This concept sits beside the idea of entropy. Entropy measures the average uncertainty or 'surprise' in a single variable on its own—for example, how unpredictable the weather is on any given day. Mutual information is essentially the overlap between two such entropies; it tells you how much uncertainty about one variable is resolved by the other. It also connects directly to feature selection in data analysis. When you have a large list of potential predictors for an outcome, mutual information helps you rank them by how much information they actually provide about that outcome, allowing you to discard irrelevant variables effectively.