Information Geometry

This topic changes how you visualize probability distributions from flat lists of numbers into shapes in a curved space. Understanding this reveals why some optimization problems are easy while others are treacherously difficult. It explains the geometry behind a method for approximating complex beliefs, called variational inference, and clarifies why gradient-based learning sometimes fails or requires special adjustments when parameters are correlated. Instead of treating your data as points on a graph, you treat them as locations on a landscape where 'distance' means 'how much information do I lose if I confuse these two states?'. This perspective is essential for building robust models in Bayesian Methods that respect the inherent structure of uncertainty.

Imagine you are mapping a mountain range, but your map only gives you coordinates (latitude and longitude), not elevation or slope. You want to know how hard it is to walk from point A to point B on this terrain. On a flat plain, distance is simple: a straight line. But mountains are curved. A step that looks small on your coordinate map might take you up a steep cliff, while another step of the same map-length might slide gently down a slope. The 'true' difficulty of moving between two points depends not just on the coordinates, but on how steep and curved the ground is at that specific location.

Information geometry treats the space of all possible probability distributions exactly like this mountain range. Each distribution—say, a bell curve defined by a mean and a variance—is a point on this landscape. The 'distance' between two distributions is not the difference in their means or variances (the map coordinates). Instead, it is measured by how much information you lose if you pretend one distribution is the other. This measure is called Kullback-Leibler divergence. Think of Kullback-Leibler divergence as a penalty score for being wrong: it quantifies the surprise or error introduced when you use one story to describe reality that follows another.

The shape of this landscape is defined by something called Fisher Information. Think of Fisher Information as a sensor that measures how 'steep' the terrain is at every point. If Fisher Information is high, the landscape is sharply curved; a tiny change in your distribution parameters leads to a large change in the actual probability content (a big jump in the mountain). If Fisher Information is low, the landscape is flat; you can tweak your parameters freely without changing the underlying story much.

For example, consider two bell curves with the same mean but different variances (widths). One is narrow and tall, the other wide and flat. In parameter space, they differ only by one number. But in information geometry, the 'distance' between them depends on how distinct their shapes are. If the curves overlap heavily, the distance is small—they tell similar stories. If they are disjoint, the distance is large. Information geometry gives you the ruler that measures this informational distance, not just the difference in numbers.

Think of a hiker trying to climb a mountain using only a standard flat map with latitude and longitude grid lines. The hiker wants to reach the summit. If they walk one mile north on the flat map, they might find themselves on a gentle meadow in one spot, but face a sheer cliff in another. The 'distance' on the map is misleading because it ignores the terrain's curvature. To climb efficiently, the hiker needs more than the map; they need to know how the slope changes beneath their feet at every step. They need to know not just which direction points up, but how 'steep' that upward path is relative to the ground itself. Information geometry provides this missing context: it scales the steps based on the local curvature (Fisher Information). It tells the hiker: 'Your next step is in a rocky area; even though your map says it's one mile, it will feel like ten miles of effort, so take smaller, more careful steps.' This adjusted direction and step size is what allows optimization algorithms to navigate efficiently without getting stuck on cliffs or wasting time on flat ridges.

Information geometry is the study of probability distributions as points on a curved surface where local distances are defined by how much information changes between nearby distributions, rather than by simple differences in numerical parameters.

This concept extends your understanding of Fisher Information by interpreting it not just as a measure of statistical precision, but as the fundamental ruler (metric) that defines distance on the curved space of distributions. It connects directly to Bayesian Inference, where comparing prior and posterior beliefs—or approximate posteriors—requires measuring how 'far apart' these distributions are in terms of information loss, often using Kullback-Leibler divergence.