Inverse Problems

Every AI system is solving an inverse problem, and recognising that is the difference between understanding what a model does and just running it. In image denoising, you observe a noisy photo and infer the clean original. In super-resolution, you observe a small blurry image and infer the sharp large one. In medical imaging, you observe X-ray measurements and infer the internal structure. Even training a neural network is an inverse problem: you observe training examples and infer the weights that produced them. This concept unlocks the tool that makes all such problems tractable — regularisation — which you will meet again in the notes on Convex Optimization and Double Descent. Master this page and you will never again wonder why different models fit the same data in different ways.

Start with something you do every day: you hear a thud outside and think ‘that was a coconut falling from the tree.’ You did not see the coconut. You heard a sound, and from that sound you guessed the cause. That is an inverse problem: from an observed effect (the sound) you recover an unknown cause (the coconut). Now imagine hearing the same thud but realising it could have been a coconut, a large mango, or a neighbour dropping a bag of rice. All three causes produce exactly the same sound. This is the core difficulty of inverse problems: many different causes can lead to the same observation. The sound alone does not tell you which one it was. The same thing happens in AI. A blurry photo could come from a moving camera, a dirty lens, or a low-resolution sensor. A low sales figure could come from high prices, bad marketing, or a competitor’s launch. The observations alone do not tell you the cause. Inverse problems are everywhere, and the hard part is that they are almost always ambiguous. This page teaches you why that ambiguity exists and what mathematicians do about it.

Imagine you are in a dark room and you hear a knock on the wall. You only hear one knock, and you want to know what made it. It could be a person knocking, a falling branch, or a rat moving. All three create the same sound. You cannot see the cause, only the effect. In mathematics, the effect is the data you measure (the knock), and the cause is what you want to find (the object). An inverse problem is exactly this: given the effect, find the cause. The analogy breaks down when you realise that in a real room, you might gather more information — knock again, listen from different positions, or shine a light. In AI, you often cannot get more data; you only have the one observation, and you must make your best guess. This is why inverse problems are 'ill-posed': the data alone does not pin down a single answer. The act of choosing one answer among many is what regularisation does — it adds a rule for which cause is more likely. Without that rule, you are stuck guessing.

An inverse problem is the task of recovering unknown causes from observed effects, where the mapping from causes to effects is not one-to-one, so many different causes can explain the same observations.

This is your first concept in the Optimization family. The library notes say inverse problems belong to Optimization, and you will see why: solving an inverse problem is an optimisation task — you search over possible causes to find the one that best explains the data. You have not yet studied Convex Optimization, which is a sibling under the same parent. That concept will show you that when the objective is convex, the search always finds the single best answer. Inverse problems are often not convex, which is why they are hard. The foundational idea here — that many causes fit the same observations — is also the seed of Double Descent, where a model with too much capacity fits the data in many ways and handles that ambiguity differently. For now, focus on the core: the inverse mapping is ambiguous, and the remedy is a rule that prefers some causes over others.