ELBO

The ELBO is the engine behind models that generate new data — images, audio, text — and it is the objective used to train variational autoencoders (VAEs), which power modern generative AI and anomaly detection. Mastering the ELBO unlocks the study of generative models, variational inference, and the inner workings of systems like Stable Diffusion. From here you can move on to deep learning and then to advanced generative architectures.

Imagine you want to understand a hidden pattern in data. You have a set of observations — say, pictures of handwritten digits — and you believe each one was generated by some underlying simple cause, like which digit it is, the angle it is tilted at, the thickness of the stroke. But you do not know these causes for each picture, and working out the full probability of the data given all possible causes is impossible: there are too many combinations. The ELBO solves this by turning the problem around: instead of computing the impossible full probability, you pick a simpler, tractable distribution that approximates the real one, then adjust that simpler distribution until it matches the data as well as possible. The ELBO is the number that tells you how well your simple approximation explains the data — the higher the better. Maximising the ELBO simultaneously improves the quality of generated samples and the fit of the model to the data.

Think of a detective trying to identify a suspect from a blurred security camera image. The detective has a set of possible suspects (the latent causes). The full probability of the image given every suspect is too heavy to compute — there are millions of suspects and each would require a detailed reconstruction. Instead, the detective starts with a rough guess about who the suspect might be (a simple distribution over suspects), then uses the actual image to refine that guess. The detective's measure of how good the guess is — how likely the image is under that suspect, minus how much the guess has been overcomplicated — is the ELBO. The detective keeps tweaking the guess to make the ELBO bigger, which means: the suspect’s appearance explains the image well, and the guess is not so far-fetched that it is just overfitting. The analogy breaks down here: the detective works with one image, but the ELBO is averaged over a whole dataset of images. Also, the detective’s guess is a single suspect, while the ELBO uses a distribution over suspects — a whole set of possibilities with probabilities attached.

The evidence lower bound (ELBO) is a computable number that lower-bounds the log probability of observed data under a model, and it is the objective that variational inference maximises to fit the model when the true log probability is too hard to compute directly.

This concept connects directly to your notes on KL Divergence and its asymmetry: the ELBO is essentially the log evidence minus the KL divergence between your approximate distribution and the true posterior. Maximising the ELBO is equivalent to minimising that reverse KL divergence. It also builds on Shannon Entropy — the reconstruction term is a cross-entropy between the actual data and the model’s reconstruction. The ELBO is the one-line identity that variational inference lives on, and it is the training objective for variational autoencoders, which are a bridge to deep learning and generative models.