Fourier Analysis of Networks

This concept unlocks a deeper understanding of why neural networks behave the way they do — why they learn simple patterns first, why they generalise so well, and why some training runs succeed while others fail. It connects to everything you will study later in computer vision and multimodal AI, and it gives you a mental model for diagnosing problems in your own network, like when a model overfits to noise or struggles with a particular kind of input.

Think of a sound. Any sound — a voice, a guitar chord, a dog bark — can be broken down into a collection of pure tones, each with its own pitch (frequency) and loudness. The same is true for any signal that varies over time or space, like an image (brightness varies across pixels) or a sensor reading. Fourier analysis is the mathematical tool that performs this breakdown: it takes a complex signal and tells you which frequencies are present and how strong each one is. A network that processes such a signal — an image, an audio clip, a time series — must somehow represent all those frequencies, and Fourier analysis lets us ask: which frequencies does the network care about? How does it learn to emphasise some and ignore others? It turns out networks have a systematic bias: they learn low frequencies (smooth, broad patterns) early in training, and only later pick up high frequencies (fine details, sharp edges, noise). This is called 'spectral bias'. It explains why networks generalise well — they first capture the fundamental structure of the data, not the noise — and why they can be fooled by imperceptible perturbations that are all high-frequency.

Imagine you are a chef learning to cook a complex dish. On your first day, you don't start with the intricate garnish — you learn to boil water, chop onions, and make a basic broth. Only after you master those broad strokes do you attempt the fine julienne and the delicate sauce reduction. A neural network does exactly this: it first learns the coarse, low-frequency structure of the data — the overall shape of a face, the general contour of a landscape — and only later, with more training, does it sharpen its ability to detect high-frequency details like the exact edge of a nose or the texture of a leaf. The chef's progression is the 'spectral bias' in action.

But the analogy breaks down: a chef can choose to focus on the garnish first if they want, but a network has no such choice. Its bias toward low frequencies is not a strategy — it's an emergent property of how gradient descent updates weights. The loss landscape is smoother for low-frequency components, so the gradient points more consistently in a useful direction, and the network descends quickly. High-frequency components have a more jagged landscape, so the gradient is noisy and the progress is slow. This is baked into the mathematics, not a decision by the network.

Fourier analysis of networks is the study of how a neural network's learned representations and training dynamics interact with the frequency content of the data, revealing that networks exhibit a systematic bias toward learning low-frequency patterns first and that convolution in space is equivalent to multiplication in frequency.

This concept builds on your existing notes on Convolutional Networks, particularly the ideas of how filters are learned and applied. In that context, you already know that a convolutional layer applies a small filter across an image to detect features. Fourier analysis provides a complementary lens: instead of looking at the filter in the spatial domain (as a grid of numbers), we can look at it in the frequency domain — what frequencies does the filter let pass? This directly links to your notes on CNN Visualization, where you saw that early layers learn edges and colours (which are low-frequency structures), and later layers learn composites of these (which include higher frequencies). The concept of receptive fields also connects: a larger receptive field allows the network to see more of the input, which in frequency terms means it can combine broader range of frequencies. This concept is also a stepping stone to understanding more advanced topics like the Neural Tangent Kernel and why overparameterised networks generalise, which appear in your library's notes on Double Descent.