Dropout

This page is the first step toward training real neural networks that don't just memorize their training data. Dropout is one of the simplest and most widely used techniques to make a network generalize — to perform well on data it has never seen. Once you master dropout, you will be ready for the next concepts in your library: regularization methods, early stopping, and eventually the full training pipeline for convolutional networks and transformers. Without dropout, your first attempts at training deep networks will likely show excellent results on the training set but disappointing results on new data. With dropout, you'll see the gap close. This is the concept that separates a model that has learned the underlying pattern from one that has merely memorized the answers.

Imagine you are studying for an exam. If you always study with the same friend, you both might get good at answering questions in the same way — but if the exam has a question you never practised together, you might both fail, because you never learned to handle it independently. Dropout is like forcing yourself to study alone sometimes. During training, we randomly turn off some of the neurons — the units that compute — so the network cannot rely on any one particular pathway. It has to learn many different ways to reach the same answer. Then, when it comes to the real exam (the test set), we turn all neurons back on, so it uses all its knowledge. This makes the network more robust, because it doesn't depend on any single feature being present. The same idea works in teams: a team that always relies on one star player collapses if that player is absent. A team where everyone can cover for each other is much stronger. Dropout builds redundancy into the network, so no single neuron becomes a crutch.

Think of a school where students are taught by a committee of teachers. Each teacher has their own style and strengths. If the same teacher always gives the final lecture, students might learn to rely on that teacher's quirks — but on the real exam, the teacher isn't there. So the school decides: for each practice session, they randomly pick a subset of teachers and they all teach together. No single teacher is always present. This forces students to learn the subject from multiple perspectives, so they can handle a question even if it's phrased differently from what they've seen. After many practice sessions, the students take the real exam, and all teachers are available for consultation (but not for teaching). The students combine all their knowledge. Dropout works exactly this way: during training, we randomly drop out (turn off) some neurons in a layer. This prevents the network from relying on any one neuron or pathway. The network must learn redundant representations. At inference (the real exam), we use all neurons, but we scale their outputs so the total signal stays consistent. Where does the analogy break? In a school, teachers have fixed knowledge; dropout doesn't change what a neuron knows, it just changes which neurons are active during a given training step. Also, in a school, you might deliberately drop a weak teacher; dropout drops neurons uniformly at random, with no judgment about which are important. Finally, in a school, the teachers are separate people; in a network, all neurons share the same training process and are adjusted together. But the core idea — forcing robustness by removing reliance on any single component — carries over perfectly.

Dropout is a training technique where, at each training step, a random subset of neurons (or their outputs) is set to zero, so the network cannot depend on any single neuron or pathway, and then at inference time, the full network is used with a scaling factor to keep the expected output consistent.

You haven't yet learned about neural networks, but you can think of this as the first regularisation technique in your arsenal. In your library, you have notes on Deep Learning, Activation Functions, and Backpropagation — those are all prerequisites that you will learn soon. Dropout is a tool that sits on top of them: once you have a trained network, dropout helps it generalise. It connects to your notes on Autoencoders, because dropout is often used there as a form of denoising. It also connects to Convolutional Networks and Transformers, where dropout is applied between layers to prevent overfitting. Right now, you have no prior knowledge, so this page starts from zero and builds up the idea of what a neuron is, just enough to understand dropout. We'll take small steps, using analogies and arithmetic, with no maths beyond addition and multiplication.