In words
What it is, why it matters, and what it is like.
Why am I learning this?
ReLU is the default activation for hidden layers in nearly every modern neural network. Because it is cheap and keeps gradients flowing on the positive side, it is what makes training deep networks feasible. Understanding ReLU means you understand why depth works in practice. It also unlocks the study of activation functions, vanishing gradients, and the architecture of large language models and vision systems.
The idea, in plain terms
Imagine a gate that lets positive numbers pass through unchanged, but blocks negatives completely, turning them into zero. For example, 5 stays 5, 0 stays 0, but -3 becomes 0. That is all ReLU does: if the input is greater than zero, it outputs that input; otherwise, it outputs zero. This simple rule is what lets deep networks learn complex patterns without slowing down.
An analogy
Think of a water pipe that only allows flow in one direction. If the pressure is positive, water flows through at exactly that pressure. If the pressure is negative (a vacuum), the pipe clamps shut and nothing flows. This is ReLU: positive pressure passes, negative pressure is cut off. The analogy works because ReLU lets information flow when the signal is positive, but kills it when negative. However, unlike a pipe, ReLU is not physical; it is a mathematical operation applied to numbers in a network.
Definition
ReLU (Rectified Linear Unit) is an activation function that outputs its input if the input is positive, and outputs zero otherwise.
Where this sits
This builds on the concept of the artificial neuron and activation functions. It is a response to the vanishing gradient problem with sigmoid, and it is the default choice for hidden layers in deep networks.
