Sigmoid and Tanh

Sigmoid and Tanh are two of the simplest and most important 'squashing' functions in AI. They appear everywhere: in the final layer of a model that must output a probability (like 'is this email spam?'), in the 'gates' of memory cells in LSTM networks, and in the design decisions that led to the modern ReLU activation. Understanding them now will make it far easier to understand why deep networks struggled before ReLU, how gradient flow works, and what it means for a function to 'saturate'. They are your first concrete look at how a number gets squeezed into a range that is actually useful for making decisions.

Imagine you have a machine that takes any number — negative, positive, huge, tiny — and squeezes it into a narrow band. Sigmoid squeezes everything into the band between 0 and 1. Tanh squeezes everything into the band between −1 and 1. This is useful because many decisions are naturally yes/no or 'how confident am I?'. A score of 0.99 from Sigmoid is a very confident 'yes'; 0.5 is 'no idea'; 0.01 is 'confident no'. Tanh is just a shifted and stretched Sigmoid: if you take a Sigmoid output (0 to 1), multiply it by 2 and subtract 1, you get a value between −1 and 1, which is exactly what Tanh does. The word 'saturating' means that as the input gets very large or very small, the output gets closer and closer to the top or bottom of the band, but never quite reaches it. This is the key property that makes them so useful — and also the property that caused a major problem in deep networks, called 'vanishing gradients'.

Imagine a pressure valve on a steam engine. The valve opens proportionally to the pressure, but it can only open so wide. No matter how much steam you push in, the valve never opens more than its maximum, say 1 unit. And if the pressure is negative (a vacuum), the valve closes to a minimum, say 0. That's exactly what Sigmoid does: it takes any 'pressure' (input number) and opens to a degree between 0 (fully closed) and 1 (fully open). Tanh is like the same valve, but restarted so that the default (zero pressure) is neutral, in the middle — it can open forward to +1 or backward to −1. The sticking point: once you push the valve near its fully open or fully closed position, pushing a bit more steam makes almost no difference — the valve is 'saturated'. That's great for giving a confident answer, but terrible if you need the system to be sensitive to small changes at the extremes. In a deep network, this is the 'vanishing gradient' problem: if all the valves are nearly saturated, a small change in one part of the engine has almost no effect on the final output, so the system can't learn.

Sigmoid and Tanh are 'squashing' (saturating) activation functions used in neural networks: they take any real number as input and produce a value in a fixed, bounded range — 0 to 1 for Sigmoid, −1 to 1 for Tanh.

This is your very first lesson in activation functions, which are the nonlinearity applied after each layer's linear transformation in a neural network. Without an activation function, stacking layers would algebraically collapse into a single linear map — so activations are what make depth meaningful. These two specific functions are the oldest ones, and they directly motivate the later invention of ReLU, which became the default for hidden layers. They also connect to Softmax (which generalises Sigmoid to multiple classes) and to the phenomenon of 'vanishing gradients' that plagues deep networks.