In words
What it is, why it matters, and what it is like.
Why am I learning this?
You use functions like these every time you need to convert a raw score into a meaningful percentage or a balanced judgment. Imagine you are building a system to decide if an email is spam. The computer calculates a 'confidence score' of 150. You cannot simply tell a human 'the confidence is 150'. You need a rule that turns that big number into something between 0% and 100%. Sigmoid does exactly this: it takes any input number and squeezes it tightly into the range 0 to 1, which you can interpret directly as a probability. Similarly, if you want a system that balances two opposing choices, like 'buy' vs 'sell', you might need values between -1 and 1. Tanh provides that balance. Understanding how these functions work is essential because they control how confidently a model expresses its answers and, critically, how much information flows back through the system to allow learning. Without this understanding, it is impossible to know why some models learn quickly while others stall.
The idea, in plain terms
Think of these functions as extreme squeezers or compressors. Take any number you can imagine: 1,000,000; -50; 0.0001; or even just 2. The Sigmoid function takes that input and transforms it into a result that never falls outside the narrow lane between 0 and 1.
Let us look at specific numbers to see how this 'squeezing' behaves:
- If you put a large positive number like 3 into Sigmoid, the output is approximately 0.95. It is very close to the top of the lane (1).
- If you put exactly 0 in, the output is exactly 0.5. This is the midpoint.
- If you put a large negative number like -3 in, the output is approximately 0.05. It is very close to the bottom of the lane (0).
As the input numbers get larger in either direction (more positive or more negative), the output gets closer and closer to the edges (1 or 0) but never actually touches them. This behavior is called 'saturation'. The function becomes flat at the edges.
Tanh works on the same principle but uses a different lane, from -1 to +1.
- Input 3 becomes roughly +0.995 (near the top).
- Input 0 becomes exactly 0 (the neutral middle).
- Input -3 becomes roughly -0.995 (near the bottom).
The key feature here is that while Tanh centers the output at zero, it still saturates. When the function saturates, its slope becomes almost flat. Imagine a hill that is very steep in the middle but becomes completely flat on the sides. If you try to push a ball down that hill near the edge, it hardly moves. In learning systems, this 'flatness' means the system stops receiving signals about how to adjust itself, which can cause the entire process to stop learning effectively.
An analogy
Imagine a master key that fits into almost any lock, but the lock has a physical stop that prevents the key from turning more than 360 degrees. If you have a key with a long shaft (a huge input number) and you try to turn it wildly in either direction, the tumblers inside the lock will hit their stops. Whether the input is 1 or 1,000, the lock's final position might be nearly identical because it has reached its mechanical limit. This is what happens when the function saturates: small changes in your input are lost because the output has nowhere left to go. Tanh is like a lock that starts in the neutral 'off' position and can turn fully clockwise or counter-clockwise, but still hits hard stops at either extreme.
Definition
Sigmoid and Tanh are activation functions that take any real number as input and map it to a fixed, bounded range: Sigmoid outputs values strictly between 0 and 1, while Tanh outputs values strictly between -1 and 1, effectively 'squashing' wide ranges of input into narrow intervals.
Where this sits
These are the classic examples of activation functions, which are the mathematical tools that introduce nonlinearity into a neural network. Without these nonlinear steps, multiple layers of simple linear calculations would cancel each other out, making depth useless. Sigmoid and Tanh were the original choices for this role before simpler alternatives like ReLU became standard for hidden layers due to speed. They also illustrate the 'vanishing gradient' problem, where the signal used to update the model's memory fades away as it travels backward through many layers because the function's slope becomes too flat.
