Softmax and Logits

You will not get far in AI without this. Every classifier — whether it decides if an email is spam, which of a thousand images is in front of a camera, or which word follows 'I love you' in a chatbot — ends in a softmax. It is the final step that turns a jumble of raw scores into a clean 'this one, with this much confidence'. Understanding softmax means you can read any model's last layer, debug why it is unsure, and later grasp larger ideas like knowledge distillation (inventing a smarter student model) and temperature sampling (making a chatbot more creative). It unlocks: Cross-Entropy Loss, Temperature and Sampling, Classifier Architectures, and ultimately LLM generation mechanics.

Imagine you have a bowl of three fruits: an apple, a banana, and a cherry. You close your eyes, pick one, and eat it. You cannot explain why, but you have a gut feeling: 'apple 60%, banana 30%, cherry 10%'. A neural network does the same thing, but it starts with rough scores, not percentages. For example, it might give scores like apple: 2, banana: 1, cherry: 0. These raw scores are called logits — they are not probabilities because they can be any positive or negative number, and they do not necessarily add up to 1. The softmax is the recipe that takes these raw scores and turns them into percentages that add up to 100%. It does two things: it makes sure every percentage is positive and between 0 and 1, and it makes the biggest raw score the biggest percentage. But it does not just take the largest and ignore the rest; it keeps the relative differences. If the apple score is much higher than the banana, the apple percentage will be close to 100%. If they are close, the percentages will be closer. The key trick is that softmax uses the mathematical constant *e* (about 2.718) to amplify differences — a small difference in logits becomes a larger difference in probabilities. That is why it is called 'soft' max — it is a gentle, smooth version of 'pick the largest', not a hard cut.

Think of a talent show. Five contestants perform, and the judges give scores on a scale of 1 to 10. The scores might be: A: 7, B: 8, C: 6, D: 9, E: 5. But the show wants to present the results as percentages of the audience's vote. They cannot just divide each score by the total (7+8+6+9+5=35), because then A would get 20%, B 23%, C 17%, D 26%, E 14%. That works, but it treats a 7 and an 8 as nearly equal. The show's producer, however, wants to reward excellence: they want the winner to stand out more. They introduce a rule: before adding up, they turn each score into something like 'e to the power of the score'. So 7 becomes e^7 (a huge number), 8 becomes e^8 (even huger), and so on. The differences get amplified: e^9 is about 8103, e^8 is about 2981, e^7 is about 1097, so the ratio between D and A is now 8103/1097 ≈ 7.4, whereas before it was 9/7 ≈ 1.3. The percentages become: D: 73%, B: 27%, A: 10%, C: 3%, E: 0.2% (roughly, after normalizing). This is exactly what softmax does — it exponentiates the logits, then divides by the sum of all exponentials, so the biggest logit dominates. But the analogy breaks down in one way: the talent show judges give scores with a limited range, but a neural network's logits can be any number, even negative. Negative logits become very small probabilities after exponentiation, which is fine — it just means 'very unlikely'. Also, while the show's producer chose the exponential rule arbitrarily, in machine learning it is chosen because it makes the math work beautifully with the cross-entropy loss function, which is how the network learns.

Softmax is a function that takes a list of raw scores (logits) and turns them into a list of probabilities that all sum to 1, by exponentiating each logit and then dividing by the sum of all exponentiated logits.

You have not built up much yet, but this connects directly to the Activation Functions you will learn next — softmax is often called the final activation. It also sets the stage for Cross-Entropy Loss, which is the standard way to measure how wrong the probabilities are, and later for Temperature — a way to make the probabilities more or less confident. In the learner's library, you have notes on 'Activation Functions' and 'Cross-Entropy Loss' as neighbours — this page is the bridge between them.