Softmax and Logits

You will not get far in AI without understanding how models make choices. Every classifier — whether it decides if an email is spam, which of a thousand images is in front of a camera, or which word follows 'I love you' in a chatbot — ends by turning a jumble of raw numbers into a clean 'this one, with this much confidence'. Understanding this final step means you can read any model's last layer, debug why it is unsure, and later grasp larger ideas like knowledge distillation (inventing a smarter student model) and temperature sampling (making a chatbot more creative). It unlocks Cross-Entropy Loss, Temperature and Sampling, Classifier Architectures, and ultimately LLM generation mechanics.

Imagine you have a bowl of three fruits: an apple, a banana, and a cherry. You close your eyes, pick one, and eat it. You cannot explain why, but you have a gut feeling: 'apple 60%, banana 30%, cherry 10%'. A neural network does the same thing, but it starts with rough scores, not percentages. For example, it might give scores like apple: 2, banana: 1, cherry: 0. These raw scores are called logits — a technical term for these initial numbers. They are not probabilities because they can be any positive or negative number, and they do not necessarily add up to 1. The softmax function is the recipe that takes these logits and turns them into percentages that add up to 100%. It does two things: it makes sure every percentage is positive and between 0 and 1, and it makes the biggest raw score the biggest percentage. But it does not just take the largest and ignore the rest; it keeps the relative differences. If the apple score is much higher than the banana, the apple percentage will be close to 100%. If they are close, the percentages will be closer. The key trick is that softmax uses the mathematical constant *e* (about 2.718) to amplify differences — a small difference in logits becomes a larger difference in probabilities by raising *e* to the power of each score. That is why it is called 'soft' max — it is a gentle, smooth version of 'pick the largest', not a hard cut.

Think of a talent show. Five contestants perform, and the judges give scores on a scale of 1 to 10. The scores might be: A: 7, B: 8, C: 6, D: 9, E: 5. But the show wants to present the results as percentages of the audience's vote. They cannot just divide each score by the total (7+8+6+9+5=35), because then A would get 20%, B 23%, C 17%, D 26%, E 14%. That works, but it treats a 7 and an 8 as nearly equal. The show's producer, however, wants to reward excellence: they want the winner to stand out more. They introduce a rule: before adding up, they turn each score into something like 'e to the power of the score'. So 7 becomes *e*^7 (a huge number), 8 becomes *e*^8 (even huger), and so on. The differences get amplified: *e*^9 is about 8103, *e*^8 is about 2981, *e*^7 is about 1097, so the ratio between D and A is now 8103/1097 ≈ 7.4, whereas before it was 9/7 ≈ 1.3. The percentages become: D: 73%, B: 27%, A: 10%, C: 3%, E: 0.2% (roughly, after dividing by the total sum of these amplified numbers). This is exactly what softmax does — it raises *e* to the power of each score, then divides each result by the sum of all such results, so the biggest score dominates. This analogy has one limitation: the talent show judges give scores with a limited range, but a neural network's logits can be any number, even negative. Negative logits become very small probabilities after raising *e* to that power, which is fine — it just means 'very unlikely'.

Softmax is a function that takes a list of raw scores (called logits) and turns them into a list of probabilities that all sum to 1, by raising *e* to the power of each score and then dividing each result by the sum of all those raised values.

You have not built up much yet, but this connects directly to Activation Functions you will learn next — softmax is often called the final activation. It also sets the stage for Cross-Entropy Loss, which is the standard way to measure how wrong the probabilities are, and later for Temperature — a way to make the probabilities more or less confident. In the learner's library, you have notes on 'Activation Functions' and 'Cross-Entropy Loss' as neighbours — this page is the bridge between them.