Naive Bayes

Naive Bayes is the gateway to probabilistic thinking in machine learning. It is the first classifier you will meet that outputs a probability — not just a label — and that lets you tune the cost of different mistakes. It unlocks: Logistic Regression (which does the same thing but learns the weights from data), Decision Trees (which split the space instead of multiplying probabilities), and K-Nearest Neighbours (which compares distances rather than probabilities). It is also the fastest way to get a working classifier for text — the building block of spam filters, sentiment analysis, and many small on-device models. Master it and you will never again be surprised that a model can be both dead wrong in theory and excellent in practice.

Imagine you are a doctor with a patient who has a fever, a cough, and a runny nose. You want to guess whether they have the flu or just a cold. You have seen thousands of patients before, so you know: among all your patients, roughly 30% have the flu and 70% have a cold. You also know that among flu patients, 80% have a fever, 60% have a cough, and 40% have a runny nose. Among cold patients, 20% have a fever, 30% have a cough, and 70% have a runny nose. Now a new patient walks in with all three symptoms. You don't know the exact combination from your records — you only have the single-symptom rates. So you make a leap: you assume that the symptoms appear independently of each other given the disease. That means you can just multiply the probabilities: for flu, 0.8 × 0.6 × 0.4 = 0.192; for cold, 0.2 × 0.3 × 0.7 = 0.042. You also multiply by the overall rates: for flu, 0.192 × 0.3 = 0.0576; for cold, 0.042 × 0.7 = 0.0294. Add those two totals (0.087) and divide each one by it to get the final probability: flu is about 66%, cold is about 34%. So you lean flu. That is Naive Bayes: use single-feature rates, multiply them together, adjust for how common each class is, then normalize so the answers add up to 100%. The 'naive' part is the independence assumption — you know it's usually false, because symptoms often travel together, but it makes the arithmetic possible and it usually still gives a useful answer.

Think of a witness in a courtroom. The witness is trying to decide whether the defendant is guilty or innocent based on three pieces of evidence: a fingerprint, an alibi, and a motive. The witness does not have a perfect record of how often those three appear together for guilty and innocent people. Instead, they have separate statistics: of the last 100 guilty verdicts, 70 had fingerprints, 50 had weak alibis, and 60 had motives. Of the last 100 innocent verdicts, 10 had fingerprints, 30 had weak alibis, and 40 had motives. Also, overall, 20% of defendants are guilty and 80% are innocent. Now a new case has all three pieces of evidence. The witness assumes those pieces are independent given guilt or innocence — that having a fingerprint does not make a weak alibi more likely, and so on. This is usually false: criminals who leave fingerprints might also be sloppier with their alibis. But the witness goes ahead because the separate statistics are all they have. They calculate: for guilty, 0.7 × 0.5 × 0.6 × 0.2 = 0.042; for innocent, 0.1 × 0.3 × 0.4 × 0.8 = 0.0096. Normalizing (0.042 + 0.0096 = 0.0516), they get guilty ≈ 81% and innocent ≈ 19%. The witness issues a verdict of 'probably guilty' with a confidence level. Where does the analogy break down? In real life, evidence is often strongly correlated — a weak alibi is more likely if there is a motive — and the witness's assumption of independence can bias the final probability. Also, real witnesses do not typically have calm, precomputed base rates; they have messy memory. Naive Bayes, in contrast, gets its numbers from data and applies them consistently. The 'naive' assumption is a deliberate simplification, not a failure of observation.

Naive Bayes is a probabilistic classifier that uses Bayes' rule and assumes that, given the class, each feature is independent of every other feature, so that the probability of a set of features is the product of the probabilities of each feature separately.

This is your first probabilistic classifier in the Machine Learning Foundations path. It sits next to K-Nearest Neighbours and Decision Trees as one of the 'classical' algorithms — the ones that are small, fast, and easy to understand. It builds directly on the ideas of Data Mining (finding patterns in large data) and leads naturally to Logistic Regression, which learns the weights from data rather than counting frequencies. In your library, you have notes on Churn Prediction and Crop Disease Detection — both are problems where Naive Bayes can be a strong baseline because it needs little data and is easy to deploy. It is also the conceptual ancestor of the 'attention' mechanism in modern language models, which also mixes probabilities across features, though in a learned way.