Naive Bayes

You want to make better guesses when you don't have complete information. Naive Bayes is a simple method for turning a list of observed facts into a clear probability—like estimating the chance that an email is spam based on specific words in it. It is one of the fastest ways to build a working model for tasks like sorting messages or judging opinions, and it gives you results you can tune if some mistakes are more expensive than others.

Imagine you are a doctor with a patient who has a fever, a cough, and a runny nose. You want to guess whether they have the flu or just a cold. You have seen thousands of patients before, so you know: among all your patients, roughly 30% have the flu and 70% have a cold. This is your starting belief, called the base rate. You also know that among flu patients, 80% have a fever, 60% have a cough, and 40% have a runny nose. Among cold patients, 20% have a fever, 30% have a cough, and 70% have a runny nose. Now a new patient walks in with all three symptoms. You don't know the exact combination from your records — you only have the single-symptom rates. So you make a leap: you assume that the symptoms appear independently of each other given the disease. That means you can just multiply the probabilities: for flu, 0.8 × 0.6 × 0.4 = 0.192; for cold, 0.2 × 0.3 × 0.7 = 0.042. You also multiply by the overall rates: for flu, 0.192 × 0.3 = 0.0576; for cold, 0.042 × 0.7 = 0.0294. At this stage, these numbers (0.0576 and 0.0294) are not yet probabilities because they do not add up to 1 (or 100%). To fix this, you normalize them by dividing each by the sum of all possible outcomes (0.087). This rescales them so they fit together perfectly: flu is about 66%, cold is about 34%. So you lean flu. That is Naive Bayes: use single-feature rates, multiply them together, adjust for how common each class is, then normalize so the answers add up to 100%. The 'naive' part is the independence assumption — you know it's usually false, because symptoms often travel together, but it makes the arithmetic possible and it usually still gives a useful answer.

Think of a witness in a courtroom. The witness is trying to decide whether the defendant is guilty or innocent based on three pieces of evidence: a fingerprint, an alibi, and a motive. The witness does not have a perfect record of how often those three appear together for guilty and innocent people. Instead, they have separate statistics: of the last 100 guilty verdicts, 70 had fingerprints, 50 had weak alibis, and 60 had motives. Of the last 100 innocent verdicts, 10 had fingerprints, 30 had weak alibis, and 40 had motives. Also, overall, 20% of defendants are guilty and 80% are innocent. Now a new case has all three pieces of evidence. The witness assumes those pieces are independent given guilt or innocence — that having a fingerprint does not make a weak alibi more likely, and so on. This is usually false: criminals who leave fingerprints might also be sloppier with their alibis. But the witness goes ahead because the separate statistics are all they have. They calculate: for guilty, 0.7 × 0.5 × 0.6 × 0.2 = 0.042; for innocent, 0.1 × 0.3 × 0.4 × 0.8 = 0.0096. Normalizing (0.042 + 0.0096 = 0.0516), they get guilty ≈ 81% and innocent ≈ 19%. The witness issues a verdict of 'probably guilty' with a confidence level. Where does the analogy break down? In real life, evidence is often strongly correlated — a weak alibi is more likely if there is a motive — and the witness's assumption of independence can bias the final probability.

Naive Bayes is a method for classifying items by multiplying the likelihood of their features appearing in each category together, then scaling the results so they represent complete probabilities.

This sits beside K-Nearest Neighbours and Decision Trees as one of the 'classical' algorithms — the ones that are small, fast, and easy to understand. It builds directly on the ideas of Data Mining (finding patterns in large data) and leads naturally to Logistic Regression, which learns the weights from data rather than counting frequencies.