Hypothesis Testing

Hypothesis testing is the formal machinery behind every claim that 'this AI model is better than that one', 'this new drug works', or 'this ad campaign increased sales'. Without it, you are comparing numbers and hoping. With it, you can say — with a controlled rate of being wrong — whether a difference you see in sample data is real or just noise. This unlocks: model evaluation, A/B testing in product work, and causal inference. When you later read 'this LLM scores 85 on the benchmark, the previous one scored 82', hypothesis testing is what tells you whether the 3-point gap is meaningful or just random variation. It is also the backbone of every 'significance' claim you will encounter in AI research papers.

You have a bag of marbles. You cannot see inside, but you have a suspicion: maybe the bag is not half red and half blue as you were told — maybe it is mostly red. You cannot dump out all the marbles (that would be like testing every person on Earth), so you take out a handful — a sample. If the handful is 8 red and 2 blue, is that enough to conclude the whole bag is mostly red? Or could that happen by chance even if the bag truly was half and half? Hypothesis testing is the procedure that answers: 'Given that the bag really is half and half, how unlikely is it that I would draw a handful this extreme?' If that probability is tiny, you decide the 'half and half' story is not believable, and you reject it. The key is that you never prove the bag is mostly red. You only say: if the null were true, seeing this would be very unlikely, so I do not believe the null. You control exactly how unlikely 'very unlikely' is — that is the significance level, usually 5% (0.05). It means: if the null is actually true, you will wrongly reject it 5% of the time, by design.

Imagine a courtroom. The defendant is presumed innocent until proven guilty. That presumption is the 'null hypothesis' — the default position that says 'no effect', 'no difference', 'no guilt'. The prosecution must bring evidence — data. The judge does not ask 'is the defendant actually guilty?' because that is unknowable. Instead, the judge asks: 'If the defendant were truly innocent, how unlikely is it that we would see evidence this strong?' This is exactly the p-value. If the evidence is so strong that it would occur by chance less than 5% of the time under innocence (p < 0.05), the judge declares 'not consistent with innocence' — we reject the null. But the judge never says 'the defendant is innocent' when the evidence is weak — only 'not enough evidence to convict'. That is the crucial distinction: failing to reject the null is not the same as proving it true. The analogy breaks down where the law requires 'beyond a reasonable doubt' — a very strict threshold — while science often uses 5% as a convention. Also, in a courtroom the judge does not decide the probability of guilt; in statistics, the evidence (data) is fixed and the hypothesis is what gets a probability — but the p-value is the probability of the data given the null, not the probability of the null given the data. That is a subtle but essential boundary.

Hypothesis testing is a formal procedure for deciding whether data provide enough evidence against a null hypothesis, controlling the rate of false rejections at a chosen level.

This is the first topic in your Statistical Inference journey. You have already studied Descriptive Statistics — means, medians, standard deviations — which tell you what your sample looks like. Hypothesis testing is the next step: it tells you whether what you see in the sample is likely to be true of the whole population, or just a fluke of sampling. It builds on Probability Theory (the idea of chance and likelihood). It directly leads to ANOVA (comparing several groups at once), Cross-Validation Partitions (testing whether a model's performance difference is real), and Model Evaluation — and it is the conceptual foundation for the modern push toward estimation and uncertainty intervals instead of just binary 'significant / not significant'. Your library notes also mention Significance Levels — that is the knob you choose here, the 5% (or whatever you set) that controls your false positive rate. You also have notes on Null and Alternate Hypotheses, which are the two competing statements this procedure adjudicates.