Hypothesis Testing

You make decisions based on limited information every day. A colleague says their new spreadsheet formula runs faster than yours. A vendor claims their delivery service is more reliable than your current one. Without a formal method, you are just looking at two numbers and hoping for the best. Hypothesis testing gives you a controlled way to decide if a difference you see in a small set of data is a real effect or just random luck. This matters because it protects you from acting on noise. When you read in a report that "Method A scored 85 and Method B scored 82," hypothesis testing tells you whether that 3-point gap is meaningful evidence or just random variation. It is the standard tool for validating claims in AI models, medical studies, and business performance.

Imagine you have a bag of marbles. You cannot see inside, but you were told the bag contains exactly half red marbles and half blue marbles. You suspect this might be wrong—perhaps the bag is actually mostly red. You take out a handful to check—a sample. Suppose your handful has 8 red marbles and only 2 blue marbles. Is this enough to prove the bag is not half-and-half? Or could such a handful happen by pure chance even if the bag truly was balanced?

Hypothesis testing answers this by asking: "If the bag really were half red and half blue, how likely is it that I would draw a handful this extreme (8 red, 2 blue)?"

In our example, let us say the math shows this specific result would happen only 3% of the time by chance if the bag were truly balanced. We compare this probability to a pre-set threshold called the significance level, usually set at 5% (or 0.05). Because 3% is smaller than 5%, we decide the balanced-bag story is unlikely to be true. We reject that story.

Crucially, we never prove the bag is mostly red. We only conclude that if the bag were balanced, seeing our data would be very rare. Therefore, we act as though the bag is not balanced. The significance level controls how often we make a mistake: by choosing 5%, we accept that if the bag really were balanced, we will wrongly reject that fact 5% of the time simply due to bad luck in sampling.

Think of a courtroom trial. The defendant starts with a presumption of innocence. This is your null hypothesis—the default assumption that nothing unusual is happening, or there is no difference between two groups. The prosecution brings evidence (data) to try to prove guilt.

The judge does not ask, "Is the defendant actually guilty?" because absolute truth is hard to find. Instead, the judge asks: "If the defendant were truly innocent, how unlikely is it that we would see evidence this strong against them?"

This calculation is the p-value. If the evidence is so strong that it would occur by chance less than 5% of the time assuming innocence (p < 0.05), the judge rules "guilty"—in our terms, we reject the null hypothesis. However, if the evidence is weak, the judge does not rule "the defendant is innocent." The judge only says "not enough evidence to convict." This mirrors statistical practice: failing to reject the null hypothesis is not proof that the null is true; it is merely a statement that we do not have strong enough evidence to doubt it.

This analogy holds well for the logic of burdening proof, but note this limitation: in law, the jury decides the probability of guilt based on testimony. In statistics, the data is fixed and observed, and we calculate the probability of seeing that data under a specific assumption. We do not calculate the probability that the null hypothesis itself is true.

Hypothesis testing is a method for deciding whether sample data provides strong enough evidence to reject a default assumption (the null hypothesis) in favor of an alternative, while keeping the risk of a false positive within a pre-defined limit (the significance level).

This concept sits next to Descriptive Statistics, which summarizes what your data looks like (e.g., averages and spreads), whereas Hypothesis Testing determines if those summaries reflect reality or sampling error. It also connects directly to Confidence Intervals, which provide a range of plausible values for a parameter rather than just a yes/no decision, offering more nuance than the binary outcome of testing.