Making Decisions with Data
Every day, people make claims. A new diet "helps you lose weight faster." A school program "improves reading scores." A company's product is "preferred by most customers." But how do you know if these claims are actually true, or just lucky results?
Hypothesis testing is the method statisticians use to answer this question. It's a structured way of using data to decide whether a claim has real evidence behind it - or whether the results could easily be explained by chance.
The Courtroom Analogy
The easiest way to understand hypothesis testing is to think about how a courtroom works.
In a trial, the defendant is presumed innocent until proven guilty. The prosecution has to present enough evidence to overcome that presumption. If the evidence is strong enough, the jury says "guilty." If not, the defendant is found "not guilty" - which doesn't necessarily mean innocent, just that there wasn't enough proof.
Hypothesis testing works the same way:
- We start by assuming nothing special is happening (the "innocent" assumption).
- We collect data (the "evidence").
- If the data is convincing enough, we reject the starting assumption.
- If not, we stick with it - not because we've proven it true, but because we don't have enough evidence to say otherwise.
Null and Alternative Hypotheses
Every hypothesis test starts with two competing statements:
The Null Hypothesis (H₀): This is the "nothing is happening" statement. It says there is no effect, no difference, no relationship. It's the default assumption - like "innocent until proven guilty."
The Alternative Hypothesis (H₁): This is the claim you're actually trying to support. It says there IS an effect, a difference, or a relationship.
A pharmaceutical company develops a new headache medication and wants to know if it works better than a sugar pill (placebo).
Null Hypothesis (H₀): The new medication is no better than the placebo. Any difference in headache relief is due to chance.
Alternative Hypothesis (H₁): The new medication provides more headache relief than the placebo.
They give the real medication to 100 patients and the placebo to another 100. After collecting the results, they use a statistical test to see if the medication group did significantly better. If the evidence is strong enough, they reject the null hypothesis and conclude the medication likely works.
The Steps of a Hypothesis Test
Here's the basic process, step by step:
- State your hypotheses. Write down the null hypothesis (nothing is happening) and the alternative hypothesis (something is happening).
- Collect data. Run your experiment or gather your observations.
- Analyze the data. Use a statistical test to calculate how likely your results would be IF the null hypothesis were true.
- Make a decision. If the results would be very unlikely under the null hypothesis, reject it. Otherwise, don't reject it.
The phrase "very unlikely" usually means less than a 5% chance, but we'll cover that threshold more in the lesson on p-values.
Two Kinds of Mistakes
No matter how careful you are, there's always a chance you'll reach the wrong conclusion. There are exactly two ways things can go wrong:
Type I Error (False Alarm)
This happens when you reject the null hypothesis even though it's actually true. You conclude something is happening when it really isn't.
In the courtroom analogy, this is convicting an innocent person.
Example: You conclude the new medication works, but it actually doesn't - the patients just happened to feel better by coincidence.
Type II Error (Missed Discovery)
This happens when you fail to reject the null hypothesis even though the alternative is actually true. You miss a real effect.
In the courtroom analogy, this is letting a guilty person go free.
Example: The medication actually does work, but your study didn't have enough patients to detect the difference, so you conclude there's no effect.
Think of a smoke detector. A Type I error is when the alarm goes off but there's no fire - a false alarm. Annoying, but not dangerous. A Type II error is when there IS a fire but the alarm doesn't go off - a missed detection. That's potentially catastrophic.
In statistics, you often have to balance these two risks. Making it harder to trigger a "detection" (requiring stronger evidence) reduces false alarms but increases the chance you'll miss something real.
How Do You Reduce Errors?
There are practical ways to manage both types of mistakes:
- Larger sample sizes make it easier to detect real effects, reducing Type II errors. More data gives you a clearer picture.
- Stricter evidence thresholds (like requiring a 1% chance instead of 5%) reduce Type I errors, but they also make it harder to detect real effects.
- Better study design - controlling for other variables, using randomization - makes your evidence more trustworthy overall.
What "Statistically Significant" Means
When you see the phrase "statistically significant" in a news article or research paper, it means the researchers performed a hypothesis test and decided to reject the null hypothesis. Their data was unlikely enough under the "nothing is happening" assumption that they concluded something real is going on.
It does not mean the result is large, important, or practically useful. A drug might produce a statistically significant improvement of 0.1% - real, but probably not worth taking. "Significant" in statistics just means "unlikely to be due to chance alone."
When Is Hypothesis Testing Used?
Hypothesis testing shows up everywhere:
- Medicine: Testing whether a new treatment is better than existing ones.
- Business: A/B testing on websites to see if a new design gets more clicks.
- Education: Checking if a new teaching approach actually improves grades.
- Government: Determining if a policy change reduced crime rates.
Hypothesis testing is a structured way to use data to evaluate claims. You start by assuming nothing is happening (null hypothesis), then check if your data provides strong enough evidence to reject that assumption. Two types of errors are always possible: false alarms (Type I) and missed discoveries (Type II). Understanding this framework helps you critically evaluate claims you encounter in news, health, and everyday decisions.