Understanding p-values and Why They Are Not What You Think

The Most Misunderstood Number in Science

You just ran an experiment. You look at the output. The screen says p equals 0.03. You feel a wave of relief. You assume there is a 97 percent chance your hypothesis is correct. This is the moment where many researchers go wrong. That number does not tell you the probability that your claim is true. It tells you something much more specific and much more limited. Misinterpreting this one number is a major reason why so many scientific studies fail to replicate. People publish results that do not hold up because they thought a small p-value meant their claim was probably true.

What a P-Value Actually Measures

To understand a p-value, you have to start with a thought experiment. Imagine a world where your hypothesis is completely wrong. This is called the null hypothesis world. In this world, any patterns you see are just the result of randomness, variance, and sampling noise.

A p-value is the probability of seeing data at least as extreme as what you observed, assuming that the null hypothesis is true. It is a measure of surprise. If the p-value is low, it means your data would be very surprising in a world where nothing is happening. It does not mean your specific theory is the reason for the data. It just means the null model is having a hard time explaining what you see. The full walkthrough of this thought experiment is in the video above.

The Mathematical Flip

People often get the direction of the logic backwards. They think the p-value is the probability of the hypothesis given the data. Mathematically, it is actually the probability of the data given the null hypothesis. These two things are not the same.

Flipping the condition in your head changes the entire meaning of the test. When you flip the direction, you create false confidence. You start believing in claims that might just be a lucky streak of noise. This mistake is costly. It leads teams to roll out changes that do nothing. It also leads them to dismiss meaningful effects that happened to land in an unlucky sample.

The Problem with the 5 Percent Rule

Most people use 0.05 as a magic cutoff. If the number is lower, they call it a win. If it is higher, they call it a failure. This cutoff is not a truth detector. It is a convention. Think of it as a policy for managing false alarms.

If you always use a 5 percent rule, you are agreeing that you are okay with a 5 percent false alarm rate over many experiments. It controls the behavior of your testing process over time. It does not tell you the truth status of one single claim. Treating the cutoff as an oracle for a specific study is a mistake. You are letting a single random draw dictate your entire conclusion.

Why Sample Size Changes Everything

A p-value is heavily influenced by how much data you have. This leads to two common traps.

1. The Massive Sample Trap. If you have millions of data points, even a tiny and meaningless difference will produce a small p-value. The test becomes so sensitive that it flags every tiny bit of noise as significant.
2. The Tiny Sample Trap. If your sample is too small, a massive and important effect might produce a large p-value. The data are so noisy that the test cannot distinguish the effect from random chance.

Significance is not the same thing as importance. A p-value tells you if a signal is likely there. It does not tell you if that signal is large enough to matter in the real world. You need other tools to answer that question.

Stop Peeking at Your Data

The p-value assumes you have a plan. It assumes you chose your test and your sample size before you looked at the results. If you change your plan halfway through, the p-value loses its meaning. This often happens through a process called peeking.

You run a test and see a p-value of 0.07. You decide to collect just a few more data points to see if it drops below 0.05. If you do this, you are no longer telling the truth. The null distribution you are comparing against assumes you did not peek. By checking repeatedly and stopping only when you get the result you want, you are guaranteed to find a false positive eventually. This is why a flexible analysis without a plan yields a steady flow of spurious discoveries.

The Correct Way to Speak Statistics

If you want to avoid these traps, you should change how you talk about your results. Instead of saying your hypothesis is probably true, try reading the p-value out loud with the null condition included. If your p-value is 0.03, say this. If the null were exactly true, data at least this extreme would show up about 3 percent of the time in this specific test design.

This phrasing keeps the focus on the null model. It reminds you that you are measuring how much your data clashes with a world of pure randomness. It is evidence against the null. It is not proof of your favorite alternative. It is certainly not a probability that your claim is true.

How to Improve Your Analysis

P-values are just one tool in the box. They are a yes or no signal about incompatibility with a null model. To get a full picture of your data, you should look beyond that one number.

1. Report effect sizes to show how much a change actually matters.
2. Use confidence intervals to show the range of plausible values.
3. Stick to a pre-defined analysis plan to avoid p-hacking.
4. Use holdout sets to validate your findings on new data.
5. Name your null model and your statistic clearly before you look at the data.

Statistics is about more than just hitting a target number. It is about understanding the uncertainty in your measurements. Once you stop treating the p-value as a badge of credibility, it becomes a much more useful tool for discovery. It becomes a precise tool rather than a vague signal.

Moving Toward Confidence Intervals

Now that you know how to read surprise under a null model, the next step is defining what sizes of effects are actually plausible. This is where confidence intervals come in. They help you state the margin of error and give you a clearer picture of the real world. After you know how to read surprise under a null, you need a clear way to state plausible sizes for the effect you care about.

Have you ever seen a result that looked significant but turned out to be totally wrong when you tried to replicate it?