What Is a P-Value and How Do You Interpret It?

A friend hands you a coin and says it might be rigged to land heads more often. You flip it 20 times. You get 16 heads.

Is that suspicious? Sixteen out of twenty is more heads than tails, sure. But coins land on heads sometimes anyway, even fair ones. How do you tell "this looks rigged" from "this is just what randomness looks like sometimes"?

That question, made precise, is what a p-value answers. Not whether the coin is rigged. Just how surprising your 16 heads would be if it weren't.

What are you actually asking?

Here's the setup we always start from: assume nothing unusual is going on. The coin is fair. The drug does nothing. The website redesign changes nothing. This assumption is called the null hypothesis, and it's not something you believe. It's a baseline you're testing against.

Then you ask one specific question: if the null hypothesis were true, how likely is it that I'd see a result this extreme, or more extreme, just by chance?

That probability is the p-value.

Drag the slider above. At 10 heads out of 20, nothing about a fair coin looks off, so the p-value sits near 1. Push it toward 16 or 17, and the shaded region, the outcomes at least that far from a 50/50 split, shrinks fast. Push it to 20 heads and the p-value collapses toward zero. A p-value is a tail area. It's the fraction of "fair coin" outcomes that would look at least as extreme as the one you actually got.

Where does the 0.05 come from?

Once you have a p-value, you compare it to a threshold you picked before looking at the data, almost always 0.05. If your p-value lands below that line, the result is called statistically significant.

But 0.05 isn't a law of nature. It's a convention, a line drawn in the sand that says "this would happen by chance less than 1 time in 20." Nothing magical happens right at that boundary. p = 0.049 and p = 0.051 are practically the same amount of evidence, even though one gets to wear the "significant" badge and the other doesn't.

Here the horizontal axis is a test statistic (think of it as "how many standard errors away from zero your result landed"). Drag the marker and watch the two shaded tails shrink continuously as the result gets more extreme. The p-value doesn't jump at 0.05. It flows smoothly, and 0.05 is just wherever we decided to draw the cutoff line.

We write the two-tailed p-value for a normal test statistic $z$ as:

where $\Phi$ is the standard normal cumulative distribution function, the area under the bell curve up to a point. All that formula is doing is what you just watched: measuring how much curve area sits beyond your result, on both sides.

The mistake almost everyone makes

Here's the part that trips people up, even scientists who use p-values every week. A p-value is not the probability that the null hypothesis is true. p = 0.03 does not mean "there's a 3% chance the coin is fair." It means something narrower: "if the coin were fair, results this extreme would happen 3% of the time."

Those are different claims. The first is about the coin. The second is about the data, assuming a coin you haven't actually confirmed.

The cleanest way to feel this difference is to simulate it.

Set the toggle to "no real effect" and run a bunch of trials. Watch the histogram of p-values that comes out. It doesn't pile up near 1, like you'd expect if p measured "probability the null is true." It spreads out flat, roughly even across the whole range from 0 to 1. And look at that shaded region on the left: about 5% of trials land below 0.05, purely by chance, even though there was never anything real to find.

That's the whole misconception in one picture. A false positive rate of 5% isn't a flaw in the method. It's built into the definition. If you run enough tests on pure noise, some of them will cross the line just by luck.

Now flip the toggle to "real effect exists" and run it again. This time the histogram piles up hard near zero. A real effect doesn't guarantee a tiny p-value on any single trial, but it shifts the odds heavily in that direction. That shift, not any single p-value, is what separates real signal from chance.

A tiny effect can still "win"

There's a second misconception, just as common: that a low p-value means the effect is big or important. It doesn't. A p-value mixes together two completely different things: how large an effect is, and how much data you collected. Enough data and even a genuinely trivial effect crosses the significance line.

The true effect in that simulation never moves. It's fixed, and it's small enough that nobody would care about it in practice. But drag the sample size up into the thousands and the p-value falls anyway, eventually diving under 0.05. Nothing about the effect changed. Only the amount of data did.

This is why "statistically significant" and "actually matters" are not the same sentence. A drug that lowers blood pressure by 0.1 points can hit p < 0.05 in a large enough trial. That doesn't make it worth prescribing. The p-value tells you the result probably isn't pure noise. It says nothing about whether the effect is big enough to care about, that's a separate question, usually answered by looking at the effect size directly.

Putting it together

Go back to the coin. You flip it 20 times and get 16 heads. Running the numbers, that gives a p-value around 0.012, well under the usual 0.05 line. Does that mean the coin is rigged?

Not quite. It means: if the coin were fair, a split this lopsided or worse would happen only about 1.2% of the time. That's suspicious enough that most people would want to investigate further, maybe flip it another 20 times. But it isn't proof. Rare events do happen, just rarely. A p-value gives you a number to weigh against how costly it would be to be wrong, not a verdict handed down from the data itself.

The short version

A p-value answers one narrow question: assuming nothing unusual is going on, how surprising is the data you actually collected? It's a tail area on a distribution, nothing more mystical than that. It is not the probability that your hypothesis is true, that number depends on evidence you haven't included, like how plausible the effect was to begin with. And a small p-value is not the same as a big, important effect, since enough data can push even a meaningless effect below any threshold you pick.

The number is useful. It's just answering a narrower, stranger question than most headlines make it sound like.

All visualizations are interactive React components running entirely in your browser, computing binomial and normal probabilities directly with no external statistics libraries. The histogram simulator draws fresh random samples on every trial using a Box-Muller transform.