The Most Common A/B Testing Mistake
A marketer runs a test, Variant B converts at 13% against Variant A's 10%, and the test gets called a week early because "B is clearly winning." The trouble is that a 3-percentage-point difference on a few hundred or thousand visitors can easily be random noise — the only way to know is to run the actual statistics, not eyeball the percentages.
This is where a two-proportion z-test comes in. It's the standard statistical method for comparing two conversion rates, and it produces two numbers that matter far more than the raw percentage difference: a z-score and a p-value.
How the Two-Proportion Z-Test Actually Works
The test starts by pooling both variants into a single combined conversion rate, on the logic that if there were truly no difference between A and B, this pooled rate is what both would converge to:
pooled p = (conversions A + conversions B) / (visitors A + visitors B)
From there, the standard error of the difference between the two rates is calculated:
SE = sqrt( p × (1-p) × (1/visitorsA + 1/visitorsB) )
The z-score is simply how many standard errors apart the two observed rates are:
z = (rateB - rateA) / SE
And the p-value converts that z-score into a probability — specifically, the probability of seeing a difference this large (or larger) purely by chance, if there were actually no real difference between the variants. A two-tailed p-value below 0.05 is the conventional threshold for "statistically significant at 95% confidence."
A Worked Example
Take 1,000 visitors and 100 conversions on Variant A (a 10% rate), against 1,000 visitors and 130 conversions on Variant B (a 13% rate). Running the math: pooled p = 0.115, standard error ≈ 0.0143, z ≈ 2.10, and the two-tailed p-value comes out to roughly 0.036 — below 0.05, so this result is statistically significant. Compare that to a smaller gap: 1,000/100 versus 1,000/105 (10% vs. 10.5%) produces z ≈ 0.37 and p ≈ 0.71 — nowhere near significant, meaning that "winner" is most likely just noise.
Why This Matters More on Small Sample Sizes
The smaller the visitor count, the larger a percentage-point gap needs to be before it clears the significance bar. A 3-point lift on 50 visitors per variant means almost nothing statistically; the same 3-point lift on 5,000 visitors per variant is a very different story. Running the z-test before declaring a winner protects against shipping a change — and confidently reporting it to stakeholders — based on what amounts to a coin flip.
Bottom Line
Conversion rate optimization only works if the underlying tests are read correctly. A raw percentage difference tells you almost nothing on its own; the z-score and p-value tell you whether that difference is real. Running the numbers before calling a test takes seconds and prevents shipping changes based on statistical noise.
Frequently Asked Questions
Run a two-proportion z-test on the two variants' conversion rates rather than comparing the raw percentages by eye. The test produces a p-value, and a two-tailed p-value below 0.05 is the conventional threshold for calling a result statistically significant at 95% confidence — above that, the difference could easily be random chance.
It's the standard statistical method for comparing two conversion rates: it pools both variants into a combined rate, calculates the standard error of the difference between them, and converts the gap between the two observed rates into a z-score and p-value. Those two numbers tell you whether an observed difference is likely real, which a raw percentage comparison on its own cannot.
Not necessarily — it depends heavily on sample size. In a worked example with 1,000 visitors per variant, a 10% versus 13% conversion rate produces z ≈ 2.10 and p ≈ 0.036, which is statistically significant, but the same 3-point-style gap on much smaller sample sizes, like 50 visitors per variant, would mean almost nothing statistically.
A two-tailed p-value below 0.05 is the conventional threshold, corresponding to 95% confidence that the observed difference isn't due to chance. A comparison like 10% versus 10.5% on 1,000 visitors per variant, for instance, produces z ≈ 0.37 and p ≈ 0.71 — nowhere near significant, meaning that 'winner' is most likely noise rather than a real effect.
Yes — an A/B test significance calculator takes visitor and conversion counts for each variant and runs the two-proportion z-test automatically, returning the z-score and p-value instead of requiring the pooled-rate and standard-error math to be done by hand. Running the numbers before declaring a winner takes seconds and prevents shipping a change based on statistical noise. It's a one-time $6.99 purchase — no subscription, no account required.