The Most Common A/B Testing Mistake
A marketer runs a test, Variant B converts at 13% against Variant A's 10%, and the test gets called a week early because "B is clearly winning." The trouble is that a 3-percentage-point difference on a few hundred or thousand visitors can easily be random noise — the only way to know is to run the actual statistics, not eyeball the percentages.
This is where a two-proportion z-test comes in. It's the standard statistical method for comparing two conversion rates, and it produces two numbers that matter far more than the raw percentage difference: a z-score and a p-value.
How the Two-Proportion Z-Test Actually Works
The test starts by pooling both variants into a single combined conversion rate, on the logic that if there were truly no difference between A and B, this pooled rate is what both would converge to:
pooled p = (conversions A + conversions B) / (visitors A + visitors B)
From there, the standard error of the difference between the two rates is calculated:
SE = sqrt( p × (1-p) × (1/visitorsA + 1/visitorsB) )
The z-score is simply how many standard errors apart the two observed rates are:
z = (rateB - rateA) / SE
And the p-value converts that z-score into a probability — specifically, the probability of seeing a difference this large (or larger) purely by chance, if there were actually no real difference between the variants. A two-tailed p-value below 0.05 is the conventional threshold for "statistically significant at 95% confidence."
A Worked Example
Take 1,000 visitors and 100 conversions on Variant A (a 10% rate), against 1,000 visitors and 130 conversions on Variant B (a 13% rate). Running the math: pooled p = 0.115, standard error ≈ 0.0143, z ≈ 2.10, and the two-tailed p-value comes out to roughly 0.036 — below 0.05, so this result is statistically significant. Compare that to a smaller gap: 1,000/100 versus 1,000/105 (10% vs. 10.5%) produces z ≈ 0.37 and p ≈ 0.71 — nowhere near significant, meaning that "winner" is most likely just noise.
Why This Matters More on Small Sample Sizes
The smaller the visitor count, the larger a percentage-point gap needs to be before it clears the significance bar. A 3-point lift on 50 visitors per variant means almost nothing statistically; the same 3-point lift on 5,000 visitors per variant is a very different story. Running the z-test before declaring a winner protects against shipping a change — and confidently reporting it to stakeholders — based on what amounts to a coin flip.
Bottom Line
Conversion rate optimization only works if the underlying tests are read correctly. A raw percentage difference tells you almost nothing on its own; the z-score and p-value tell you whether that difference is real. Running the numbers before calling a test takes seconds and prevents shipping changes based on statistical noise.