📊

A/B Test Significance Calculator

Check whether the difference between your control and variant conversion rates is statistically significant with a two-proportion z-test.

Control visitors

Control conversions

Variant visitors

Variant conversions

Confidence level

Loading calculator…

How it works

p̂ = (cA + cB) ÷ (nA + nB) · SE = √[ p̂(1 − p̂)(1/nA + 1/nB) ] · z = (pB − pA) ÷ SE · p-value = 2·(1 − Φ(|z|))

A/B test significance uses a two-proportion z-test. It pools the two conversion rates into one estimate, computes how many standard errors apart the observed rates are (the z-statistic), and turns that into a two-tailed p-value using the normal distribution. A result is called significant when the p-value falls below your chosen threshold (5% for 95% confidence), meaning a gap this large would be unlikely if the two versions truly converted at the same rate.

Worked example

Control: 100 of 2,000 convert (5.0%). Variant: 140 of 2,000 (7.0%). Pooled p̂ = 240 ÷ 4,000 = 0.06, SE = √[0.06·0.94·(1/2000+1/2000)] ≈ 0.00751. z = (0.07 − 0.05) ÷ 0.00751 ≈ 2.66, p ≈ 0.0078 — below 0.05, so the 2-point lift is significant at 95%.

Sources: Two-proportion z-test — Wikipedia

Frequently asked questions

What does the p-value actually mean?

It is the probability of seeing a difference at least this large purely by chance if the two versions really converted at the same rate. A small p-value (below 0.05 for 95% confidence) is evidence the versions differ; it is not the probability that the variant is better.

How long should I run the test before trusting this?

Reach your planned sample size before reading significance, and let the test cover full business cycles (whole weeks) so weekday and weekend behaviour are represented. Peeking early and stopping the moment p dips below 0.05 inflates false positives.

Is this one-tailed or two-tailed?

Two-tailed — it tests whether the rates differ in either direction, which is the safe default. A one-tailed test only detects an improvement and roughly halves the p-value, so only use it when a decrease is impossible or irrelevant.

Related calculators

Two-Proportion Z-Test Calculator

Compare two proportions (conversion rates, success rates) with a pooled z-test — get the z statistic, two-tailed p-value and a plain-language verdict.

A/B Test Sample Size Calculator

Find how many visitors per variant your A/B test needs to detect a given lift at your chosen confidence and power.

Conversion Uplift Calculator

Compare a control and variant conversion rate to get the absolute and relative uplift — the two lift numbers people constantly confuse.

Confidence Interval for a Proportion Calculator

Estimate a confidence interval for a population proportion (a percentage or rate) from your successes and sample size using the Wald method.

Z-Score Calculator

Turn a raw value into a z-score — how many standard deviations it sits above or below the mean — with its percentile under a normal curve.

This calculator is for educational purposes. Double-check important results before acting on them.