Free A/B Test
Significance Calculator

Statistical significance tells you how likely it is that the difference you see between two variants is real and not due to random chance. When a result is statistically significant at 95% confidence, it means there is only a 5% probability that the difference happened by chance. It does not tell you how large the difference is, only that it is unlikely to be accidental.

This calculator uses a two-proportion Z-test. You enter the number of visitors and conversions for both your control (A) and variant (B). The calculator computes the conversion rate for each, pools the two proportions to get a shared standard error, then calculates a Z-score and converts it to a p-value and confidence level. It also shows the relative uplift and, if the result is not yet significant, an estimate of how many more visitors you need.

A p-value is the probability of seeing a difference this large (or larger) between two variants purely by chance, assuming there is no real difference. A p-value of 0.05 means there is a 5% chance the result is random. The lower the p-value, the stronger the evidence that the difference between A and B is real. A p-value below 0.05 is typically treated as statistically significant.

95% confidence is the standard for most digital marketing tests. It means you accept a 5% chance of a false positive. Use 99% confidence for tests with high business impact, such as pricing changes or checkout flow changes, where the cost of a wrong decision is significant. Use 90% confidence only for low-stakes exploratory tests where speed matters more than certainty.

The sample size depends on your current conversion rate and the minimum improvement you want to detect. As a rough guide, if your control converts at 5% and you want to detect a 20% relative uplift (to 6%), you need roughly 8,000 to 10,000 visitors per variant at 95% confidence. This calculator shows you the required sample size whenever your current result is not yet significant.

Use the Plan a test mode at the top. Enter your current conversion rate, the smallest relative lift you want to detect (your minimum detectable effect), your confidence level and statistical power, and the calculator returns the visitors needed per variant. Add your daily traffic and it converts that into how many days to run. Larger baselines and larger detectable lifts need less traffic, while small lifts can need tens of thousands of visitors per variant. Whatever the maths says, run for at least one to two full weeks so weekday and weekend behaviour are both captured.

A single uplift figure such as +20% hides how uncertain it is. The confidence interval is the range the true uplift most likely falls within — for example +20% might carry a 95% interval of −3% to +43%. If that interval crosses 0%, the data cannot rule out that your variant is actually no better, or even worse, so it is not a proven winner even when the headline number looks positive. A narrower interval means a more trustworthy result, and intervals tighten as you collect more data. This calculator shows the confidence interval on every result.

Statistical significance means the result is unlikely to be random. Practical significance means the result is large enough to be worth acting on. A test can be statistically significant with a 0.1% uplift if you have millions of visitors, but a 0.1% improvement may not be worth changing the page for. Always check both: a real difference that is too small to matter is not a useful finding.

No. Stopping a test the moment it hits significance is called peeking, and it inflates your false positive rate significantly. Run your test until you reach the sample size you calculated before the test started. If you did not pre-calculate a sample size, run the test for at least one to two full business cycles (typically one to two weeks) before making a decision.

This calculator uses a two-tailed test, which checks whether B is different from A in either direction. A one-tailed test checks only whether B is better than A. Two-tailed tests are more conservative and are the correct default for A/B testing, because variants can sometimes perform worse than control, and you need to know that too.

This is a case where statistical significance and practical significance diverge. With a very large sample size, even a 0.05% difference can be statistically significant, but implementing a change for a 0.05% lift is rarely worth the effort. Set a minimum detectable effect before you start your test and only act on results that clear both the statistical and practical bars.

Yes, completely free. No account, no credit card, no limit on how many tests you can calculate. All the computation runs in your browser — nothing is sent to any server.