Confidence intervals and tests Updated November 2026

A/B test significance

An A/B test is significant when the difference in conversion rates is unlikely to be chance: run a two-proportion z-test and check that the p-value is below 0.05. Report the relative lift, (rate B - rate A) / rate A, together with a confidence interval, and fix the sample size before starting.

Syntax

lift = (pB - pA) / pA;   z = (pB - pA) / sqrt( p * (1 - p) * (1/nA + 1/nB) )

Part	What it is
`pA, pB`	conversion rates of control and variant
`nA, nB`	visitors in each group
`p`	the pooled conversion rate

Examples

A: 50 / 1000 = 5.0%. B: 70 / 1000 = 7.0%
lift = (0.07 - 0.05) / 0.05

Result: 40% relative lift (2 percentage points absolute)

Same test: pooled p = 0.06, SE = 0.01062
z = 0.02 / 0.01062 = 1.88; p = 2 * (1 - 0.9701)

Result: p = 0.060, not significant at 0.05

Sample size per group to detect 5% vs 6% at alpha 0.05, power 0.8
(1.96 + 0.84)^2 * (0.05 * 0.95 + 0.06 * 0.94) / 0.01^2

Result: about 8,150 visitors per group

Excel p-value from z
=2 * (1 - NORM.S.DIST(ABS(z), TRUE))

Result: two-sided p-value

Common mistakes

Stopping the test as soon as p dips below 0.05; checking repeatedly inflates the false positive rate far above 5%, so set the sample size in advance.
Reporting a 40% lift from a test with p = 0.06 as a win; the confidence interval includes zero, so the lift is not established.
Running a test with too few conversions; with 20 conversions per arm even a large real lift usually comes out non-significant.

Practice A/B test significance

These drills check your answer on live data, free, no account needed.

A/B test: relative liftIn an A/B test of a checkout page, the control (A) converted 236 of 2,000 visitors and the variant (B) converted 253 of 1,800. What is the relative lift of B over A, as a percent of A's rate? A drop is negative. Give the percent rounded to 2 decimal places.
A/B test: p-value and decisionAn A/B test of a signup form: control 177 signups out of 1,500 visitors, variant 174 out of 1,200. Using a two-sided two-proportion z test with a pooled standard error, what is the p-value? Then decide at α = 0.05. Round to 4 decimal places.
Two-proportion z testTwo branches piloted a cross-sell offer. Branch A: 46 of 200 customers accepted. Branch B: 84 of 600 accepted. Test H0: pA = pB against H1: pA ≠ pB. Find the z statistic (A minus B, pooled standard error), then decide at α = 0.01. Round to 2 decimal places.

Questions

How many visitors do I need for an A/B test?

It depends on the baseline rate and the smallest lift worth detecting. Going from 5% to 6% at 80% power needs about 8,000 per group; smaller lifts need far more. Use the sample size formula before launching.

What is the difference between absolute and relative lift?

Absolute lift is the difference in rates (7% - 5% = 2 points). Relative lift divides by the control rate (2 / 5 = 40%). Report both to avoid confusion.

Should I use a one-sided or two-sided test?

Two-sided unless a worse variant is truly irrelevant. One-sided halves the p-value and is often misused to claim significance.

Two-proportion z-testHow to test whether two proportions differ with the two-proportion z-test: pooled proportion, standard error, z statistic and p-value, with a worked example.
p-valueA p-value is the probability of a result at least as extreme as the observed one if the null is true. How to compute it from z and t, and common misreadings.
Statistical powerStatistical power is the probability a test detects a real effect, 1 - beta. What drives it, the 80% convention and sample size formulas with examples.
Sample sizeSample size formulas n = (z sigma / E)^2 for a mean and n = z^2 p(1 - p) / E^2 for a proportion, worked examples giving 97 and 385, and rounding rules.