How Many Visitors Do You Need to A/B Test? (The Maths Nobody Shows You)

There is no universal number, but the ranges are unforgiving: at a 5% conversion rate, detecting a 20% relative improvement takes roughly 8,155 visitors per variant, and at a 2% rate chasing a 10% improvement it’s about 80,700 per variant. The figure is driven by four inputs you control or accept: your baseline conversion rate, the minimum effect you want to detect, your significance level and your statistical power. Below is the full calculation so you can compute your own number rather than trusting a widget.

Almost every article on this topic gives you a round figure. A thousand visitors per variant. Three hundred conversions. A hundred conversions per arm. These numbers are not wrong so much as unanchored: they’re a particular answer to a particular set of assumptions that nobody stated.

The actual calculation is short enough to do by hand, and doing it once changes how you plan tests permanently.

The four inputs that determine the number

1. Baseline conversion rate. How your current page performs. Lower baselines need dramatically more traffic, because rare events carry more relative noise. This is the input people underestimate most.
2. Minimum detectable effect (MDE). The smallest improvement you care about, usually expressed as a relative change. Going from 5% to 6% is a 20% relative lift. Smaller MDEs cost enormously more traffic, and the relationship is roughly quadratic: halving the effect you want to catch quadruples the sample.
3. Significance level (α). Your tolerance for declaring a winner that isn’t real. The convention is 5%, giving a two-sided critical value of Z_α/2 = 1.96.
4. Statistical power (1 − β). Your probability of detecting a real effect when one exists. The convention is 80%, giving Z_β = 0.8416. Power is the input teams silently sacrifice, and it’s the reason so many tests come back “inconclusive”.

Two things people forget: the number is per variant, so a simple A/B needs double it, and the sample must be decided before the test starts, not discovered by watching the dashboard.

The formula, and one cell worked out by hand

For comparing two proportions with a two-sided test:

n per variant = (Z_α/2 + Z_β)² × [ p₁(1−p₁) + p₂(1−p₂) ] ÷ (p₂ − p₁)²

At the conventional 95% significance and 80% power, the leading constant is fixed:

(1.96 + 0.8416)² = 2.8016² = 7.85

Take a 5% baseline and a 20% relative lift, so p₁ = 0.05 and p₂ = 0.06:

• p₁(1−p₁) = 0.05 × 0.95 = 0.0475
• p₂(1−p₂) = 0.06 × 0.94 = 0.0564
• sum = 0.1039
• (p₂ − p₁)² = 0.01² = 0.0001
• n = 7.85 × 0.1039 ÷ 0.0001 = 8,156 per variant

So about 8,155 per variant using unrounded constants, or 16,310 visitors for the full test. Every number in the table below comes from that same formula. I derived them rather than sourcing them, precisely so you can check any cell with a calculator and four lines of arithmetic.

Required sample size per variant

Two-sided test, 95% significance, 80% power. Double each figure for the total across both arms.

Baseline conversion rate	Detect +10% relative	Detect +20% relative	Detect +50% relative
2%	80,679	21,106	3,822
5%	31,231	8,155	1,468
10%	14,748	3,838	683

Three things jump out of that grid.

The top-left cell is the trap most teams walk into. A 2% baseline is completely normal for cold paid traffic, and a 10% relative improvement is a realistic result from a good copy rewrite. Detecting it honestly costs 161,358 visitors across both arms. Most companies running A/B tests are nowhere near that and don’t know it.

The bottom-right cell is the encouraging one. If your page converts at 10% and you’re making a genuinely large change, 683 per variant is achievable. Big swings on healthy pages are cheap to measure. Small tweaks on low-converting pages are not.

And the columns aren’t linear. Moving from a 50% MDE to a 10% MDE at a 5% baseline takes you from 1,468 to 31,231, a factor of about 21. That’s the quadratic term doing its work.

Translating sample size into calendar time

Traffic per month is what turns those numbers into a decision. Using the 5% baseline row and a 20% MDE (16,310 visitors total):

Monthly traffic to the page	Time to complete one test
500	32.6 months
2,000	8.2 months
10,000	1.6 months

And the same exercise at a 5% baseline with a 10% MDE (62,462 total) stretches to 125 months at 500 visitors a month, and still takes over six months at 10,000.

Run this for your own page before you design a test. If the answer is longer than your planning horizon, the test isn’t a test, it’s a way of feeling rigorous while learning nothing.

Why underpowered tests produce confident, false winners

This is the part that does real damage, because an underpowered test doesn’t fail loudly. It produces a winner.

Here’s the mechanism, and you can verify it with one line of arithmetic. Suppose you have 1,000 visitors per variant and a 3% baseline. The standard error of the difference between two proportions is:

SE = √(2 × p(1−p) ÷ n) = √(2 × 0.03 × 0.97 ÷ 1000) = 0.00763, or 0.76 percentage points.

To cross the 95% significance threshold, the observed difference must be at least 1.96 × 0.76 = 1.50 percentage points. From a 3% baseline, that’s a 49.8% relative lift.

Read that again, because it’s the whole problem. With 1,000 visitors per variant, the only results your test is capable of calling significant are ones showing a roughly 50% improvement or better. Real copy changes rarely do that. So when your underpowered test does light up green, the honest interpretation isn’t “we found a 50% winner”. It’s “noise happened to produce a 50% swing, which is exactly what noise does at this sample size”.

This is why the winners from small tests systematically overstate their effect, and why so many of them evaporate when you ship them. Conditional on winning, the measured lift must have been large; the underlying truth usually wasn’t. Add the common practice of checking results daily and stopping when significance appears, and the real false-positive rate climbs well above the nominal 5%. There’s more on that failure mode in Your A/B Test ‘Winner’ Might Be a Coin Flip.

An inconclusive result from an underpowered test is disappointing. A significant one is worse, because you’ll act on it.

What to do when the maths says “six months”

The reflexive response is to lower the bar: drop power to 60%, or use a one-tailed test, or call it at 90% confidence. All of these reduce the required sample by making your conclusions weaker in ways that won’t show up in the report. Occasionally justified. Usually self-deception.

The better response is to stop testing variants and start diagnosing friction. These are different activities that get confused because both are called “optimisation”.

	A/B testing	Friction diagnosis
Question answered	Which of these two wins?	Where and why do readers stall?
Traffic required	Thousands per variant	None
Output	A number and a direction	A ranked list of specific problems
Best used	Refining a page you understand	Before launch, or on a page that isn’t working
Fails when	Sample is too small	You need a precise effect size

Concretely, when the sample size maths rules testing out:

1. Find the friction first. A page losing people because the headline doesn’t explain the category has a problem no button test will surface. Diagnosis generates hypotheses; testing only referees between them.
2. Make the big change, not the small one. If you can only ever detect large effects, only attempt large effects. Rewriting the value proposition might be measurable at your traffic. Changing the button copy will not be.
3. Use non-statistical evidence deliberately. Qualitative reads, session recordings, support questions and sales objections are all valid inputs. They don’t produce p-values and they don’t need to.
4. Sequence it properly. Diagnose, fix the obvious leaks, grow traffic, then test the genuine trade-offs you couldn’t reason your way through.

Getting the diagnosis step done without traffic is what Buyer Clone is built for. You paste a URL and a panel of AI buyer-persona agents reads the page as different customer types would, reporting where each one stalls, doubts, bounces or converts, plus a ranked conversion brief. Under ten minutes, no snippet, no visitors required. There’s a full sample report at /sample, and the free tier costs nothing if you want to see your own page through it. If you’re pre-launch specifically, the fuller playbook is in How to Test a Landing Page When You Have No Traffic Yet.

What this calculation doesn’t cover

The formula above is the standard fixed-horizon, two-proportion, two-sided case, and it assumes several things that may not hold for you.

It assumes independent visitors, a stable traffic mix over the test window, one comparison rather than many, and a sample size fixed in advance. If you’re running four variants, you need a multiple-comparisons correction and more traffic still. If you’re using sequential testing or a bandit algorithm, the stopping rules and the maths are different. Strong seasonality, a big PR spike, or a change in ad targeting mid-test will all break the assumption that your two arms are drawn from the same population. And revenue-per-visitor tests need a continuous-outcome calculation rather than this one.

Our own limits are worth stating too. Buyer Clone doesn’t measure conversion rates and can’t substitute for a properly powered experiment. It’s strong on structural friction: unclear copy, missing proof, weak CTAs, pricing opacity. It’s directional only on emotional nuance and on exact pricing sensitivity, and once you have real traffic it complements user research rather than replacing it. Use it to decide what’s worth testing, then use a real test to settle it.

Frequently asked questions

How many visitors do I need per variant for an A/B test?

It depends on your baseline conversion rate and the size of effect you want to detect. At 95% significance and 80% power: about 8,155 per variant to detect a 20% relative lift from a 5% baseline, about 3,838 from a 10% baseline, and about 80,679 to detect a 10% relative lift from a 2% baseline. Compute your own with the formula above rather than using a rule of thumb.

Is 1,000 visitors enough to A/B test?

Only for very large effects on well-converting pages. At a 3% baseline with 1,000 visitors per variant, the smallest difference your test can call significant is around a 50% relative lift, so anything smaller is invisible and anything that does show up is probably noise. Treat results at that sample size as directional at best.

What happens if I stop an A/B test early?

Stopping when you first see significance inflates your false-positive rate well above the 5% you think you’re running at, because you’re taking many looks at the data and acting on whichever one crosses the line. Fix the sample size before the test starts and check the result once. If you genuinely need to peek, use a sequential testing method designed for it.

Can I reduce the traffic I need by lowering statistical power?

Yes, and you’ll usually regret it. Dropping power from 80% to 60% cuts the required sample meaningfully, but it means you’ll miss a real improvement four times in ten, which quietly turns genuine wins into “no difference detected”. Reducing your MDE ambition, or testing on a higher-converting page, are better levers.

What should I do if I don’t have enough traffic to test at all?

Diagnose friction instead of comparing variants. Comprehension and trust problems show up in a handful of readers and don’t require statistical power, so a structured cold read, a few real humans, or a synthetic buyer panel will all tell you more than an underpowered experiment. Come back to testing once your traffic clears the threshold your own calculation gives you.