What sample size do you need for an A/B test?

For most conversion-rate tests, you need roughly 1,000 conversions per variant — not visitors, conversions — before you can trust the result. The exact number depends on three variables you control: your baseline conversion rate, the minimum effect size worth detecting, and your acceptable risk of a false positive. Get any of these wrong and you'll ship a loser thinking it's a winner, or shelve a winner because it looked flat.

The three inputs that determine your sample size

Every sample-size calculator asks for the same three things, and founders routinely guess at all three. First, your baseline conversion rate: pull this from analytics for the exact page and action you're testing — not sitewide averages. Second, minimum detectable effect (MDE): how small a lift is still worth shipping? A change that improves checkout from 3.0% to 3.3% is a 10% relative lift. If your traffic is thin, you may need to set a higher MDE (say, 20%) just to get a result in a reasonable timeframe. Third, statistical significance threshold: the standard is 95% confidence (p < 0.05), meaning a 1-in-20 chance of a false positive. Some teams drop to 90% to move faster; few should go lower than that.

These three inputs feed a standard formula, but you don't need to do the math by hand. Tools like Evan Miller's sample size calculator or Optimizely's stats engine will give you a number in seconds. The output is a per-variant visitor count — you need that many visitors in each bucket, not combined. If the calculator says 4,000 and you have two variants, you need 8,000 total visitors before you read the result.

One trap founders fall into: confusing visits with conversions. The meaningful quantity is the number of times the conversion event fires, not how many people loaded the page. If your page converts at 2% and the calculator says you need 1,000 conversions per variant, you need 50,000 visitors per variant — 100,000 total. That's a very different planning problem.

Why peeking early destroys your results

The single most common mistake in startup A/B testing is checking results daily and stopping the test the moment one variant looks better. This is called peeking, and it inflates your false-positive rate dramatically. If you peek ten times during a test, the true false-positive rate can balloon from 5% to over 30% even though your dashboard still says 'p < 0.05.' You are not seeing statistical significance — you are seeing noise that happened to cross a threshold.

The fix is to decide your sample size before you start, then not read the result until you hit it. Write the target number in a shared doc. Lock the dashboard if you have to. This feels unnatural when you're used to moving fast, but it's the only way the probability math actually holds. Sequential testing methods (like those used by Optimizely Stats Engine or Statsig) can let you peek safely, because they adjust the significance threshold dynamically — but only if the platform was designed for it from the start.

For early-stage teams with low traffic, the implication is uncomfortable: you may not have enough volume to run valid A/B tests on most things. If it takes six months to accumulate enough conversions, you should be making the decision qualitatively — user interviews, session recordings, founder intuition — rather than waiting for a test that will almost certainly be underpowered.

Low-traffic startups: when to skip the test entirely

If your product is generating fewer than a few thousand conversion events per month, formal A/B testing is usually the wrong tool. The sample sizes required for statistically valid results will take so long to accumulate that the business will have changed around you before you get an answer. A homepage copy test that needs 10,000 conversions per variant at a 1% baseline rate means 2,000,000 visitor-sessions — that's years of data for most early-stage products.

The better approach at this stage is to do fewer, bigger changes. Instead of testing button colors, test entirely different value propositions. Instead of moving one element, redesign the whole page. Larger changes produce larger effects, which means the minimum detectable effect is bigger, which means your required sample size drops sharply. A change that moves conversion from 2% to 4% (a 100% relative lift) requires roughly one-eighth the sample size of a change that moves it from 2% to 2.4%.

For qualitative signal, five user interviews will surface the majority of major usability problems — a well-established finding in UX research practice. Combine that with heatmaps, scroll maps, and session recordings to identify friction, then make bold changes based on what you learn. Reserve A/B tests for the moment you have enough volume that results come back in two to four weeks, and for decisions where the directional signal from qualitative work is genuinely ambiguous.

Practical setup to avoid the most expensive mistakes

Before you start any test: write a one-paragraph hypothesis ('We believe changing X to Y will increase Z because of reason W'), calculate your required sample size, set a hard end date, and assign one person to be the decision-maker who will not read the dashboard until the end date. This takes fifteen minutes and prevents the most common failure modes.

Split traffic randomly and simultaneously — never test Monday versus Tuesday, or this week versus last week. Seasonality, day-of-week effects, and external events will confound sequential comparisons in ways that are nearly impossible to detect after the fact. Use your A/B platform's built-in randomization, and verify that the split is actually 50/50 by checking it after 100 sessions, not 10,000.

Finally, track secondary metrics alongside your primary one. A variant that improves sign-ups but kills activation is not a winner. A change that improves checkout completion but doubles support tickets is not a win. Define what 'winning' means across two or three metrics before the test starts, and stick to that definition. Changing the success criteria after you see the data is the statistical equivalent of moving the goalposts — it looks like rigor but produces random decisions.

The one thing to do

Calculate your required sample size before starting any test, write it down, and don't read the results until you hit it — everything else is noise.

Frequently asked questions

Is 100 conversions per variant enough to call a test significant?

Almost never. At 100 conversions per variant, your test only has power to detect very large effects (30–50% relative lifts). For typical conversion-rate improvements of 10–20%, you'll need 1,000 or more conversions per variant to reach 80% statistical power at 95% confidence.

What's the difference between statistical significance and practical significance?

Statistical significance tells you the result is unlikely to be random noise. Practical significance tells you whether the effect is large enough to matter to the business. A test with 500,000 visitors can detect a 0.1% lift as statistically significant, but that lift may not justify the engineering cost of shipping the change.

Can I stop a test early if the result looks obvious?

Only if you're using a platform built for sequential testing (like Statsig or Optimizely Stats Engine), which adjusts confidence thresholds to account for early stopping. With standard fixed-horizon tests, stopping early — even when results look decisive — inflates your false positive rate significantly.

How do I run A/B tests if I have less than 10,000 monthly visitors?

Focus on making large, bold changes rather than incremental tweaks — bigger effects require smaller sample sizes. Supplement with user interviews and session recordings to build directional confidence. Reserve formal A/B tests for decisions where you have enough volume to get results within four weeks.

Sources

More playbook answers · Growth Prophet home