How long should you run an A/B test before you can trust the result?

Run your A/B test until you reach statistical significance (typically 95% confidence) and have covered at least one full weekly business cycle — almost always a minimum of two weeks, often four. Stopping early because the numbers look good is the single most common mistake founders make, and it produces decisions that hurt more than no test at all. The right duration depends on your traffic volume, effect size, and how much of your user behavior is day-of-week dependent.

Why 'it looks significant' is not a stopping rule

Most founders stop a test the moment the dashboard shows a green checkmark or a p-value below 0.05. This is called peeking, and it inflates your false positive rate dramatically. If you check a test 10 times during its run and stop when it first looks good, you're not running a 95% confidence test — you're running something closer to a coin flip dressed up in statistics.

The fix is to calculate your required sample size before you start, using a power calculator (there are free ones from Evan Miller and others), and commit to running until you hit that number. Write the end date or sample size target down before you launch the test. If you don't do this, the temptation to stop at the first spike will win every time.

This matters more for startups than for large companies because small sample sizes are noisier. A 30% lift on 200 users is nearly meaningless; that same lift on 2,000 users starts to mean something. If your traffic is too thin to reach significance in four weeks, you probably should not be A/B testing that feature at all — run the change for everyone and measure before-and-after instead.

The minimum viable test duration in practice

Two weeks is the practical floor for most consumer products, and four weeks is safer for anything with a weekly usage pattern. The reason is simple: user behavior on Monday looks nothing like behavior on Saturday, and if your test runs Tuesday through Thursday, you're measuring a slice of your audience, not your audience.

Beyond day-of-week effects, you also need to account for novelty bias. Users who see something new often engage with it more simply because it's different — not because it's better. This effect typically decays within the first week. If you stop your test before it decays, you'll ship changes that produce a short-term bump and a medium-term regression.

For B2B SaaS with monthly billing cycles, the calculus changes again. A pricing page test might need to run through a full billing period to capture real conversion decisions, not just clicks on a 'Start Trial' button. Map your test duration to the actual decision cycle of your user, not to how quickly your p-value drops.

Traffic volume and effect size: the tradeoff you have to make explicit

The three variables that determine test duration are: your baseline conversion rate, the minimum effect size you care about, and your daily traffic. If you have 10,000 daily visitors and you're testing a change you expect to move conversion by 20%, you might reach significance in a few days. If you have 500 daily visitors and you're looking for a 5% lift, you might need six months — which means the test is not worth running.

Be honest about what effect size actually matters to your business. A 2% lift on checkout conversion sounds small, but if your monthly revenue is $500,000, that's $10,000 per month. A 2% lift on a signup button that feeds a low-conversion funnel might be noise. Define the minimum detectable effect that would actually change a business decision, then calculate whether you can reach it in a reasonable timeframe. If you can't, make the decision another way.

One practical heuristic: if your power calculation says you need more than eight weeks to detect the effect you care about, stop and either find a bigger lever to test, increase traffic to that page, or ship the change and run a pre-post analysis. Eight weeks of split traffic is a long time to delay learning.

When to stop a test early — the only legitimate reasons

There are exactly two legitimate reasons to stop a test before your planned end date. The first is that you're seeing clear harm: the variant is producing a statistically significant negative result on a metric that matters (not just a vanity metric), and continuing the test would cost you real money or damage user trust. In this case, stop and revert.

The second is a true external event that breaks the validity of the test — a major product change deployed mid-test, a press spike that floods one cohort with atypical traffic, or a platform outage that affected only one variant. These contaminate the results, and continuing to run produces garbage data. Stop, document what happened, and restart clean.

Notice that 'the variant looks like it's winning' is not on this list. Neither is 'we need to ship something this sprint.' Shipping a result you don't trust because of schedule pressure is a fast way to build a product on top of false assumptions, and those compound badly over time. Garry Tan's engineering philosophy — captured in his gstack framework — centers on the idea that you should be direct about quality and fix the whole thing, not just the demo path. The same applies to experimentation: a test result that only looks good on the surface is a bug, not a feature.

Building a testing discipline that doesn't require heroics

The founders who get the most out of A/B testing treat it as infrastructure, not an event. That means having a pre-written test plan template, a shared log of running tests with their planned end dates, and a standing rule that no one calls a test early without a written justification reviewed by at least one other person.

For early-stage startups with thin traffic, consider running fewer tests with higher-confidence thresholds rather than a constant stream of underpowered experiments. Three well-run tests per quarter that you can actually trust beats twelve tests per quarter that generate noise. Paul Graham's observation about writing — that putting ideas into words forces precision you didn't know you lacked — applies directly here: writing down your hypothesis, your success metric, and your minimum effect size before you launch a test forces a rigor that prevents most of the common mistakes.

Finally, keep a test log that includes not just the winner but the actual numbers, the sample sizes, and the dates. Six months from now you'll want to know whether a result held up, whether you ever retested it, and what the confidence interval was. Most startup teams lose this institutional memory and end up retesting things they already know, or worse, re-litigating decisions that were already settled with data.

“A great deal of knowledge is unconscious, and experts have if anything a higher proportion of unconscious knowledge than beginners.”

— Paul Graham, source

The one thing to do

Set your sample size target before you launch the test, write the end date down, and don't let anyone call the result early — that single discipline eliminates most A/B testing mistakes founders make.

Frequently asked questions

Can I stop my A/B test as soon as it hits 95% statistical significance?

Not reliably. Hitting 95% confidence before your planned sample size is often a false positive caused by peeking. Commit to your sample size target upfront and don't stop early unless you're seeing clear harm.

What if I don't have enough traffic to run a proper A/B test?

Run the change for 100% of users and compare a clean before-and-after window, being careful to account for seasonality. Underpowered A/B tests produce worse decisions than honest pre-post analysis because they create false confidence.

How do I calculate how long my test needs to run?

Use a free sample size calculator — Evan Miller's is the most widely used. Input your baseline conversion rate, your minimum detectable effect, and your desired significance level. Divide the required sample size by your daily traffic to get the number of days.

Does it matter which days of the week my test runs?

Yes, significantly. Always run tests across at least one complete week so that weekday and weekend behavior are both represented. For products with strong weekly cycles, two full weeks is the minimum to avoid day-of-week bias skewing your results.

Sources

More playbook answers · Growth Prophet home