How do you avoid false positives in A/B testing?

False positives in A/B testing happen when you declare a winner that isn't real — typically because you stopped too early, ran too many variants, or set your significance threshold too loosely. The fix requires discipline before the experiment starts, not after you see the numbers. Build your stopping rules, sample size, and success metrics into the test design upfront, and you'll avoid most of the mistakes that produce convincing-looking but worthless results.

Why false positives are so common — and so costly

The core problem is that founders and growth teams are incentivized to find wins. When you're under pressure to show progress, a p-value of 0.049 feels like a green light. But statistical significance at the 95% confidence level still means you'll see a false positive one in twenty times just by chance. If you're running dozens of tests per quarter — as most growth-focused teams do — some of those 'wins' are noise by definition.

The cost isn't just wasted engineering time shipping a change that does nothing. It's worse than that: you may ship something that subtly degrades the experience, mistake random variation for a causal insight, and build a distorted mental model of what your users actually want. False positives compound. Each bad 'learning' becomes a prior that shapes the next test design, and over time your experimentation program starts optimizing for a fiction.

The teams that run trustworthy experiments treat statistical rigor as a product quality bar, not a bureaucratic checkbox. The same instinct that makes you want to fix a bug before it reaches production should make you want to fix your test design before it corrupts your data.

Fix the design before you run a single visitor through it

The single most effective lever against false positives is pre-registration: decide your primary metric, your minimum detectable effect, your required sample size, and your stopping date before you launch. Write these down somewhere your team can see them. Once the test is live, those parameters are locked. This sounds obvious but almost no one does it consistently under shipping pressure.

Sample size calculation is non-negotiable. Use a power calculator with your actual baseline conversion rate, a realistic minimum effect size you'd actually care about (not the biggest lift you're hoping for), 80% power, and a 95% confidence threshold. The resulting number is almost always larger than founders expect — often 10x to 50x what they guessed. This is uncomfortable because it means tests take longer. But running underpowered tests is how you manufacture false positives at scale.

One practical forcing function: create a simple one-page test brief that every experiment requires before engineering touches a line of code. The brief should include the hypothesis, the single primary metric, how the sample size was calculated, the exact end date, and what you'll do with each possible outcome. Teams that skip this brief are the teams that peek at results daily and stop tests the moment they see something they like.

Peeking, stopping early, and the multiple comparisons trap

Peeking at results before your predetermined sample size is reached is the most common source of false positives in practice. The reason it's so dangerous is that p-values fluctuate dramatically during a test. A variant that looks like a 40% lift after 200 sessions might be indistinguishable from the control at 2,000 sessions. Every time you check results and consider stopping, you're effectively running a new hypothesis test — and each additional look inflates your true false positive rate well beyond the 5% you think you're accepting.

If your team genuinely needs to monitor tests in real time — for example, to catch a broken experience that's tanking conversions — use sequential testing methods like the Sequential Probability Ratio Test (SPRT) or Bayesian updating rather than standard frequentist significance. These methods are designed for interim analysis and maintain your Type I error rate even with multiple looks. Standard t-tests and chi-square tests are not.

Multiple comparisons compound this problem. Testing five variants against a control, tracking seven different metrics, and segmenting results by device and geography isn't one test — it's potentially hundreds of tests, most of which you'll only report if they're significant. This is sometimes called 'p-hacking' but it often happens accidentally, not maliciously. The defense is to designate exactly one primary metric before the test starts and treat everything else as exploratory, clearly labeled as such in your reporting.

Novelty effects and the minimum runtime rule

Even a well-powered, properly terminated test can give you false positives if users are simply reacting to novelty rather than genuine preference. A new button color or layout change often gets higher engagement in week one just because it's different. Users notice the change and interact with it out of curiosity. If your test runs for three days, you may be measuring novelty, not value.

A minimum runtime of one to two full business cycles — typically two weeks — is a practical floor for most consumer and B2B SaaS products. This ensures you capture weekday and weekend behavior, one or two full user habit loops, and enough time for the novelty effect to decay. For products with longer usage cycles (annual billing, multi-session onboarding), the minimum runtime needs to extend accordingly.

A related issue is the new-user bias problem. If you're testing on all traffic including returning users, the proportion of new versus returning users in each variant can drift over time and between buckets, especially if you launch mid-week. Segment your analysis by user type or, better, restrict experiments to new users only when you're testing acquisition or first-run experience. Mixing cohorts with very different baseline behaviors inflates variance and produces unreliable estimates.

Build a culture that respects null results

Organizational incentives are the root cause that statistical techniques can't fix on their own. If the implicit expectation is that every A/B test should find a win, teams will find wins — by stopping early, by selectively reporting metrics, by running tests on segments until something turns significant. This isn't always conscious fraud; it's the predictable output of a culture that doesn't value learning from null results.

The reframe that actually works is treating a well-run null result as equivalent in value to a positive result. A null result from a properly powered test tells you something real: the change you thought mattered doesn't matter, at least not for this metric on this audience. That's genuinely useful information that should update your product strategy. Teams that document and share null results build better intuition over time than teams that only report winners.

Practically, this means including null results in your experiment retrospectives, tracking your win rate as a diagnostic (a team with a 90% win rate is almost certainly running badly designed tests or reporting selectively — healthy win rates are usually 20–40%), and rewarding the quality of experiment design rather than just the outcome. The experiments that teach you the most are often the ones where the results surprised you.

The one thing to do

Write down your primary metric, sample size, and end date before launching any test — and make those parameters non-negotiable once the experiment is live.

Frequently asked questions

What significance threshold should I use for A/B tests?

95% confidence (p < 0.05) is the standard, but consider raising it to 99% for high-stakes changes like pricing or core onboarding flows. For exploratory tests with low implementation cost, 90% is defensible as long as you treat those results as hypotheses to be confirmed rather than final decisions.

Can I run multiple metrics in a single A/B test?

Yes, but designate exactly one primary metric before launch and treat everything else as secondary and exploratory. Apply a Bonferroni correction or similar adjustment if you plan to make decisions based on multiple metrics. Never choose your primary metric after seeing the data.

How do I handle tests where I need results faster than a proper sample size allows?

If you genuinely can't wait for a properly powered test, be honest about what you're doing: call it a directional signal, not a validated result. Run the test for the time you have, note the confidence interval, and treat it as a prior for a properly powered follow-up test rather than a shipping decision.

Is Bayesian A/B testing better than frequentist for avoiding false positives?

Bayesian methods handle repeated peeking more gracefully and give you probability estimates that are easier to reason about. But they don't eliminate false positives — they just express uncertainty differently. You still need to set a meaningful prior, choose a decision threshold, and run the test long enough for the posterior to stabilize.

Sources

More playbook answers · Growth Prophet home