How do you A/B test when you don't have enough traffic?
With low traffic, classical A/B testing is a trap — you'll either wait months for statistical significance or make decisions on noise. The honest answer is that you need to shift from frequentist split-testing to faster, more deliberate methods: sequential testing, qualitative validation, and staged rollouts that let you act on early signals without fooling yourself.
Why standard A/B tests fail below ~1,000 conversions/week
A standard A/B test needs a minimum detectable effect size before you run it. If your baseline conversion rate is 3% and you want to detect a 20% relative lift (to 3.6%), you need roughly 15,000 visitors per variant to reach 80% power at p<0.05. At 500 visitors a week, that's 60 weeks — by which time your product has changed three times and the test is meaningless.
The deeper problem is peeking. Most founders run a test for two weeks, see a 15% lift, and ship it. But stopping early when results look good inflates your false positive rate dramatically. You're not running a test; you're doing selective observation with extra steps. The result is a roadmap full of 'validated' changes that didn't actually move the needle.
Before reaching for any testing framework, get honest about your sample size. Use a power calculator (Evan Miller's is free and accurate) with your real baseline rate, realistic effect size, and the traffic you actually have. If the runtime exceeds six weeks, a classic A/B test is the wrong tool for your stage.
Sequential and Bayesian testing: act earlier without lying to yourself
Sequential testing methods — particularly Bayesian A/B testing — are designed for exactly the low-traffic situation most early-stage products face. Instead of setting a fixed sample size in advance and waiting, Bayesian approaches let you update your beliefs continuously as data arrives and make a decision when you have enough confidence for the stakes involved.
The practical output is a probability: 'Variant B has an 87% chance of being better than control.' You choose a threshold appropriate to the decision — maybe 90% for a major pricing change, 75% for a copy tweak you can easily reverse. Tools like VWO's Bayesian mode, GrowthBook (open source), or even a simple Beta distribution calculation in a spreadsheet give you this without a stats PhD.
The key discipline is pre-committing to your threshold and your stopping rule before you start. Write it down: 'We will ship this variant if it reaches 85% probability of being better, or we will call it inconclusive and move on after 4 weeks.' This prevents the same peeking problem that plagues frequentist tests. The method only works if you follow the rule even when results are disappointing.
For very small samples — under a few hundred conversions — even Bayesian tests will give you wide credible intervals. Treat them as directional signals, not verdicts. Combine the quantitative signal with qualitative data (user interviews, session recordings) before making irreversible decisions.
Fake door tests and qualitative methods that don't require traffic volume
The most underused low-traffic validation technique is the fake door test. Instead of building two versions of a feature and measuring downstream conversion, you show users a button or option that doesn't exist yet. When they click it, they see a 'coming soon' message or a brief survey. You measure click-through as a proxy for intent — often a much higher-signal event than a downstream conversion, and achievable with a fraction of the traffic.
Fake doors work best for new features and pricing tiers. Put a 'Teams Plan' button on your pricing page before you build teams. If 8% of your visitors click it, you have evidence of demand. If 0.3% do, you've saved months of engineering. The ethical version tells users immediately that the feature is coming and offers them early access — this also builds a waitlist you can use for qualitative follow-up.
For copy and messaging tests specifically, five-second tests (tools like Lyssna or UsabilityHub) and moderated user interviews give you directional answers with 5–10 participants. The standard in UX research is that five users will surface about 85% of major usability problems. This isn't statistical power in the A/B sense, but it's genuinely predictive for questions like 'does this headline communicate what we do?' where a conversion test would need thousands of visitors.
The honest framework: use fake doors and qualitative methods to decide what to build, use Bayesian tests to decide whether your implementation worked, and reserve classic A/B tests for high-traffic, high-stakes decisions where the runtime is reasonable.
Staged rollouts as a testing discipline
When you're shipping a change that affects all users, a staged rollout — releasing to 10% of users, then 25%, then 100% — gives you a form of causal inference that doesn't require waiting for statistical significance on a split test. You're not comparing A vs. B simultaneously; you're watching whether your metrics move as you increase exposure. This catches large effects reliably, even with low absolute traffic.
The requirement is a monitoring setup that lets you catch regressions fast. Instrument your core metrics — activation rate, conversion, retention day-7 — and set explicit rollback triggers before you start. 'If conversion drops more than 15% relative during the 10% rollout, we roll back within 24 hours.' The staged rollout is only a testing discipline if you have the operational rigor to actually roll back.
For product changes where you can measure user-level outcomes, holdout groups are more statistically clean than time-based comparisons. Keep 5-10% of users on the old experience permanently during a rollout and compare them to the exposed group. This controls for seasonal effects and other confounders that make before/after comparisons unreliable.
The broader mindset shift at low traffic is from 'test everything' to 'test the right things.' Prioritize tests where the effect size you're looking for is large (>20% relative lift), the decision is reversible if you're wrong, and the cost of a false positive is low. For bet-the-company changes — pricing restructures, fundamental UX overhauls — invest in qualitative certainty before you ship, not in underpowered quantitative tests that give you false confidence.
The one thing to do
Run a power calculation before starting any A/B test — if your runtime exceeds four weeks, switch to Bayesian testing, fake door tests, or staged rollouts instead.
Frequently asked questions
How much traffic do I need for an A/B test to be valid?
It depends on your baseline conversion rate and the effect size you want to detect. Run a power calculation before starting. A common starting point: if you can't get 500+ conversions per variant within four weeks, a classic A/B test will likely produce unreliable results and you should use Bayesian testing or qualitative methods instead.
What is a fake door test and when should I use it?
A fake door test shows users a button or feature option that doesn't yet exist, then measures how many click it as a proxy for demand. Use it when you want to validate whether users want a feature before investing engineering time to build it — you can get meaningful signal with as few as 200-300 visitors.
Can I use Bayesian A/B testing with only 200 visitors a week?
Yes, but interpret results as directional, not definitive. At very small sample sizes, your credible intervals will be wide, meaning high uncertainty. Bayesian testing is more honest about this uncertainty than frequentist p-values, but you should still combine it with qualitative research before making major irreversible decisions.
Is it ever okay to just ship without testing?
Yes — for reversible changes with low downside risk, shipping and monitoring is faster and more informative than an underpowered test. Reserve formal testing for decisions where being wrong is costly and hard to undo, like pricing changes or fundamental navigation restructures.
Sources
- gstack: skillify/SKILL.md — Garry Tan