How do you run a beta program that actually teaches you something?

A beta program is not a marketing event — it is a structured way to recruit your first real users, watch them struggle with your product, and iterate fast enough to find product-market fit before you run out of runway. Done right, it compresses months of guesswork into weeks of evidence. Done wrong, it produces a waitlist of strangers who never activate and a false sense of traction.

Recruit beta users manually — every single one

The instinct is to set up a landing page, run some ads, and let signups roll in. Resist it. At the beta stage you need people you can actually call, interrogate, and drag back to your product when they ghost you. Paul Graham's observation about Stripe is instructive here: even a product solving an urgent, obvious problem required founders to physically install it for early users rather than waiting for organic adoption. If Stripe needed that, your product does too.

Manual recruitment means going to where your target users already spend time — Slack communities, LinkedIn, industry forums, niche subreddits, in-person meetups — and having a real conversation before you ask for anything. You are not looking for people who say 'that sounds cool.' You are looking for people who describe the exact problem your product solves, unprompted, in their own words. Those are the beta users worth having. Fifty users recruited this way will teach you more than five hundred who filled out a form.

Keep a simple spreadsheet: who they are, how you reached them, what problem they told you they have, and what they agreed to do (use the product for X weeks, give feedback calls, etc.). Beta programs fail most often not from lack of users but from lack of accountability on both sides. A personal recruitment conversation creates a social contract that a signup form never does.

Design for learning, not for launch

Most founders structure their beta as a pre-launch runway — a holding pattern before the 'real' release. This is backwards. The beta is where you find out whether you have built the right thing at all. Every decision about how you run it should optimize for the speed and quality of learning, not for optics.

That means giving beta users access to incomplete, rough product rather than waiting until it is polished. A finished-feeling beta teaches you almost nothing, because users adapt their behavior to what seems 'done' and stop telling you what is actually broken. Roughness signals that their feedback matters and that things can still change. It also dramatically compresses your iteration cycle — you can make five changes in a week instead of one.

Define in advance what you are trying to learn from this cohort. Not 'do people like it?' but specific, falsifiable questions: Do users complete the core workflow without help? Do they return after day three? What is the first thing that makes them abandon a session? When you have real questions, you know what to watch for. Without them, you collect data that feels interesting but does not change any decision.

Plan for at least two rounds of beta cohorts, not one. Your first cohort will expose structural problems with the product. Your second cohort tests whether you actually fixed them. Founders who treat the beta as a single event usually graduate to a public launch with the same core problems they had at the start, just better-disguised.

Do the support and onboarding yourself

Founders routinely try to automate or delegate beta onboarding because it does not scale. This is exactly wrong. Paul Graham's point about doing things that do not scale applies directly here: the unscalable manual work of the beta phase is not a cost to be minimized, it is the mechanism by which you learn what you need to build.

When you personally onboard every beta user — walking them through the product on a video call, watching their screen as they click through it for the first time, answering their confused questions in real time — you see the product the way they see it. You notice the moment they hesitate before clicking a button. You hear the question they ask that reveals an assumption you never knew they had. You cannot get this information from analytics. You cannot get it from a survey. You can only get it by being present.

Handle all beta support yourself, or split it between the two founders, for as long as you possibly can. Every support ticket is a signal. When you start seeing the same question from multiple users, that is a product decision masquerading as a support problem. Fix the product, not the FAQ. The goal is to eventually make yourself unnecessary for onboarding — but you cannot design that system until you have done it manually enough times to understand what it actually involves.

Set a weekly cadence: a short check-in message to every active beta user, plus a 20-minute call with at least two or three of them. Do not make these calls optional or low-priority. They are the most valuable hours you will spend that week.

Measure activation and retention, not signups

The number that feels good to report — total beta signups — is almost always the least useful number to track. What matters is whether users are doing the core action your product is built around, and whether they come back to do it again.

Define your activation event before the beta starts: the specific action a user must take to count as 'activated.' For a project management tool it might be creating and assigning a task. For a data product it might be running a query and exporting the result. This should be a single, concrete, observable event — not a fuzzy sense that someone 'got value.' Once you have that definition, track what percentage of beta users reach it within their first session, and within their first week.

Retention tells you whether you have built something people actually want to keep using. Paul Graham's framework for thinking about startup growth makes clear that small differences in weekly retention rates compound into enormous differences in outcomes over months. A beta that shows even modest week-over-week retention among a small cohort is a far stronger signal than a large waitlist with no engagement data.

Build a simple dashboard — even a manually updated spreadsheet — that shows, for each beta user: date joined, date of first activation event, whether they were active last week, and their current streak. Look at this every Monday. The patterns will tell you who is getting value (learn from them), who dropped off (call them immediately and find out why), and what is actually broken in your funnel.

Graduate the beta before it becomes a crutch

A beta program that never ends stops being a learning mechanism and becomes a way to avoid committing to a real product. Set a fixed end date when you start — six to ten weeks is usually enough for two iteration cycles — and stick to it. The deadline forces decisions that open-ended beta programs defer indefinitely.

In the final two weeks, do a structured exit interview with every beta user who activated. These conversations have a specific goal: to understand the gap between what they hoped for and what they got, and to find out whether they would pay for this product if it solved that gap. Listen for the moment a user describes a problem your product does not yet solve, then describes what they are currently doing instead. That workaround is your roadmap.

Use the beta graduation moment to convert your best users to paying customers. The transition is easier than founders expect, because users who have been using a product for six weeks and talking to you regularly have already decided whether it is worth paying for. Asking for payment is not an imposition — it is a clarification of a relationship they have already formed. Offer beta users a discounted first year to reward their participation, but charge real money. Free forever arrangements train users to devalue the product and give you no signal about actual willingness to pay.

Finally, write down everything you learned before you move on. Not a polished document — a raw dump of what surprised you, what you were wrong about, and what you would do differently. Most founding teams skip this step and lose the institutional memory of the beta within a month. That memory is what keeps you from repeating the same experiments in your next growth phase.

“Nearly all startups have to recruit users manually. You can't wait for users to come to you.”

— Paul Graham, source

The one thing to do

Before anything else, personally recruit ten beta users this week by finding people who already have the problem you solve and asking them directly — not through a form.

Frequently asked questions

How many beta users do you actually need?

Between 20 and 100 is the practical range for most B2B or prosumer products. Fewer than 20 and you cannot see patterns; more than 100 and you cannot maintain the personal contact that makes beta feedback honest. Quality of recruitment matters far more than quantity.

Should beta users pay?

Ideally yes, even a nominal amount. Payment signals genuine intent and filters out people who signed up out of curiosity rather than need. If charging is not possible yet, require a meaningful commitment — a scheduled onboarding call, a promise to submit weekly feedback — to create the same filtering effect.

What is the single biggest mistake founders make in beta programs?

Treating the beta as a pre-launch marketing activity rather than a learning exercise. This leads to optimizing for signup numbers, delaying feedback collection until the product is 'ready,' and graduating to launch without actually answering the questions the beta was supposed to answer.

How do you handle beta users who go completely silent?

Call them within 48 hours of their last activity. Silent dropout is the most valuable signal in your beta — it usually means they hit a wall they did not bother reporting. A five-minute conversation with a churned beta user is often worth more than a dozen responses from engaged ones.

Sources

More playbook answers · Growth Prophet home