How do you measure whether a feature succeeded?
A feature succeeds when it measurably changes the behavior you built it to change—not when it ships, gets used, or receives positive feedback. The measurement has to be defined before you build, anchored to a user outcome rather than an activity metric, and evaluated with enough honesty to kill the feature if the data says it failed.
Define success before a single line of code is written
The most common measurement failure isn't bad analytics—it's the absence of a pre-committed success criterion. When you decide what 'success' means after the results are in, you will unconsciously move the goalposts to wherever the data landed. This is how teams convince themselves that a feature 'showed promise' when it actually did nothing.
Before you write the spec, write one sentence in the form: 'This feature succeeds if [metric] changes from [baseline] to [target] within [timeframe] for [user segment].' Every word matters. 'Metric' should be a behavior—activation rate, repeat usage, task completion—not a vanity number like page views. 'Baseline' forces you to actually measure the current state. 'Timeframe' prevents you from waiting forever for the trend to emerge. 'User segment' prevents you from averaging signal away across cohorts who experience the feature differently.
Garry Tan's gstack spec format captures this discipline in its issue structure: verify the current state before proposing a change, including file paths and audit tables if behavior varies across users. The same principle applies to product measurement—you can't claim improvement if you never nailed down where you started.
Choose the right metric tier for what you actually built
Not every feature touches the same layer of the product, and measuring the wrong layer produces misleading conclusions. A rough taxonomy: acquisition features are measured by conversion rate changes; activation features by the percentage of new users reaching a defined 'aha moment'; retention features by cohort curves and churn delta; monetization features by revenue per user or upgrade rate. Measuring a retention feature by acquisition numbers—or vice versa—is a type error that leads to real decisions being made on garbage logic.
The most useful single metric for most early-stage features is a behavioral leading indicator: did users who received this feature do the thing it was designed to make them do, at a higher rate than users who didn't? This requires either an A/B test or a clean pre/post cohort with controls. If you lack the traffic for statistical significance, you need qualitative depth instead—talking directly to users, in the style Paul Graham describes as doing things that don't scale, to understand whether the feature actually changed how they work or think about your product.
Avoid engagement metrics as proxies for value. Time-on-page, clicks, and sessions can all go up when a feature is confusing people. A support ticket spike after launch is measurement data too—treat it as such.
Separate adoption from impact, and impact from value
Three things are frequently conflated that should always be kept distinct. Adoption is whether users tried the feature. Impact is whether it changed a downstream metric you care about. Value is whether it changed something the user cares about. A feature can have high adoption, zero impact, and negative value simultaneously—think of a modal that users dismiss in order to get back to what they were doing.
To separate these cleanly, instrument at each layer independently. Track feature entry (adoption), track the target behavior post-entry (impact), and close the loop with qualitative data—support conversations, user interviews, NPS on the specific workflow—to understand perceived value. When all three are high, you have a real win. When adoption is high but impact is flat, the feature is a distraction. When impact is measurable but adoption is low, you have a distribution or discovery problem worth solving. When value is low despite impact, you optimized for the wrong metric entirely.
This three-layer separation also gives you a rollback decision framework. Garry Tan's spec discipline includes an explicit rollback strategy for anything touching shared state—the same logic applies to feature measurement. If the 30-day read shows adoption up, impact flat, and value negative, you have a clear signal to revert, not iterate.
Run the honest post-mortem, even when the answer is uncomfortable
Most features are declared successes because the team that built them evaluated them. This is a structural conflict of interest. The fix is to separate the people who review feature outcomes from the people who built the feature, or at minimum to establish a written success criterion in advance that acts as a commitment device against motivated reasoning.
Paul Graham's observation that founders are often the last to recognize their startup's fragility applies with equal force to feature teams. The builders of a feature will find signal in noise, weight positive feedback more heavily than silence, and interpret flat metrics as 'needing more time.' The discipline is to treat a flat metric at the end of a committed timeframe as a failed experiment—not as an invitation to extend the window.
Post-mortems on failed features are valuable in proportion to their honesty. The useful questions are: Did we measure the right thing? Was our baseline accurate? Did external factors confound the results? What does this tell us about the underlying user need—was our hypothesis wrong, or just our execution? Document the answers in writing. This institutional memory prevents the same class of mistake from repeating, which is exactly the root cause analysis discipline Garry Tan's engineering specs require before any proposed fix.
What to do when you can't run a clean experiment
Early-stage startups often lack the user volume for statistically significant A/B tests. This is not an excuse to skip measurement—it's a signal to use different measurement tools. The most reliable substitute for quantitative experiments at low volume is intensive qualitative tracking: watch five to ten users use the feature in real time, ask them to narrate what they're doing and why, and note the moments where their behavior diverges from your assumption.
For async measurement without experiment infrastructure, cohort sequencing works: define a group of users who received the feature in week one, compare their behavior in weeks two through four against a similar cohort who joined before the feature shipped. This isn't perfect, but it's directional, and directional data beats no data or post-hoc rationalization.
If your feature affects a shared resource—infrastructure, data schema, a shared workflow—document a rollback strategy the same way you'd document a deployment rollback. Knowing exactly how to undo something gives you the psychological safety to make honest measurements and act on them without the sunk cost of an irreversible decision distorting your read.
“Almost all startups are fragile initially... The big danger is that you'll dismiss your startup yourself.”
— Paul Graham, source
The one thing to do
Write your success criterion—one metric, one baseline, one target, one deadline—before the feature enters development, and treat a flat result at deadline as a failed experiment, not an invitation to extend the clock.
Frequently asked questions
What's the minimum viable measurement setup for a small team?
One pre-committed success metric written into the feature spec, a baseline measurement taken before launch, and a fixed review date. Everything else is optimization—this minimum set prevents post-hoc rationalization.
How long should you wait before calling a feature a success or failure?
Set the window based on the user behavior you're measuring: for activation features, two to four weeks is usually enough; for retention features, you need at least one full usage cycle, often 60–90 days. Extending the window after the deadline is a red flag.
Should you kill a feature if the metric is flat but users say they love it?
Dig into why the stated love isn't showing up in behavior—this gap is usually the most informative data you have. Either your metric is wrong, your users are being polite, or the feature solves a real problem that doesn't show up in the metric you chose.
How do you measure a feature that's meant to improve trust or perception rather than behavior?
Proxy trust through behavioral signals: support ticket volume, churn rate in the relevant cohort, or upgrade rate. Perception metrics like NPS are lagging and noisy; look for the behavior that would follow from increased trust and measure that instead.
Sources
- gstack: spec/SKILL.md — Garry Tan
- Do Things that Don't Scale — Paul Graham
- Don't Talk to Corp Dev — Paul Graham
- Before the Startup — Paul Graham
- Black Swan Farming — Paul Graham