Picture a Monday meeting. Signups are up 12% on last week, so the new onboarding screen is declared a success. Revenue is flat, but nobody mentions it. Two weeks later a test on the pricing page "wins" after three days, ships, and the effect is gone by the end of the month. Nothing in this story is exotic. Each step is a way of reading a number that felt like an answer and was not one.
Product teams measure because opinions are cheap and users are expensive to annoy. The craft has two halves. The first is choosing numbers that mean something: a few outputs you care about and the inputs that move them. The second is finding out whether a change you made caused a number to move, which is what a controlled experiment does. Companies that run thousands of online experiments a year, such as Microsoft, Google, Booking.com and Netflix, have published a lot about what goes wrong, and most of this piece comes from that literature.
- Who takes it
- Part 2 of the Product Management minor, a short route that sits beside many majors. A minor is not a degree of its own; it adds a skill to a degree. Pairs with: Computer Science, Data Science, Business Administration, Marketing. Product analytics, growth and online experimentation are taught in data science, business and computer science programmes, often as one elective on top of a statistics course, and are a core skill in product-management job descriptions.
- What it is for
- Choosing metrics that guide decisions, running a test you can trust, and knowing when a number is lying.
- Comes after
- Product Management: The Job (36/95) and Probability & Statistics (34/05).
- Leads to
- Launching & Growing Products (36/99), which uses retention and experiments to decide whether a launch worked.
- What this piece gives you
- You will not be a data scientist after this. You will be able to size a test, spot a peeked result, read a retention curve and fill in an experiment plan. Easier first reads: Experiments & Causal Inference (36/34), for why randomising works, and Marketing Research & Analytics (34/59). This piece applies them to a product.
Metric trees: one number on top, levers underneath
A good metric system starts with a few output metrics: results you care about, such as weekly paying customers or hours of music played. They are slow and they respond to many things at once, so you cannot "work on" them directly. Underneath sit input metrics: things a team can change this quarter that should move the output, such as the share of new users who finish setup. Drawing the arrows from inputs to the output gives a metric tree, and the best trees are arithmetic: the output is literally the product or sum of the branches.
Here is an invented learning app. Weekly paying learners equal visitors, times the share who sign up, times the share of those who reach a first lesson ("activate"), times the share of those who pay. Move the sliders and see which stage is leaking people and what the same change does at each stage.
Without JavaScript: 20,000 visits × 8% × 40% × 10% = 64 paying learners, and at $12 that is $768 a week. Adding 2 percentage points to each stage separately gives +16, +3.2 and +12.8 paying learners.
The table the tool prints is the lesson. A relative gain of 10% at any stage gives the same 10% more output, because the stages multiply. But two extra percentage points are worth very different amounts depending on where they land, because 2 points on a base of 8% is a quarter more signups and 2 points on a base of 40% is a twentieth more activations. Product teams pick the stage where the room to improve is largest relative to the cost, which is rarely the one that gets the most meeting time.
Funnels, activation and the pirate metrics
A funnel is the tree laid on its side: how many people reach each step, and how many drop out. Activation is the step where a new user first gets the value the product promised, often called the "aha" moment. Teams find it from data, by asking which early action best predicts that people come back, and then test whether pushing people to it helps. That last step matters, because the people who do an early action are different from those who do not, and a correlation in the funnel is a hint, not a proof (see 36/34).
Dave McClure, an investor, gave the funnel a name in 2007 in a talk titled "Startup Metrics for Pirates": AARRR, because pirates say "aarrr". It is a checklist of five questions, not a law of nature, and real users loop around it rather than marching through it in order.
| Stage | The question | An example metric |
|---|---|---|
| Acquisition | How do people find us? | Visits by channel |
| Activation | Do they get a good first experience? | Share finishing setup in 24 h |
| Retention | Do they come back? | Share active in week 4 |
| Referral | Do they tell others? | Invites sent per user |
| Revenue | Do they pay? | Paid rate, revenue per user |
Eric Ries, in The Lean Startup (2011), warned against vanity metrics, numbers that only go up and so flatter you, such as total registered users or all-time downloads. His alternative was cohort-based, actionable metrics, which is where we go next: the question is not "how many have ever signed up" but "of the people who signed up in March, how many are still here".
Retention: the curve that flattens
Group users by the week they started; each group is a cohort. A retention curve shows what share of a cohort is still active on day 1, day 7, day 30. Two shapes matter. In one the curve falls and then flattens: a core of people have made the product part of their routine. In the other it keeps decaying towards zero. A flattening curve means every new cohort adds a lasting base, so growth can build. A decaying curve is a leaky bucket: you must keep pouring in new users just to stand still.
Retention depends on how you define "active". Counting someone who opened the app is different from counting someone who did the core action (sent a message, played a song, filed an expense). Pick the action that means a user got value, and say which one you used. DAU/MAU, daily active users divided by monthly active users, is a quick stickiness ratio: 25% means the average monthly user shows up about 7.5 days out of 30.
Flattens at your level Same day 1, decays towards zero
Without JavaScript: with 40% back on day 1, flattening at 15% and settling at speed 0.25, about 21% remain on day 7 and 15% on day 30. The same cohort decaying to zero keeps under 1% by day 30. The curves are an invented model, not data. 10,000 DAU over 40,000 MAU is 25%.
The model is a teaching device: r(t) = floor + (day 1 − floor) × e−k(t−1). Real curves are noisier and have bumps at weekends and at renewal dates. What carries over is the question to ask of every product: does the line go flat, and at what height?
How many users an A/B test needs
In an A/B test, users are assigned at random to the current version (A, the control) or a changed one (B, the treatment), and you compare a metric. Random assignment is what lets you say the change caused the difference (36/34). Three choices come before launch. The randomisation unit: usually the user, so that one person does not see both versions; if you randomise by user but count page views as if they were independent, your error bars are too narrow. The minimum detectable effect (MDE): the smallest change worth detecting. And the error rates: a false-positive rate α, conventionally 5%, and power, the chance of catching a real effect of that size, conventionally 80%.
For a conversion rate, the users needed in each arm are about:
The square in the denominator is the whole story: halve the effect you want to detect and you need four times the users. Small effects on rare events are expensive, which is why a small product often cannot detect a 2% relative gain and should test bolder changes instead.
Without JavaScript: baseline 10%, a 10% relative gain (11%), 80% power and α = 5% need about 14,750 users per arm, 29,500 in all, which is 15 days at 2,000 users a day and so three full weeks. A 5% relative gain needs about four times as many.
Round the duration up to whole weeks, because behaviour differs on weekdays and weekends.
Peeking, mismatches and novelty: how a test lies
The most common way to ruin a test is to look at it every day and stop when the p-value dips under 0.05. A p-value of 5% promises a 5% false-positive rate if you look once, at the planned end. Each extra look is another chance for noise to cross the line, and the chances add up. This is peeking, or optional stopping, and Armitage, McPherson and Rowe described the arithmetic in 1969: with 5 equally spaced looks the false-positive rate rises to about 14%, with 10 looks about 19% and with 20 looks about 25%. The tool below checks this on an A/A test, where nothing differs at all, so any "winner" is false. It runs 2,000 simulated tests with a fixed seed, so you get the same answer each time.
Without JavaScript: in a simulation of 200,000 A/A tests at a nominal 5%, stopping at the first significant look gave 5.0% false positives with 1 look, 8.4% with 2, 14.4% with 5, 19.5% with 10 and 25.3% with 20, matching the published figures.
The fix is a rule, not willpower: decide the sample size in advance and read the result once, or use a method designed for repeated looks. Sequential testing methods (group-sequential designs with spending functions, or "always-valid" p-values) spend the 5% budget across the looks so you may stop early and stay honest. If you must watch a dashboard, treat it as a health check, not a verdict.
Two more traps have a check you can run. A sample ratio mismatch (SRM) means the split you got is not the split you designed: if you planned 50/50 and the arms hold 50,900 and 49,100 users, something is wrong with assignment, logging or filtering, and the result cannot be trusted whatever it says. Kohavi and colleagues recommend testing every experiment for it, usually with a chi-square test and a strict threshold such as p < 0.001 to avoid false alarms. And novelty and primacy effects: users poke at anything new, then settle, so the first days overstate (or understate) the lasting effect. Run at least a full week and longer if the change is visible.
Without JavaScript: 50,900 against 49,100 with a planned 50/50 split gives χ² = 32.4 and p about 1 in 80 million: a mismatch. 50,200 against 49,800 gives χ² = 1.6 and p = 0.21: fine.
How often ideas win, and when a "win" is real
Most ideas do not work. Ron Kohavi and colleagues reported in 2009 that only about one third of the ideas tested on Microsoft's experimentation platform improved the metric they were designed to improve; the rest were flat or harmful. Jim Manzi wrote in 2012 that about 10% of Google's controlled experiments led to business changes. Rates differ by company and by how well optimised the product already is: the better the product, the harder it is to improve. The practical lesson is that "we tested it and it won" is a claim about a rare event.
That rarity changes what a significant result means. If only a small share of ideas truly help, a test with a 5% false-positive rate will still produce some false wins from the large pile of ideas that do nothing, and these can be a big share of the wins you see. Kohavi often cites Twyman's law for this: any figure that looks interesting or different is usually wrong. A result that is surprisingly large deserves a second look (an SRM, a logging bug) before a celebration.
Without JavaScript: of 1,000 ideas where 10% help, 80% power and a 5% false-positive rate find 80 real winners and 45 false ones, so about 64% of the winners are real. Where one third of ideas help, the share is about 89%. The tool treats every idea that does not help as having the full α chance of looking like a winner, a simple simplification.
Metrics that mislead: Goodhart, guardrails and NPS
Goodhart's law says that when a measure becomes a target, it stops being a good measure. Charles Goodhart made the point about monetary policy in 1975; the anthropologist Marilyn Strathern gave it the wording above in 1997. Product examples are everywhere: reward a team for time spent in the app and notifications multiply; reward support for short calls and calls end before problems are solved. The best-known cautionary case is Wells Fargo, where aggressive cross-selling targets were followed, as the US Consumer Financial Protection Bureau announced in 2016, by employees opening around two million accounts customers had not asked for, and a $185 million fine in total with other regulators.
The defence is a guardrail metric: something you promise not to harm while chasing the main one, such as page load time, unsubscribe rate, refunds or support contacts per user. A test that lifts clicks and doubles unsubscribes has not won. Pair every target metric with at least one counter metric.
The Net Promoter Score is a popular single question: how likely are you to recommend us, 0 to 10? Promoters (9 and 10) minus detractors (0 to 6) gives the score, from −100 to +100. Fred Reichheld introduced it in the Harvard Business Review in 2003 as "the one number you need to grow". It is easy to collect and easy to compare, but critics note that a 2007 study by Keiningham and colleagues in the Journal of Marketing found it did not predict firm growth better than other satisfaction measures, and that one score hides why people feel as they do. Use it as a conversation starter, with the free-text follow-up, not as the whole dashboard.
Measuring people: privacy, consent and ethics
Every number in this piece comes from watching people, and the law now says so. The EU's General Data Protection Regulation (GDPR) has applied since 25 May 2018; it requires a lawful basis for processing personal data, such as consent or a legitimate interest that you have weighed against users' rights, and sets fines of up to 20 million euros or 4% of worldwide annual turnover for the gravest breaches. Brazil's Lei Geral de Proteção de Dados (LGPD, Law 13.709 of 2018) has applied since September 2020, with fines enforced since August 2021 by the national authority, the ANPD, of up to 2% of Brazilian revenue per infraction, capped at 50 million reais. India's Digital Personal Data Protection Act was passed in 2023. California, Canada, South Korea, China and many other places have their own rules. This is a description, not legal advice: ask the lawyer or data protection officer in your own organisation.
Ethics goes beyond law. In 2014, researchers at Facebook and Cornell published a study in PNAS in which the news feeds of 689,003 users were altered for a week to see whether the mood of posts shaped the mood of what people wrote. The paper drew a wave of criticism about consent, and the journal's editor published an expression of concern. The practical habits that follow are collect only what the question needs, prefer aggregate data, set a deletion date, and ask whether you would be comfortable explaining the test to the users in it.
Template: an experiment plan
Writing the plan before the test is the cheapest protection against peeking, moving goalposts and arguing afterwards. Fill it in; the "Copy as text" button gives you a plain list to paste into your tracker. Nothing is stored or sent.
What a graduate just sees
- Every output metric has input metrics under it, and the useful question is which input has the most room.
- A retention curve that flattens is a business; one that decays to zero is a leaky bucket, whatever the signup chart says.
- Halve the effect you want to detect and you need four times the users.
- A p-value is only 5% if you look once. Looking daily and stopping when it dips is how false winners are made.
- A split that is not the split you designed (SRM) is a bug until proven otherwise.
- When most ideas fail, a surprising win is more likely a mistake than a discovery.
- Any target metric needs a guardrail, or the team will find the cheapest way to hit it.
- Counting people is a privacy decision before it is a technical one.
Where it shows up next
- Launching & Growing Products (36/99): product-market fit is a retention question, and a launch is judged by cohorts, not by launch-day traffic.
- Experiments & Causal Inference (36/34): the why behind randomisation, interference between users, and what to do when you cannot randomise.
- Shipping Software (36/76): canary releases and feature flags are the machinery that makes an experiment possible.
- UI & UX Design (36/75): small usability tests tell you why; A/B tests tell you how much.
- Data Ethics (36/37) and Marketing Research & Analytics (34/59): consent, fairness and sampling beyond the product.
If you want to go deeper
- 📘 Textbooks. Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020), the practitioner's reference on exactly the traps above; and Alistair Croll and Benjamin Yoskovitz, Lean Analytics (O'Reilly, 2013), on choosing the one metric that matters at each stage.
- 📘 Gentler. Stefan Thomke, Experimentation Works (Harvard Business Review Press, 2020), on building a culture of testing; and Eric Ries, The Lean Startup (2011), for vanity versus actionable metrics.
- 🌐 Free course. Udacity, A/B Testing, a free course taught by Google staff covering metrics, ethics, experiment design and analysis.
- 🌐 Free tools. Evan Miller's sample size calculator and the book's companion site.
Choose few metrics, tie each to the inputs that move it, watch cohorts instead of totals, and treat a test as a promise made in advance: the size, the length, the one reading. Most ideas fail, so a trustworthy "no" is as valuable as a "yes". And remember that behind every metric is a person who did not agree to be a data point unless you asked.
Try this: pick one number your team celebrates. Write down the one thing it is supposed to cause, one way it could rise while that thing falls, and the guardrail that would catch it.
Sources. McClure, "Startup Metrics for Pirates" (2007); Ries, The Lean Startup (2011); Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020), including the Microsoft one-third figure (Kohavi et al. 2009) and Manzi (2012) on Google; Armitage, McPherson and Rowe, "Repeated significance tests on accumulating data," Journal of the Royal Statistical Society A, 1969; Fabijan et al., "Diagnosing sample ratio mismatch in online controlled experiments," KDD 2019; Goodhart (1975) and Strathern (1997) on the law that bears his name; Reichheld, "The One Number You Need to Grow," Harvard Business Review, December 2003; Keiningham et al., Journal of Marketing 71(3), 2007; Kramer, Guillory and Hancock, PNAS 111(24), 2014; Regulation (EU) 2016/679 (GDPR), Art. 83; Lei 13.709/2018 (LGPD), Art. 52; CFPB press release on Wells Fargo, 8 September 2016. All app and test numbers in the tools are invented for illustration.
- 📘 Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments
- 📘 Croll and Yoskovitz, Lean Analytics
- 🌐 Udacity, A/B Testing