Product Metrics Anti-Patterns

Vanity metrics are metrics that look impressive in a presentation but are disconnected from the product's actual health or trajectory. Common vanity metrics in SaaS: total registered users (includes everyone who ever signed up, including churned, never-activated, and long-inactive users — looks large but is a terrible proxy for the active user base); page views (easily inflated by low-quality traffic, bots, or confusing UX that makes users click multiple times to accomplish a task); total app downloads (for mobile products); LinkedIn followers; and press release mentions. They persist because they are easy to grow, always increase (they are cumulative counts that can only go up), and produce satisfying narrative charts for all-hands presentations. The antidote is pairing every vanity metric with its quality-adjusted equivalent: instead of total users, report MAU + activation rate; instead of page views, report pages with > 30-second engagement; instead of downloads, report 7-day retention. The quality-adjusted metric reveals whether the vanity metric growth is meaningful.

Survivorship bias in product analytics occurs when teams analyze only the users who are still present in the data, ignoring those who have left — producing systematically optimistic conclusions. Classic SaaS example: a team wants to understand which onboarding actions lead to retention. They analyze the behavior of their current active users in week 1 and find that 85% of active users added a team member during onboarding. Conclusion: "adding a team member early is the key behavior." Anti-survivorship analysis: analyze the cohort of all users from 90 days ago (not current active users) and compare the week-1 behavior of those who are still active vs. those who churned. This reveals whether adding a team member actually predicts retention or whether it is simply common behavior for the majority of users regardless of outcome. "Retention correlation analysis" without cohort control is almost always survivorship-biased. Product Ops trains PMs on cohort-controlled analysis methodology as a standard practice in user research interpretation.

The most dangerous and common analytical mistake in product operations: observing that two things happen together and concluding that one causes the other. Real example: a SaaS company observes that users who set up the daily digest email notification have 4× higher 90-day retention than users who do not. They conclude: "if we force all new users to set up the daily digest, we will dramatically improve retention." They implement mandatory notification setup in onboarding. Retention does not improve. The truth: users who voluntarily set up daily digest notifications are already engaged and committed — they would have retained regardless. The notification setup is a symptom of high engagement, not its cause. Test before you conclude: the only reliable way to establish causation is a randomized controlled experiment — randomly assign some users to a group that is nudged toward the "correlated" behavior and compare their retention to a randomized control. If the randomly nudged group improves, causation is supported. Product Ops should require that every "discovery" from product analytics passes a basic causation validation — either through experiment or a plausible causal mechanism — before it informs a product decision.