Blog
Sample-size myths that waste mobile sprints
Mobile teams often inherit a superstition: run every experiment for two calendar weeks. That habit feels disciplined. It is usually a substitute for thinking about baseline conversion, minimum detectable effect, and how noisy your funnel actually is.
Myth: more days always means more power
Extra days help only if they add independent users into the assignment pool. If the same power users recycle through your experiment, you are mostly collecting correlated noise. Write the unit of randomization clearly — user, device, or account — before you stretch the calendar.
Myth: industry benchmarks replace your variance
Someone else’s onboarding conversion is not your variance. Pull four weeks of your own baseline, include weekend dips common in Thai consumer apps, and estimate MDE as a business decision, not a spreadsheet default.
What we teach instead
In Experiment Signal Lab, learners complete a one-page power brief: baseline, MDE, expected weekly eligible users, and a kill date. If the math says you need three months for a 0.3pp lift, the brief forces a choice — enlarge the effect, change the metric, or cancel.