Hypothesis testing

Opinions about what will work are cheap and often wrong. A test turns the question into evidence. It also forces a team to agree on what counts as success before the data arrives, which stops a weak result from being reframed as a win afterwards.

It matters most when being wrong is expensive. A small test before a large build can save months spent on the wrong thing.

• Write the hypothesis in one sentence: if X is changed, then Y will happen, measured by Z.
• Pick one main metric. Several metrics make it easy to find something that looks like a win.
• Decide the sample size and the test length before starting, and do not stop early because the numbers look good.
• Keep a control group that sees the original version, so the result is a comparison and not a before-and-after guess.
• Change one thing at a time where possible, so the result has one explanation.
• Record every test, including failures, so the same idea is not tested twice by someone who forgot.

A hypothesis is a prediction that can turn out wrong. "A shorter sign-up form will increase trial starts" is a hypothesis. "Let's improve the form" is not, because nothing could prove it false.

In a formal test, the team also states the default assumption that nothing changed, called the null hypothesis. The experiment then asks whether the data is strong enough to reject it. A result is judged with statistical significance, commonly shown as a p-value.

• Peeking at the results daily and stopping at the first good-looking number.
• Reading "no significant difference" as proof that there is no effect. It may only mean the test was too small to see one.