A/B testing

Without a test, a redesign or a new subject line is judged on opinion, and a page that looks better can easily convert worse. The random split is what makes the result trustworthy. Because both groups come from the same audience at the same time, a difference between them can be put down to the one change, not to who happened to see it or to what week it was.

• Write the hypothesis and the deciding metric before starting: what changes, and which number says it worked.
• Change one element at a time. If the headline, button and form all change together, a win cannot be traced to any of them.
• Split the audience randomly and run both versions at the same time.
• Decide the sample size or the run length up front, then wait for it. Check statistical significance before calling a winner.
• Record every result, including the losses, so the next test builds on what was learned.

Version A is what you run today, usually called the control. Version B changes exactly one thing. The audience is split at random, so half see A and half see B, and after enough people have seen each, the two are compared on one number chosen in advance. A newsletter sender might test two subject lines and compare open rates. A shop might test a short product page against a long one and compare purchases.

• Stopping the moment one version pulls ahead. Early gaps often close with more data.
• Testing a tiny audience. A list of a few hundred people cannot reveal a small difference, so test a bolder change or skip the test.