Start with this quarter's focus
Write one line at the top of the backlog: "This quarter we are improving ...", naming a single metric from your scorecard. If you have not chosen one, review your 12-metric scorecard with Review your 12-metric scorecard and choose one metric to focus on first. The scoring only makes sense relative to a goal.
The sources of ideas and the basic scoring methods are in Building your backlog. This chapter is about what you do with that list every quarter.
Archive before you score
Scoring dead ideas wastes time. Move an idea to an "archive" tab if any of these is true:
- It has sat untouched for two quarters.
- The page, product or channel it targets has changed.
- A past result already answers the question. Check your learnings log first.
- Nobody on the team can say which metric it would move.
Aim to keep 20 to 30 active ideas. If you have 80, you are keeping a diary, not a backlog.
Re-score impact in revenue terms
The 12 metrics multiply, so a 10% lift in any one of them is worth roughly 10% more revenue. That makes impact simple to estimate: how big a lift could this idea plausibly deliver on its metric? Score on a 1 to 5 scale with fixed anchors, for example 1 for under 2%, 3 for around 5% and 5 for 10% or more.
Then apply the focus rule. An idea that targets this quarter's focus metric keeps its score. An idea that targets another engine loses a point, unless it is cheap enough to run in a day.
Re-score confidence with what you learned
Confidence is the score that should move the most between quarters. Raise it when a past test, a customer interview or a clear data pattern supports the idea. Lower it when similar tests failed. Write the evidence in one line next to the score. If you cannot, the confidence is 1 or 2.
Score effort in real working days
Count everyone's time, not just yours: design, development, copy, setup, analysis. Use days, and round up. An experiment that needs a developer for a week is not "easy" because the idea is simple. Underestimating effort is how teams end up running one experiment all quarter.
Rank and cut the slate
Calculate impact times confidence, divided by effort, and sort. Sanity-check the top ten by eye. If something obvious sits low, find out why before you override the formula. Then pick the slate: roughly two or three experiments per month, as set out in Run your monthly experiment cycle. Check that at least the top five target the focus metric.
Record what changed
Note the date and the three biggest movements: what rose, what fell, what was archived. It takes five minutes and saves you from re-arguing the same ideas next quarter.
Common mistakes
- Inflating scores on ideas the team is excited about.
- Scoring from last quarter's numbers instead of from the current focus.
- Treating effort as a feeling instead of counting days.
- Keeping every idea forever.
- Re-scoring in the middle of a running cycle and changing the plan every week.
How you know it works
The active list has 20 to 30 ideas, each with a metric, a score and a line of evidence. The top five all point at this quarter's focus. You can explain why number one beats number two in one sentence. At the next review you can see which scores moved and why, and no experiment has been running longer than planned.
