How to plan your AB tests [Free Template]
Quarter beginnings usually resonate with growth marketers. Maybe because that’s when they are preparing their tactical plan, establishing their goals, or just because it's the time for them to think more deeply about their strategies. Regardless, it hasn’t been different to our customers; and that’s exactly the time of the year they usually reach us asking for help to plan their AB tests.
The demand is mainly about some of these questions:
- How to organize the hypothesis?
- Which hypothesis should I prioritize?
- How to estimate the test duration beforehand?
- Which metrics should I analyze?
- Which method is better for analyzing AB tests?
- How to document the process for future reviews?
The step by step for a good AB testing plan
We've already covered some of those topics in other blog posts. Let's revisit them now.
How to organize the hypothesis?
Listing testing hypothesis requires both data knowledge and creativity, and there's nothing better than brainstorming. Invite all your team members to a 1-hour meeting and let them brood about what should make the conversion rates improve. The more diverse this team is, the better: ask help from designers, copywriters, engineers, data scientists, marketing, and product people.
You should list from 10 to 20 hypotheses to plan your quarter tests.
Which hypothesis should I prioritize?
Deciding where to start can be one of the most challenging steps. Luckily, a smart method can help you with that: the ICE scoring.
The ICE score model is widely used by growth and product teams to prioritize features and experiments. It helps you evaluate each option by pointing out its impact, your confidence in its potential result, and the ease of implementation. Then, you can rank all options by multiplying these three values to calculate the score.
How to estimate the test duration beforehand?
From a purely statistical perspective, estimating the test duration is easy after determining the sample size. However, you have to take some things into account:
- What is your current conversion rate?
- What is the minimum improvement you expect to detect in your experiment?
- How many variations will the test have?
All these factors can affect the duration. But it is also important to highlight that you will only know it after your test runs. If the impact of the variant over the baseline is too small, you would probably want to run the test for at least a little while to observe statistical confidence.
You can use the calculator we provide in our free template.
Which metrics should I analyze?
This should be the easiest step. Usually, your primary metric is very straightforward and highly related to your business goal. However, we strongly suggest you define secondary metrics to help you in the analysis: it is not unusual to run experiments that don't impact the primary conversion rate but change the mid-funnel metrics significantly.
Which method is better for analyzing AB tests?
The most used methods are the frequentist and the Bayesian.
The frequentist inference was developed in the 20th century and became the dominant statistical paradigm, widely used in experimental science. It is a statistically sound approach with valid results, but it presents limitations that aren't attractive in AB testing. On the other hand, the Bayesian approach has become the industry standard based on our benchmark, providing richer decision-making information, although the frequentist is still widely used.
How to document the process for future reviews?
Documenting AB tests should be a very straightforward exercise, but many folks dread this aspect of running experiments. It doesn't need to be demanding, so we made a template to help you organize the most critical information. It should guide you on documenting the hypothesis, the target metrics, the results, and so on.
A free template guide for you
To help you plan your AB tests, we've designed a free template in a spreadsheet format.
This guide should provide you with:
- A list of ideas to test on your website
- A tool to help you prioritize your experiments using the ICE score
- A calculator to estimate how long you should run your tests
- A template for documenting your experiments
Feel free to download it and share it with your friends if you find it useful!