When performing an A/B test in Movio Cinema the optimal size of each split is automatically calculated for you.
This calculation is dependent on two major factors:
- The number of total recipients for the campaign, and
- If you think one test will significantly outperform the other
When configuring your A/B campaign you are asked if you think one test will significantly outperform the other. Movio Cinema would consider one test significantly outperforming the other if there was more than a 2 percentage points difference.
For example, if you think Test A will have a 25% open rate, and Test B will have a 20% open rate, this is a 5 percentage points difference.
If you select ‘Yes’ the tests will be sent to a smaller number of recipients in order to maximize the number of members who receive the final campaign.
If you select ‘No’ the tests will be sent to a larger number of recipients in order to increase accuracy in identifying the winning test.
Figure 4 below shows the size of tests for each of the two scenarios. For example, if the targeted number of members is 500,000 and you thought one test would significantly outperform the other, the percentage of members in each test would be around 0.5%. However, if you think each test would perform similarly, the test sizes would be around 4.5%. Note each test has an even split of members.
