Do you really know the effectiveness of your email marketing campaigns? As a marketer it’s tempting to make assumptions based on ‘what we think works’, rather than ‘what we know works’. So, how do we move away from guessing?

A/B testing and its components - reporting, testing and optimizing, is one way to know if your campaigns are effective. As Movio Expert Sarah Lewthwaite writes in her Mastering Cinema Marketing Campaigns blog series: “Regularly reviewing results ensures that you understand which of your campaigns are working, and which require improvements in order to achieve the best possible ROI”. For example, if your campaign displays low open rates this could mean it’s time to change up your subject line. However, knowing which subject line is going to yield you optimum opens and conversions is where testing comes in.

Testing can give you insight into which elements work in a campaign. This in turn helps to measure conversions, optimize campaigns and improve effectiveness. A/B Testing is one way to effectively analyze your campaigns. This testing method is not a new concept in the marketing world; Google ran their first A/B test in 2000 and even Obama’s campaign team used the technique in 2007 during his nomination bid to secure more funding, volunteers and signups.

What is A/B testing?

Essentially the A/B testing method allows you to compare two different versions of something on the same channel to determine which one performs better. In this blog, we will focus on email A/B tests. Some of the elements you can test include subject line, preheader text, length of copy, personalization, imagery and CTA’s. An example A/B test you run could include the following workflow;

• Create two versions of the same email - version A has a different subject line from version B
• Send out both versions at the same time to a sample percentage of your target audience
• Whichever version gets the most opens would be the ‘winning’ version and would be sent to the remainder of your audience

6 steps to A/B Testing

Now that you know you want to A/B test, how do you actually go about executing one? There are six essential steps to follow:

1. Set your goal - As with all campaign set up, you want to approach A/B testing with a clear goal to achieve and conversion to improve. Your ultimate objective could be to improve film ticket sales from an email, or retain loyalty members with an exclusive offer.
2. Decide what to test - In order to achieve your goal, which elements can you test to optimize your conversions? If it’s ticket purchases you may test the wording or placement of your CTA buttons which are encouraging ticket sales.
3. Create your variations -If you’re testing opens, then you’ll need to create two different subject lines and/or preheader texts. If it’s clickthroughs, you’ll create two different email versions.
4. Run your test - Establish the time you want to send out your ‘winner’ and send out the two test variations. In Movio Cinema we automatically determine the optimal size of your target audience based on statistical confidence and loyalty database size. Your winner will be sent based on the results of the test.
5. Analyze your results - Once your test is complete it’s time to evaluate and learn from the results. Did one version result in more opens and consequently more conversions to achieve your goal? Gaining insights from these tests allows you to optimize your future campaigns for ongoing success. Furthermore, live and ROI reporting in Movio Cinema allows you to see if the conversion metric goal you set has been achieved.
6. Repeat -  A/B tests should not be a one-time activity, they should be an ongoing part of your marketing strategy. It’s best to run a test multiple times to build up stronger evidence.

What makes Movio Cinema’s A/B Testing unique?

When it came to building out A/B Testing functionality in Movio Cinema we identified the essential elements to make this tool the most effective it could be. Some of the below features are unique to Movio Cinema:

• Automatic test size selection based on statistical modeling and loyalty database size, to optimize open and clickthrough rates
• Live reporting on both tests for opens, clicks, unique open rates, delivery rates, as the test runs its course
• Incremental uplift reporting on the winning campaign

Best practices help to refine

As with most marketing principles there are some best practices you can follow to make sure you’re getting the most out of A/B testing:

• Have a clear plan of what you want to test and why. Set your objective before running tests.
• Only test one element at a time, otherwise your results will be unclear.
• Test for a substantial amount of time so you engage a large enough group to be accurate. However, test for too long and other variables you can’t control could have influence.
• Do multiple tests before making conclusions. For example, if you’re trying to find out if using someone’s first name in a subject line makes a difference to its open rate, test this on multiple campaigns to confirm patterns.
• Start slow and steady. Test strategically rather than testing everything in all campaigns all at once.
• Report and record all your results of any A/B test to help you easily draw conclusions on what works and what doesn’t. This will help you and your team when planning and building future campaigns.
• Be realistic, not all A/B tests are going to produce amazing results. The key here is to keep going, so tests are seen collaboratively rather than in isolation. There is also lessons to be learned from negative results.

Summary

In summary A/B testing is an essential tool in the marketer’s toolbox, to be used alongside other mechanisms and tactics.