Every 4 years, the US Presidential elections are making lots of headlines worldwide, but it’s also a good place to look for ways to effectively predict your customer’s behavior. Have you ever wondered how the media can declare a candidate to be the winner before all of the ballots are counted? The political process has long used surveys and exit polls to help predict an election’s result. By identifying and isolating different groups, these polls are used to help calculate how the rest of the population will behave come election day.
Similarly, when running a targeted marketing campaign, a control group is used to help marketers quantify the success of their campaign by comparing the behavior of the targeted group with that of a sample size similar group that didn’t receive the campaign.
Control groups have a wide range of applications. They are often used in scientific research and political science, and in the case of our product, Movio Cinema, they can provide amazing insights on the success of targeted marketing campaigns.
So what exactly is a control group?
It’s a statistically significant subset of your customer database - for Movio Cinema customers, their loyalty member database - that is isolated and excluded from a campaign. This enables you to accurately measure the performance of your campaign through the transactions and behaviors of the audience you’ve identified as your targeted group versus that of your control group.
By using control groups, marketers are able to answer the question, “Did my offer or campaign influence the customer’s behavior or was it behavior that customers would have exhibited anyway?”
So what do you need to know about control groups? How big should they be? How will the results with control groups influence how you implement future campaigns?
Eyes on the prize
Before you execute a campaign you need to have an objective or goal - what is it that you want to achieve? It might be that you wish to re-engage with lapsed loyalty members or maybe you want to increase visitation? Your motivations can be monetary or behavioral as in order to refine your marketing you will need to look at more than revenue. Studying customer behavior may not bring you financial gains immediately but it will give you key insights that will allow you to better understand your customers and target them better at a later date.
As part of setting your goal, ensuring that you have defined a timeline for running your campaign is a really important step. This will impact any concurrent campaign you may want to run, or will restrict you from doing so. The duration is relevant to the context of the content as well.
Example:
- Campaign - run a ‘last chance to see’ campaign to cinema loyalty members for a movie that is in its last week of screening
- Goal - increase box office revenue for a specific film
- Consider - targeted customers will only have a limited time to take advantage of this offer
- Duration - the duration of a campaign depends on the offer and the expected level of uptake - in this example one week. Once this date has passed you can analyze your results via the Movio Cinema ‘Campaign Report’ dashboard.
Avoid ‘cross contamination’
It is imperative that both your target and control groups don’t take part in any other campaigns while they are part of an existing one. This is so that the control group continues to exhibit normal behavior that will not be influenced in any way by any other factors. However, if you send regular standard recurring communications like a newsletter you can continue to do so, as long as they are sent to both your control and targeted groups. It will become very difficult to understand the cause of behaviors if either of your groups become ‘contaminated’ by anything that can influence their behavior.
How to build a control group
In order to have a statistically significant control group, you should use a random sample of the group that you would like to target. This ensures that the customers in the control group will share similar traits to that of the target group. Use of an automated system to randomize the control group members is highly recommended. In Movio Cinema, if you construct a sample group using the segmentation tool, the software will allow you to remove a random audience to form a control group.
Choosing the right sample and control group size
The size of your control group will influence the amount of sampling error in the resulting calculations. The most commonly analyzed behavioral measure is visitation rate, so we’ll focus our discussion on this. Sampling error is a consequence of the fact that only a small (or not so small) sample of the targeted group is excluded as a control. This set of randomly chosen customers will exhibit a certain visitation rate over the campaign period, but had a different set of customers been randomly selected, this percentage would have been slightly changed, and each sampled rate will be slightly different than the true population value. The difference in these rates is the sampling error.
Rule of thumb: Campaigns with a small number of targeted customers may require a higher percentage assigned to the control group, so as an idea, for anything less than a couple of thousands of customers you can set the control group percentage around 10-20%. If the overall targeted customer number is higher, then 5% for the control group may be sufficient. For example:
- If running a campaign against 400 customers, you want your control group number to be significant enough, but also have as many of them available to receive the offer. In this case, your control group can have from 40 to 80 customers.
- However, if running a campaign against 10,000 customers, the control group can be calculated based on a 5% of that total number of customers. A control group of 500 customers is statistically significant to give the marketer a good idea of the effectiveness of a campaign.
In Movio Cinema, the control group size is determined by two user-selectable values - the ‘confidence level’ and the ‘margin of error’. The confidence level indicates the level of confidence that, if the campaign were repeated, the true value for the population would be within the margin of error of the sample value. A 95% confidence level specifies that the true population value would be within the margin of error in 19 of 20 repeated experiments, while a 99% confidence level specifies that this would occur 99 times out of 100.
If we return to our election analogy, the margin of error is the plus/minus figure that is typically reported in the media’s opinion polls. This provides a measure of the maximum difference between the response rate of the sample and the response rate of the population as a whole. For a given sample size, the margin of error depends on the response rate, with the maximum error occurring at a response rate of 50%. It is important to note that the margin or error is an absolute rather than a relative value. For example, a result of 10% plus or minus 3% means the true rate is between 7% and 13%, not 9.7% and 10.3%.
The control group auto-calculation tool in Movio Cinema will help you get the optimal number of loyalty members for your campaign, using the above standards.
Expected Response Rate
In some cases, depending on the type of the campaign, you may have an idea of what sort of uplift to expect. This is something that people usually don’t consider when running campaign experiments, but it can play a role in deciding how large the control group should be. When dealing with customers that don’t normally react to email campaigns, you may expect only a small uplift, so you need to ensure that the control group is large enough to distinguish between two similar visitation rates. However, if you expect a large uplift from a particular campaign, a smaller control group can be constructed, as the target visitation rate should be well outside even a large margin of error. The advantage of this approach is that for campaigns that are expected to be very successful, fewer targeted customers need to be sacrificed to the control.
Measuring the results
Once your targeted marketing campaign is complete, you can start the process of compiling and analyzing the results. As mentioned earlier, you can view results in the ‘Campaign Reports’ dashboard in Movio Cinema, and if a control group was used, the performance metrics are comparatively displayed. Because Movio Cinema seamlessly integrates with cinema point-of-sale, one interesting metric is ‘Return on Investment’. You can see, an actual monetary value, how effective your campaign was. Therefore, when you analyze the campaign, you can see the evolution of your control group and the target group’s visitation rate.
In the case of a very successful campaign, you can still engage with your control group members, by sending them the same campaign so that they can take advantage of the offer. You can easily target them using the Movio Cinema segmentation tool and execute the same campaign again for this group.
It would be wrong to assume that running a campaign across your overall loyalty members' database will give you a clear value of how successful that campaign was unless you use a control group in Movio Cinema. You can see data related to the targeted group, but without the values for the control group, there is no way to tell how those numbers are influenced by the offer in the campaign or if it’s just normal behavior.
In summary
Targeted marketing campaigns can provide immediate results but you can ensure better effectiveness just by using control groups. There are a few steps to follow:
- Randomly select your control group members once you’ve identified your targeted group, ideally by using automated tools that will remove the human factor from this process.
- Remember that the integrity of your results is assured only by removing all other factors that can influence the customer's behavior.
- Finally, it is a matter of taking advantage of the ability to analyze the results and learn from each campaign.
You should see your campaigns as continuous ways to learn, improve and refine to deliver the best offers to your customers.
