What is Holdout Group Testing?
Holdout testing is the process of reviewing your email marketing program to quantify if the campaigns being sent are generating increased engagement/conversions vs not sending anything. Are these campaigns/automation really giving an incremental increase in engagement/conversions/revenue that would otherwise not be realized.
This is like an A/B test, as it has two segments, but one segment receives no emails. They are excluded from the mailings. You then measure the purchase/conversion behavior of the two segments to define the level of increased engagement that the segment being emailed has over the segment being excluded. In most cases (one would hope), the group receiving email will have a higher conversion rate.
More to think about:
There are many costs to factor in after you get the results. You need to weigh the cost of the email program you are testing. Cost of the ESP to send the campaigns, cost to design, code, test, and setup/deploy the campaigns, count your manhours. Use that as part of the cost of doing campaigns. A small increase in revenue may be outweighed by the costs of that program.
Depending on your list size, type of campaign/automation and mailing frequency, this type of test may take 30-90 days to complete.
Why you should do this?
The main reason for running holdout tests is to understand if you are cannibalizing natural sales that would have happened if no marketing email were sent. For an e-Commerce company that regularly sends discounts and/or offers to its customers, this process enables you to determine if those discounts/offers are in some way cannibalizing the profits of your business. The worst-case scenario for the company would be the segment of customers who did not receive an email end up purchasing your products at the same or higher rate. That essentially means you’re giving away your profit margin when it’s not needed.
Let’s use a Cart Abandonment program as an example:
If you are doing testing, running a cart abandonment program or really any email marketing you should have a holdout group. A holdout group is a group that does not get that specific type of mailing. That’s your baseline of “what if we didn’t send them anything? What is there natural progression through the buyer journey if not prodded by email.
For example: If you run a cart abandonment program, you should first know what your natural cart abandonment return rate is with no program running. In other words, how many people come back on their own, with no messaging to buy what’s in their cart. How many come back the first day, the second day, the third and so on.
Once you know your natural return rate you can then measure the true value of a program. If your natural return (conversion) rate is 5% and when running your program, you see a total of an 11% return rate, then you know the program generated an additional 6% return. The program didn’t create 15% because you now know your natural return is 6%. Those percentages will change month-over-month so reviewing your programs regularly is imperative.
Now you can measure true revenue (to a point, as there will always be variables out of your control that will affect results) from that program as well. If over a 12-month period the natural returns delivered an average of $10,000 per month, and the Program total delivered $22,000 per month, you know the programs value over the baseline natural return. These numbers will vary over time with some peaks and valleys and will be influenced by sales and holidays, so adjust as needed.
Wrapping this up:
Some companies hold out 10% of subscribers (sometimes using a dynamic segment of subscribers for each test process) as a complete blackout during testing or all year. If you have a very effective and robust email marketing program, it seems silly to ignore the boost of revenue to that 10% group, so I suggest doing your testing a few months each year but shift the test months each year. If you are an e-commerce company and do heavy volume/revenue during the holidays, don’t test then, grab all the revenue you can during the holidays. Test throughout the rest of the year.
It should be noted that if you are going to test or do a holdout group, your segment or list size should be at very least 50,000 subscribers. Anything less would be statistically irrelevant. Also, make sure your holdout group matches the sending group. If you are sending to actives, the hold group should be actives. Don’t use a random segment process, random tends to be anything but random. Try to give fair advantage to each group to succeed/fail equally.
Thanks for stopping by our blog and if you’ve found this valuable, please share it on your social channels.
If you need help with this type of testing or any other aspect of your email marketing program, we offer a free 30-minute phone consultation. No hard sell just a call to see if we might be a fit for your needs. Contact us here.