What is a Holdout Testing?
Holdout testing is the process of reviewing your email marketing program to quantify if the campaigns being sent are generating increased engagement metrics/conversions vs not sending anything. Are these email campaigns/automation really giving an incremental increase in engagement metrics/conversions/increase in revenue that would otherwise not be realized.
This holdout test is like an A/B test, as it has two segments, but one segment receives no emails. They are excluded from the mailings. You then measure the purchase/conversion behaviour of the two segments to define the level of increased engagement that the segment being emailed has over the segment being excluded. In most cases (one would hope), the group receiving email will have a higher conversion rate.
For additional needs, you can also contact email marketing experts and get the results to the fullest.
More to think about:
There are many costs to factor in after you get the results. You need to weigh the email marketing packages of the program you are testing. Cost of the ESP to send the campaigns, cost to design, code, test, and setup/deploy the email campaigns, count your manhours. Use that as part of the cost of doing campaigns. A small increase in revenue may be outweighed by the costs of that program.
Depending on your list size, type of email campaigns/automation and mailing frequency, this type of holdout test may take 30-90 days to complete.
Why you should do Holdout Testing?
The main reason for running holdout group tests is to understand if you are cannibalizing natural sales that would have happened if no marketing email were sent. For a company that regularly sends discounts and/or offers to its customers via ecommerce emails, this process enables you to determine if those discounts/offers are in some way cannibalizing the profits of your business.
The worst-case scenario for the company would be the segment of customers who did not receive an email end up purchasing your products at the same or higher rate. That essentially means you’re giving away your profit margin when it’s not needed.
Let’s use a Cart Abandonment program as an example:
If you are doing testing, running a cart abandonment program or really any email marketing campaigns you should have a holdout group. A holdout group is a group that does not get that specific type of mailing. That’s your baseline of “what if we didn’t send them anything? What is there natural progression through the buyer journey if not prodded by email.
For example: If you run a cart abandonment program, you should first know what your natural cart abandonment return rate is with no program running. In other words, how many people come back on their own, with no messaging to buy what’s in their cart. How many come back the first day, the second day, the third and so on.
Read More about browse abandonment email & it’s examples.
Once you know your natural return rate you can then measure the true value of a program. If your natural return (conversion) rate is 5% and when running your program, you see a total of an 11% return rate, then you know the program generated an additional 6% return. The program didn’t create 15% because you now know your natural return is 6%. Those percentages will change month-over-month so reviewing your programs regularly is imperative.
Now you can measure true decrease or increase in revenue (to a point, as there will always be variables out of your control that will affect results) from that program as well. If over a 12-month period the natural returns delivered an average of $10,000 per month, and the Program total delivered $22,000 per month, you know the programs value over the baseline natural return. These numbers will vary over time with some peaks and valleys and will be influenced by sales and holidays, so adjust as needed.
Wrapping this up:
Some companies hold out 10% of subscribers (sometimes using a dynamic segment of subscribers for each holdout test process) as a complete blackout during testing or all year. If you have a very effective and robust email campaign management program, it seems silly to ignore the increase in revenue to that 10% group, so I suggest doing your testing a few months each year but shift the test months each year. If you are an e-commerce company and do heavy volume/revenue during the holidays, don’t test then, grab all the revenue you can during the holidays. Test throughout the rest of the year.
It should be noted that if you are going to Holdout test or do a holdout group, your segment or list size should be at very least 50,000 subscribers. Anything less would be statistically irrelevant. Also, make sure your holdout group matches the sending group. If you are sending to actives, the hold group should be actives. Don’t use a random segment process, random tends to be anything but random. Try to give fair advantage to each group to succeed/fail equally.
Thanks for stopping by our blog and if you’ve found this valuable, please share it on your social channels.
———————————————-
If you need help with this type of testing or looking for an email marketing agency, we offer a free 30-minute phone consultation. No hard sell just a call to see if we might be a fit for your needs.
