What is Holdout Group Testing?

Holdout testing is the process of reviewing your email marketing program to quantify if the campaigns being sent are generating increased engagement/conversions vs not sending anything. Are these campaigns/automation really giving an incremental increase in engagement/conversions/revenue that would otherwise not be realized.

This is like an A/B test, as it has two segments, but one segment receives no emails.  They are excluded from the mailings.  You then measure the purchase/conversion behavior of the two segments to define the level of increased engagement that the segment being emailed has over the segment being excluded.  In most cases (one would hope), the group receiving email will have a higher conversion rate.

More to think about:

There are many costs to factor in after you get the results.  You need to weigh the cost of the email program you are testing.  Cost of the ESP to send the campaigns, cost to design, code, test, and setup/deploy the campaigns, count your manhours.  Use that as part of the cost of doing campaigns.  A small increase in revenue may be outweighed by the costs of that program.

Depending on your list size, type of campaign/automation and mailing frequency, this type of test may take 30-90 days to complete.

Why you should do this?

The main reason for running holdout tests is to understand if you are cannibalizing natural sales that would have happened if no marketing email were sent. For an e-Commerce company that regularly sends discounts and/or offers to its customers, this process enables you to determine if those discounts/offers are in some way cannibalizing the profits of your business. The worst-case scenario for the company would be the segment of customers who did not receive an email end up purchasing your products at the same or higher rate. That essentially means you’re giving away your profit margin when it’s not needed.

Let’s use a Cart Abandonment program as an example:

If you are doing testing, running a cart abandonment program or really any email marketing you should have a holdout group.  A holdout group is a group that does not get that specific type of mailing.  That’s your baseline of “what if we didn’t send them anything?  What is there natural progression through the buyer journey if not prodded by email.

For example: If you run a cart abandonment program, you should first know what your natural cart abandonment return rate is with no program running.  In other words, how many people come back on their own, with no messaging to buy what’s in their cart.  How many come back the first day, the second day, the third and so on.

Once you know your natural return rate you can then measure the true value of a program.  If your natural return (conversion) rate is 5% and when running your program, you see a total of an 11% return rate, then you know the program generated an additional 6% return.  The program didn’t create 15% because you now know your natural return is 6%.  Those percentages will change month-over-month so reviewing your programs regularly is imperative.

Now you can measure true revenue (to a point, as there will always be variables out of your control that will affect results) from that program as well.  If over a 12-month period the natural returns delivered an average of $10,000 per month, and the Program total delivered $22,000 per month, you know the programs value over the baseline natural return.  These numbers will vary over time with some peaks and valleys and will be influenced by sales and holidays, so adjust as needed.

Wrapping this up:

Some companies hold out 10% of subscribers (sometimes using a dynamic segment of subscribers for each test process) as a complete blackout during testing or all year.  If you have a very effective and robust email marketing program, it seems silly to ignore the boost of revenue to that 10% group, so I suggest doing your testing a few months each year but shift the test months each year.  If you are an e-commerce company and do heavy volume/revenue during the holidays, don’t test then, grab all the revenue you can during the holidays.  Test throughout the rest of the year.

It should be noted that if you are going to test or do a holdout group, your segment or list size should be at very least 50,000 subscribers.  Anything less would be statistically irrelevant.  Also, make sure your holdout group matches the sending group.  If you are sending to actives, the hold group should be actives.  Don’t use a random segment process, random tends to be anything but random.  Try to give fair advantage to each group to succeed/fail equally.

Thanks for stopping by our blog and if you’ve found this valuable, please share it on your social channels.

———————————————-

If you need help with this type of testing or any other aspect of your email marketing program, we offer a free 30-minute phone consultation.  No hard sell just a call to see if we might be a fit for your needs.  Contact us here.

Are you excited and ready for the holiday season?

If you don’t have your email marketing plan put together for the holidays, you’re already behind. Having an organized calendar and a target-oriented customer plan is essential for holiday email marketing success. Plan your holiday email templates as the holidays start in October, so let’s get you caught up now!

Holidays To Plan For…

Halloween
This can be your opportunity to incorporate a spooky, festive spirit in your emails!

Thanksgiving
Say thanks for real. Also, say thanks with sales deals where appropriate.

Black Friday
Okay, so this isn’t a “real holiday,” but it might as well be. Focus on key products and your top-grossing customers for brand leverage and higher ROI.

Cyber Monday
This day is for online shoppers. Make sure your deals are different from your Black Friday deals and have them be online-focused.

Hanukkah
Don’t miss this eight-day festival of lights. It starts late this year (on Christmas Eve) so it can fold nicely into your other holiday campaigns.

Christmas
The big kahuna, if you will. It’s self-explanatory.

Seasonal Elements and Holiday Email Marketing Ideas for Your Campaigns

The key around the holidays is to get your message out, have fun, and keep the holiday spirit going. Here are a few ideas we’ve cobbled together for you:

  • Holiday-themed headers – increase the festive mood with inviting and colorful email themes.
  • Gift-focused service footers – Include offers, coupon codes, gift guides, and links to relevant landing pages
  • Holiday-themed color patterns – Grab attention and boost excitement by incorporating holiday colors.
  • Animated GIFs and Video – Eye-grabbing content can make your holiday email templates stand out!
  • Gift Guides – A great branding technique! Gift guides let your subscribers educate themselves on the various categories of products and services you offer.
  • LastMinute Shopping – Holidays are defined by people forgetting to get gifts. Don’t forget to slide in some last-minute deals.

Final Notes

While you’re incorporating some of these holiday email marketing tips into your campaign schedule, here are some other things to keep in mind:

  • Remember your landing pages. Your emails need to land clickers on a well-designed and optimized landing page to better ensure success.
  • Test for errors before sending. Being safe now is better than being sorry later.
  • Increase frequency for the holiday season with caution. Subscribers generally expect to receive more mail during the holiday season. You can do the same. Just monitor your unsubscribe rate to ensure you’re not going too far and sending too much.

——-

Building a successful holiday email marketing plan can be a challenge. InboxArmy can put together a service package that makes sense for all your holiday email marketing needs (and beyond)!

Multi-ESP Support

We support 50+ ESP vendors. Be it enterprise platforms such as SalesForce and Oracle Marketing Cloud or small and medium sized business platforms such as Mailchimp, Klaviyo, Aweber, and ActiveCampaign, we’ve got you covered.

Many More

Top