What is Holdout Group Testing?

Holdout testing is the process of reviewing your email marketing program to quantify if the campaigns being sent are generating increased engagement/conversions vs not sending anything. Are these campaigns/automation really giving an incremental increase in engagement/conversions/revenue that would otherwise not be realized.

This is like an A/B test, as it has two segments, but one segment receives no emails.  They are excluded from the mailings.  You then measure the purchase/conversion behavior of the two segments to define the level of increased engagement that the segment being emailed has over the segment being excluded.  In most cases (one would hope), the group receiving email will have a higher conversion rate.

More to think about:

There are many costs to factor in after you get the results.  You need to weigh the cost of the email program you are testing.  Cost of the ESP to send the campaigns, cost to design, code, test, and setup/deploy the campaigns, count your manhours.  Use that as part of the cost of doing campaigns.  A small increase in revenue may be outweighed by the costs of that program.

Depending on your list size, type of campaign/automation and mailing frequency, this type of test may take 30-90 days to complete.

Why you should do this?

The main reason for running holdout tests is to understand if you are cannibalizing natural sales that would have happened if no marketing email were sent. For an e-Commerce company that regularly sends discounts and/or offers to its customers, this process enables you to determine if those discounts/offers are in some way cannibalizing the profits of your business. The worst-case scenario for the company would be the segment of customers who did not receive an email end up purchasing your products at the same or higher rate. That essentially means you’re giving away your profit margin when it’s not needed.

Let’s use a Cart Abandonment program as an example:

If you are doing testing, running a cart abandonment program or really any email marketing you should have a holdout group.  A holdout group is a group that does not get that specific type of mailing.  That’s your baseline of “what if we didn’t send them anything?  What is there natural progression through the buyer journey if not prodded by email.

For example: If you run a cart abandonment program, you should first know what your natural cart abandonment return rate is with no program running.  In other words, how many people come back on their own, with no messaging to buy what’s in their cart.  How many come back the first day, the second day, the third and so on.

Once you know your natural return rate you can then measure the true value of a program.  If your natural return (conversion) rate is 5% and when running your program, you see a total of an 11% return rate, then you know the program generated an additional 6% return.  The program didn’t create 15% because you now know your natural return is 6%.  Those percentages will change month-over-month so reviewing your programs regularly is imperative.

Now you can measure true revenue (to a point, as there will always be variables out of your control that will affect results) from that program as well.  If over a 12-month period the natural returns delivered an average of $10,000 per month, and the Program total delivered $22,000 per month, you know the programs value over the baseline natural return.  These numbers will vary over time with some peaks and valleys and will be influenced by sales and holidays, so adjust as needed.

Wrapping this up:

Some companies hold out 10% of subscribers (sometimes using a dynamic segment of subscribers for each test process) as a complete blackout during testing or all year.  If you have a very effective and robust email marketing program, it seems silly to ignore the boost of revenue to that 10% group, so I suggest doing your testing a few months each year but shift the test months each year.  If you are an e-commerce company and do heavy volume/revenue during the holidays, don’t test then, grab all the revenue you can during the holidays.  Test throughout the rest of the year.

It should be noted that if you are going to test or do a holdout group, your segment or list size should be at very least 50,000 subscribers.  Anything less would be statistically irrelevant.  Also, make sure your holdout group matches the sending group.  If you are sending to actives, the hold group should be actives.  Don’t use a random segment process, random tends to be anything but random.  Try to give fair advantage to each group to succeed/fail equally.

Thanks for stopping by our blog and if you’ve found this valuable, please share it on your social channels.


If you need help with this type of testing or any other aspect of your email marketing program, we offer a free 30-minute phone consultation.  No hard sell just a call to see if we might be a fit for your needs.  Contact us here.

According to Finder, 86% Americans are planning to spend their hard-earned cash on pre-Christmas sales like Black Friday, Cyber Monday, and Singles Day. Adobe Analytics has predicted that retailers will bring in as much as $7.5 billion on Black Friday in 2019, which is more than 20% increase over 2018.

As the big Friday of 29th November is just around the corner, we are here with awe-inspiring email examples that would help you up the email game this Holiday season.


Subject line: Black Friday Deals on LG Signature Kitchen Packages

The brand has sent an interesting email with perfect imagery to encourage purchases. Although the images are appealing enough, the design seems a bit overwhelming for the recipient. However, the good part is that they have included their offer right at the top of the email.

  2. Bella Ella Boutique, LLC

Subject line: New markdowns up to 70% off

The email begins by creating a sense of urgency in the subscriber’s mind with the statement “16 DAYS UNTIL BLACK FRIDAY”. It will tempt the subscriber to get on the VIP text list by building “fear of missing out”. The hero image is catchy and informs the subscriber about the exciting 70% off. The email also has attractive images to display the dresses on sale. They have used fancy typography to lay emphasis on specific sections of the email. The last section with GIFs is also quite engaging for the reader.

3. Mattress Firm

Subject line: Want a bigger bed AND up to $600 in savings?

With a nice preheader text and view in browser link, the email follows all the best practices of creating an email. Furthermore, it also breaks the monotony by doing away with the column layout. Besides their Black Friday sale- early access, they have also promoted the military offer for Veteran’s Day.

4. saferwholesale.com

Subject line: Black Friday Preview – Up To $40 OFF DELIVERY @ SaferWholesale – Call Now!

It is a short and sweet email that perfectly syncs with the mood of Black Friday. The black and red color are perfectly used to create that festive aura. It adheres to the best practice of having 80:20 text to image ratio. There are three different CTAs the correctly redirect the reader to the relevant landing page.

Over to you

If you are struggling with a resource crunch and looking for professional email templates for the Holiday season, just share your requirements with us and we would love to help you out.

Thanksgiving Day is one of the most favorite holidays in America. In 2018, 165M+ shopped over the Thanksgiving period of five days. 65% of Thanksgiving weekend shoppers said deals drove them to buy. Moreover, it is predicted that total US retail sales will reach 3.8% to $1.008 trillion. If you want to make the most of the first-ever trillion-dollar holiday season, you should focus on creating better emails that would give you the best results in terms of conversions

That said, let’s have a look at the email inspirations for Thanksgiving Day.

1. Adorable Essentials

Subject line: Spice Pumpkin & Grace Dress 50% Off – Perfect for Thanksgiving!!

The hero image perfectly matches the colors of fall and gives Thanksgiving Day vibes to the email recipient. The offer is highlighted in plain text so that there is no renderability issue and all the recipients are able to know the purpose of the email. It is an important aspect of email accessibility too. The email ends with promoting their reward program and social media icons.

It would have been good to have a preheader and link to view the email online.

2. 41 Olive

Subject line: Make Your Thanksgiving Feast That Much Better!

The email has talked about their offer right in the hero image, along with a clear CTA. The next section showcases the flavors of Fall with distinct CTAs for each product. Also, note how the fonts change in each section. It is a good idea to share the map of the store in the last section followed by the links to Facebook and Twitter social profiles.

3. BannerBuzz

Subject line: Thanksgiving Cheer Comes Early This Year!

It is a perfectly designed email with a “View in Browser” link and contact details mentioned right at the top of the email. They have clearly conveyed their offer and the code to avail it. The CTA is included in the first fold to drive more conversions. They have promoted their individual products in the next section with separate CTAs for each. The theme of the email matches the Thanksgiving season quite well.

4. Rue La La

Subject line: Gather ’round! Thanksgiving Weekend Style is on.

Rue La La sends out a short and sweet email to promote their Thanksgiving Day outfits for different purposes. The GIF is sure to grab the attention of the email subscribers and entice them to make the purchase.

However, it would have been better to have more informative alt-text for this email.

5. Jeni’s Ice Creams

The hero image is enough to tempt the recipient to try out their delicious ice creams but in case it’s not, the supporting copy would do the magic. The email follows all the email marketing as well as design best practices to ensure maximum conversions.

So, are you set to design an awesome Thanksgiving Day email and amp up your sales?

If you need any assistance for your holiday email marketing, let Inbox Army be of help.

Multi-ESP Support

We support 50+ ESP vendors. Be it enterprise platforms such as SalesForce and Oracle Marketing Cloud or small and medium sized business platforms such as Mailchimp, Klaviyo, Aweber, and ActiveCampaign, we’ve got you covered.

Many More