What is Holdout Group Testing?

Holdout testing is the process of reviewing your email marketing program to quantify if the campaigns being sent are generating increased engagement/conversions vs not sending anything. Are these campaigns/automation really giving an incremental increase in engagement/conversions/revenue that would otherwise not be realized.

This is like an A/B test, as it has two segments, but one segment receives no emails.  They are excluded from the mailings.  You then measure the purchase/conversion behavior of the two segments to define the level of increased engagement that the segment being emailed has over the segment being excluded.  In most cases (one would hope), the group receiving email will have a higher conversion rate.

More to think about:

There are many costs to factor in after you get the results.  You need to weigh the cost of the email program you are testing.  Cost of the ESP to send the campaigns, cost to design, code, test, and setup/deploy the campaigns, count your manhours.  Use that as part of the cost of doing campaigns.  A small increase in revenue may be outweighed by the costs of that program.

Depending on your list size, type of campaign/automation and mailing frequency, this type of test may take 30-90 days to complete.

Why you should do this?

The main reason for running holdout tests is to understand if you are cannibalizing natural sales that would have happened if no marketing email were sent. For an e-Commerce company that regularly sends discounts and/or offers to its customers, this process enables you to determine if those discounts/offers are in some way cannibalizing the profits of your business. The worst-case scenario for the company would be the segment of customers who did not receive an email end up purchasing your products at the same or higher rate. That essentially means you’re giving away your profit margin when it’s not needed.

Let’s use a Cart Abandonment program as an example:

If you are doing testing, running a cart abandonment program or really any email marketing you should have a holdout group.  A holdout group is a group that does not get that specific type of mailing.  That’s your baseline of “what if we didn’t send them anything?  What is there natural progression through the buyer journey if not prodded by email.

For example: If you run a cart abandonment program, you should first know what your natural cart abandonment return rate is with no program running.  In other words, how many people come back on their own, with no messaging to buy what’s in their cart.  How many come back the first day, the second day, the third and so on.

Once you know your natural return rate you can then measure the true value of a program.  If your natural return (conversion) rate is 5% and when running your program, you see a total of an 11% return rate, then you know the program generated an additional 6% return.  The program didn’t create 15% because you now know your natural return is 6%.  Those percentages will change month-over-month so reviewing your programs regularly is imperative.

Now you can measure true revenue (to a point, as there will always be variables out of your control that will affect results) from that program as well.  If over a 12-month period the natural returns delivered an average of $10,000 per month, and the Program total delivered $22,000 per month, you know the programs value over the baseline natural return.  These numbers will vary over time with some peaks and valleys and will be influenced by sales and holidays, so adjust as needed.

Wrapping this up:

Some companies hold out 10% of subscribers (sometimes using a dynamic segment of subscribers for each test process) as a complete blackout during testing or all year.  If you have a very effective and robust email marketing program, it seems silly to ignore the boost of revenue to that 10% group, so I suggest doing your testing a few months each year but shift the test months each year.  If you are an e-commerce company and do heavy volume/revenue during the holidays, don’t test then, grab all the revenue you can during the holidays.  Test throughout the rest of the year.

It should be noted that if you are going to test or do a holdout group, your segment or list size should be at very least 50,000 subscribers.  Anything less would be statistically irrelevant.  Also, make sure your holdout group matches the sending group.  If you are sending to actives, the hold group should be actives.  Don’t use a random segment process, random tends to be anything but random.  Try to give fair advantage to each group to succeed/fail equally.

Thanks for stopping by our blog and if you’ve found this valuable, please share it on your social channels.

———————————————-

If you need help with this type of testing or any other aspect of your email marketing program, we offer a free 30-minute phone consultation.  No hard sell just a call to see if we might be a fit for your needs.  Contact us here.

The abandoned-cart reminder is one of the most successful emails you can send to your customers. But you need to set them up the right way to encourage more shoppers to act on them if you want to get the best results.

First: Why you should try to bring customers back

Consider these statistics:

Cart abandonment happens all the time, especially around holidays when traffic from both first-time shopping and loyal customers increases to your site. Cart abandonment rates range from 58% to 81% overall, although abandonment is higher in some sectors than others.

More than 8 in 10 carts are abandoned on finance, travel and nonprofit websites, while retail and ecommerce cart abandonment is holding steady at around 77%.

Cart-reminder emails can generate some of the best engagement of any triggered or broadcast email you send. Here’s how they stack up:

  • 45% average open rate
  • 21% average click-through rate
  • 7% average conversion rate

They help you recover sales. Some people leave items in carts on purpose and plan to come back later. But, one of the main reasons people abandon carts permanently is because they had problems at checkout. One company estimated that a large ecommerce site could see a 35.3% hike in recovered sales just by redesigning their checkout process.

They give you a competitive advantage. The 2018 Email Marketing Census from the UK Direct Marketing Association shows only 29% of all email marketers send abandonment emails. So, the field is still wide open for marketers who can take that extra step.

How cart reminders can fail

As these statistics show, the decision to send an abandoned-cart reminder is a no-brainer. But just sending an email by itself won’t guarantee that your customers will come back to finish checking out.

This is the point where many marketers shoot themselves in the foot with their reminder emails. How you structure your reminder email is just as important, if not more so, than the decision to set up an abandonment program.

And here’s another key point: You’ll probably need more than one set of rules governing your reminder email’s timing and content to accommodate the different reasons customers come to your site.

Different reminder models, different purposes. While it’s true that sending one reminder email shortly after your customer abandons the site is better than sending none at all, it’s also true that there’s no exact formula for how you should structure your reminders.

The conventional wisdom around reminder emails is that you generally need more than one, that you should send your first email soon after the customer abandons the site and that you should follow up with one or two more emails within 24 to 48 hours after the first reminder email goes out.

Take a look at the series of two cart abandonment emails by ASICS.

That format, which could take as long as three days to a week to deliver all your emails in the abandonment series, works fine in uncomplicated abandonment scenarios.

Now, suppose you’re running a one-day flash sale. That standard three-message email sequence isn’t going to work beyond the first email, assuming you send it out before the sale ends.

In fact, you’ll end up aggravating your customers because they won’t be able to buy their items at the sale price if they return after the sale is over. If you compound that error by including the flash-sale price in the email message instead of the price they’ll encounter when they return, you might as well kiss those customers goodbye forever.

Solution No. 1: Add more rules

Every successful cart-abandonment reminder program is driven by rules that govern when to launch reminder emails and what content to include in them. Here are several popular ones:

  • Send the first email immediately after abandonment. Then, time follow-up emails according to activity, or lack of it, on the first email.
  • Withhold an incentive to encourage returning to the cart until the last email of the series.
  • Vary the incentive according to customer value (higher value to first-time visitors or loyalty-club members).
  • No incentive or lower-value offer for customers who have used an abandoned-cart incentive in the last 30 to 90 days.

Adding rules to your email program can help you create a more flexible program that accounts for abnormalities or anomalies in the abandonment procedure.

Take that flash-sale scenario we mentioned earlier. Let’s say you want to clear out some excess inventory before you head into the fourth-quarter holiday season. Or, you want to get some incremental sales on what normally would be a slow sales day, like a Sunday, and you offer some eye-popping discounts to attract browsing.

Here’s an example by Huckberry in which they have offered a coupon code for free product shipping.

The trade-off for those big discounts is a short shopping window. So, to bring back your abandoners, change up your rules for that campaign to launch your abandonment email sooner, deliver your series in shorter intervals and schedule the last email to go out before the sale ends.

Solution No. 2: Extend redemption time with a post-cart-abandonment program

Use this to retain high-value customers (they buy more often and/or at full price) or customers who rack up higher-priced carts. You can create a rule to trigger this program based on cart value or customer data.

The effect is that you will extend the time your abandonment program runs, giving your customers more time to come back and check. This is especially handy for high-value carts because those customers might need more time to decide whether to go ahead.

Giving them a few extra days to check their finances, talk to a spouse or partner or wait for the next paycheck could mean the difference between a purchase and a lost cause. Your emails continue to arrive at carefully spaced intervals, giving you opportunities to answer questions or head off objections, but you aren’t pestering them constantly.

Use the time to vary your content

That brings up another point about successful emails. It’s not just how you time your messages but what you put in your messages that can bring customers back.

People abandon carts for many reasons, and not just because they wandered away or objected to your shipping fees. Here’s a sample of reasons:

  • They got interrupted and had to break off their sessions.
  • They ran into problems in your checkout process.
  • They need time to think or are shopping your competitors.
  • They’re using their carts like wish lists and will return later to sort through items.

Your email copy should reflect these potential scenarios, too. Adopting a helpful, service-oriented tone and adding links to customer service, FAQs, buying guides or other relevant destinations on your website can overcome objections and add trust.

But you can also use devices like countdown timers to gently prod your customers into acting, especially if their merchandise is part of a flash or limited-time sale. We have used this device in many client abandoned-cart emails, and we always see a measurable uptick in conversions, often when tested against a standard message with no timer.

FARFETCH sends a nice cart abandonment email with a countdown timer that evokes the fear of missing out and entices the user to complete the purchase.

Don’t forget to test!

This is vital, no matter which format you choose to create for your abandonment models. Test everything to see what works for you and your customers! Here’s a quick list:

  • Subject line
  • Number of emails
  • Timing (first email and intervals between emails for a multi-email process)
  • Incentives (what kind of incentive, like a percentage discount, cash discount, free shipping, upgraded shipping and at what point in your series to introduce it)
  • Personalization (name, product info, link to cart, etc.) versus one-size-fits-all

Call on us to help you set up a great program

Whether you want to investigate launching your own abandoned-cart program or need to get better results from your present program, our strategy experts are eager to show you what’s possible.

We’re happy to offer you a free 30-minute consultation call to see whether we’re a good fit to help you recover more sales and revenue by testing the solutions we’ve offered here or whether you might try other tested tactics in our arsenal.

Just call us at 800-ARMY-253 (800-276-9253) or leave a message on our website. We’ve seen the results that well-thought-out abandonment programs can produce, and we’d love to share them with you!

Multi-ESP Support

We support 50+ ESP vendors. Be it enterprise platforms such as SalesForce and Oracle Marketing Cloud or small and medium sized business platforms such as Mailchimp, Klaviyo, Aweber, and ActiveCampaign, we’ve got you covered.

Many More

Top