AB Testing

6 Rules to Follow When A/B Testing Your Email


Every audience is different, period. One incentive to your audience could fall flat, while another soars. That is why it’s critical to test, test and test some more by split testing your content, incentives, time of send and more. The options are exponential on what you can test, but that doesn’t mean you should!

In this blog, we’re going to break down what guidelines you need in place to ensure you’re A/B test yields results that will actually improve your marketing strategy in the long run. To run a test that will yield repeatable, statistically significant results, you need to adapt a systematic approach. Before we breakdown 6 guardrails to put in place when running split tests, let’s breakdown what an A/B test in email actually is!

What is an A/B test?

A/B testing is a simple way to test your current design (A) against changes to your page/email/ad (B) and determine which one produces the most positive results. This technique can be used to make sense of metrics such as sign-ups, downloads, purchases, and so on, to identify which version will increase or maximize an outcome of interest.

A simple tweak in your email campaign or website could significantly increase your bottom line; that’s why testing MUST be your #1 priority. Without further ado, lets look at A/B guidelines to follow when running a test to ensure you are getting the most marketing bang for your buck.

Know Your Baseline Results

You won’t know if the two versions of you’re A/B test yielded positive results if you don’t have a good grasp on your baseline metrics. A baseline result means you already know what your average open, click-through and conversion rates are. For example, If you ran an A/B test and your conversion rate for A was 1.3% and your conversion rate for B was 3.23%, but your baseline conversion rate is 4.5%, the test had a winner, yes, but what you tested will not positively impact your bottom line in the long run. You want to test option A and B against each other, but you also want to know that whichever one does better in the test is also doing better than your current results.

Test ONLY One Element at a Time

This is the foundation for yielding successful (and true) results. For example, don’t test two different layouts and two different CTAs at once. This will mess up your control condition because you’ll have nothing to compare your changes to. If you were to test multiple elements against each other in an A/B test, you wouldn’t be able to identify the element that resulted in more opens, clicks, conversions, etc.

Test Your Elements at the Same Time

When testing, the two variations of the test must be run simultaneously. When each test is run can drastically impact and skew your results. For example, according to MailChimp, the most optimal days to send email campaigns are between Tuesday and Friday, so if one element was tested on Tuesday and one on Friday, it could not be determined if the results were statistically significant. That is because you can’t factor in any variables that might have changed between Tuesday and Friday.

Measure the Data that Matters

Like I mentioned previously, there are many different things you can test like email layout, images, subject lines, CTA, but remember that each of those things is likely to have an effect on different parts of the conversion process.

For example, if you are deciding that you are going to test two different CTA buttons, it wouldn’t make any sense to see which version had the most opens. Instead, if you are testing two buttons, the data you will want to hone in on is CTR. You would, however, look to which email got better open rates if you were testing two different subject lines.

Test Your Whole List, Not Just a Sub-Set

The larger your test sample, the more accurate, repeatable and reliable your results will be! If you are only using a small sub-set of your list, the results may not produce statistically significant results.

Another consideration to keep in mind is to randomly split test. If you identify two groups that have different characteristics/demographics, you cannot guarantee your results aren’t skewed based on the make-up of these groups. Handpicking groups can negatively impact your results. You want to gather empirical data, not biased data, to find out which version of your a/B test lead to better conversions.

Ensure Your Results Are Statistically Significant

If you cannot find the results to be statistically significant, then you’ve got a failed A/B test. It’s shockingly easy to get results that are due to random chance. ”Statistical significance” coincides with another A/B testing term called “confidence level.” The confidence level is the probability that the measured conversion rate differs from the control page conversion rate for reasons other than chance alone.

You should have a confidence level of at least 90-95% before you can determine that your results are statistically significant. For example, if you had a very low response to an email campaign sent out the day before Christmas, you should consider that the holiday might have negatively impacted your open rates. Numbers are important, but you must also be able to analyze the numbers logically to gain a conclusive summary of the results.  Quality trumps quantity any day of the week in A/B testing. If in doubt, run the test again to validate the results!

If you need some help crunching the numbers, grab this A/B calculator to be confident that the changes you made have really improved your conversions.

Don’t Guess, Test!

At Email on Acid, testing is at the core of our mission. After you’ve finished setting up the perfect A/B test for your campaign, ensure the email looks fantastic in EVERY inbox. Because every client renders your HTML differently, it’s critical to test your email across the most popular clients and devices.

Try us free for 7 days and get unlimited access to email, image and spam testing to ensure you get delivered and look good doing it!

Avatar for Alex Ilhan
Author: Alex Ilhan

Hailing all the way from England, Alex brings his email development expertise along with an endless stream of cups of tea and British cynicism. Follow him on Twitter: @omgitsonlyalex.

Avatar for Alex Ilhan
Author: Alex Ilhan

Hailing all the way from England, Alex brings his email development expertise along with an endless stream of cups of tea and British cynicism. Follow him on Twitter: @omgitsonlyalex.

4 thoughts on “6 Rules to Follow When A/B Testing Your Email”

  1. John,
    I’ve seen the GetDataDriven link shared a few times in the email community for calculating significance.

    I’m curious, how do you apply their example of Conversions/Pageviews to email? Are you using the same metric for calculating in email A/B testing? Or substituting clicks/opens/emails sent for the pageviews to measure significance of an email-based test that might be driving to various landing pages on the destination site?

    It seems like a great tool for us to leverage, just curious how others are applying it to email.


  2. Pete, to use the calculator for email the “visitors on page” number could be your email sent volume or, for more accuracy use the received number to pull out any deliverability issues. Then the “Conversion” number could be opens for subject line tests, clicks for CTA tests – or if testing conversion you can use number of transactions resulted from the email.
    Hope that helps!

  3. Good one…!!!

    A/B testing is an important test that every business marketer should follow. Thanks for sharing this.

  4. Thanks, Joel! That does help. I don’t know if there is much out there in the way of benchmarks for significance (of various types of tests…) but I guess that just gets chocked up to the benchmark for the given metric (open, click, conversion rate) that you are attempting to test/optimize.

    I will definitely move forward experimenting with the calculator with your suggestion and see how well that works for us. Thanks!

Comments are closed.