How to Kick Bias’ Butt in Marketing Testing

For email marketers who perform split tests, otherwise known as A/B testing, congratulations. As deep as we are into the age of digital marketing, not many practitioners have the time nor moxie to test campaigns, despite the obvious benefits of improved intelligence and optimized performance.

For the enlightened, we commend you but also issue a caution: leave your biases at the door. You are where you are today because of how well you understand your audience and your intuitive ability to relate to them. However, your intuition might ultimately be derailing your testing efforts.

Intuition powers your expertise, but both are in fact “doubly biased.” To be clear, biases aren’t necessarily bad; they just aren’t always correct or incorrect. On the “correct” end of things, our intuition that a bear is dangerous prevents us from walking up to pet it. As far as “incorrect,” biases can cage you into paradigms of thought that can limit the creative and performance potential of marketing campaigns, such as social ideas (only women do grocery shopping ), scientific theories (AI will never be sophisticated enough to be sold to the general public), and business culture (our brand voice is XYZ, never ABC). The rigidity of intuition can prevent marketers from seeing the larger picture, identifying an obvious answer, or testing outside the box.

Biases are hard to suppress but can be mitigated by being aware that they exist. The best way to account for your inert biases is to first observe, then test outside the box, and finally analyze the outcome.

Use observation to lay out:

  1. What you are testing
  2. What you aren’t testing (there could be something better out there!)
  3. How your audience is responding to your tests

You can make observations by eyeballing your data, but given your propensity for bias, please consider using an unbiased computer-based tactic instead. We have a helpful tool that can find behavioral trends in your data. 

The observation stage offers an opportunity to discover a trend or reveal gaps in knowledge. For instance, you may realize that a good portion of your best performing email subject lines contain personalized elements, like the recipient’s name or location. Alternatively, you may observe that your team frequently tests language that conveys urgency like “don’t miss this” and “time is running out” with poor results. It’s worth noting that this is in line with what Persado has observed across the thousands of tests we’ve run in the US retail industry – urgency consistently appears towards the bottom of our emotional ranking. Perhaps you’ll see encouragement language like “carpe diem” or “it’s super simple” had positive results in a few subject lines, but you haven’t tested enough encouragement language to make any conclusions.

After observation comes testing-outside-the-box, but unfortunately this phase has two opportunities for bias to rear its ugly head.

  • BIAS = not testing enough messages

… yes, that means more than two

It’s often said that you’ll miss 100% of the shots you don’t take. It works exactly the same with marketing. A/B tests, by their very nature, only look at two possibilities, even though there are so many meaningful variations of a single message. Test what you haven’t put into market yet and take those missed shots! Otherwise, the conclusions you draw will inevitably be biased. Unless, of course, you’re psychic.

  • BIAS = not testing messages with variety

To stick with the sports metaphor, let’s say you’re shooting consecutive three-pointers but you’re not sure if that’s a fluke or a talent because you recently bought a new pair sneakers, switched the brand of your energy bars, and moved from an outdoor to an indoor court. So how can you isolate which of these changes are responsible for your newfound athleticism, without inserting your own bias… or self-interest? Is it also possible that each of these factors are responsible in different magnitudes for your baskets? The only thing you can do is hold the other variables constant while you experiment with (i.e change) just one.

  • Experiment 1 → New sneakers; old energy bars; outdoor court
  • Experiment 2 → Old sneakers; new energy bars; outdoor court
  • Experiment 3 → Old sneakers; old energy bars; indoor court

In marketing, it’s equally important to design tests in a similar fashion. Here’s a rough skeleton of what that would look like:

  • Experiment A → [Emotional Tone 1: urgency] [Offer description 1: 40% Off]
  • Experiment B → [Emotional Tone 1: urgency] [Offer description 2: Storewide Sales]
  • Experiment C → [Emotional Tone  2: luck] [Offer description 1: 40% Off]
  • Experiment D → [Emotional Tone  2: luck] [Offer description 2: Storewide Sales]
  • Experiment E → [Emotional Tone  3: curiosity] [Offer description 1: 40% Off]
  • Experiment F → [Emotional Tone  3: curiosity] [Offer description 2: Storewide Sales]

This blog post does a great job of elaborating on the above experiment process.

Now that you’ve observed and tested a variety messages, it’s time to analyze the results. Without this final step, a data tree falls in the forest but there are no marketers around to hear the sound. In other words, an experiment is only as good as the informed actions you take because of it. Did you consistently see one offer description perform better than another?  Which emotional tone worked best? Keep track of this data in a swipe file so you can revisit and iterate off of what you are seeing bring the most lifts.

Without knowing why a message works, marketers only scratch the surface of a campaign’s potential. Making assumptions based on your bias to fill in the gaps is a costly missed opportunity for brands to better understand and connect with their customers. Taking a scientific approach helps to eliminate bias and provides complete clarity to the human eyes, brain, and heart.

Ready to kick bias in the butt so that you can really get to know your audience? Persado leverages machine learning to create a bias-free feedback loop between your marketing message and audience response. Hit us up at to find out more about how our award-winning platform works.

Katherine Dessenon heads Persado’s campaign management team, meaning she regularly leads intense negotiations between the humans and the robots who generate the content for brand customers. More about Katherine in this employee spotlight.