Mobile A/B Testing Adds an “n”

As human beings, we’re naturally wired to try and find the best outcome. A/B testing is an invaluable tool to help achieve this aim. When I first heard about A/B testing several years ago, I was mesmerized by the prospect. Having the ability to get feedback from my target audience on two potential realities at the same time was such a powerful concept. In the e-mail and website world, marketers went on to test all manner of things: headlines, subject lines, offers — you name it. If we could split the traffic and report on results, we would test it.

Mobile messaging is no different. In fact, it’s arguably even more important since our mobile devices are so personal. Irrelevant messages can be the difference between someone keeping or uninstalling your app; notifications must provide value to drive conversion. Our goal is to help you identify what content conveys that value best.

Urban Airship has offered the ability to test two different versions — a classic A/B test — of a notification for quite some time. But we've heard that you want more so, today, we're unveiling A/B/n testing.

A/B/n Testing: Expand Your Possibilities

The “n” in A/B/n allows testing of up to 26 variants at the same time. You may ask, “Do I need to test 26 variants at once?” If you’re new to A/B testing, maybe not immediately; but as time goes on you’ll certainly find yourself in situations where it is valuable to test more than two versions of something. Get a few creative people into a room to discuss messaging and you’ll very quickly come up with a number of likely solutions. Particularly when you consider all the elements you can test:

• Headline

• Message tone

• Calls to action

• Offer

• Interactivity —  e.g. one or two calls to action

• Button text

• Landing pages

• And emojis!

You can even test the impact of sending a message through just push notifications, in-app messaging or both. Plus, we make it easy to copy and tweak additional variants in our user interface. Want to see it in action? Here’s a video that demonstrates how.

Mobile A/B Testing Best Practices

If you haven’t used A/B testing much, all you need to get started is at least two versions of a message. Sure, I just talked about the value of testing more variants and, if you’re an avid tester, testing numerous versions is a great strategy. But if you’re just getting started, we recommend you start with two variants and then build more sophisticated tests as time goes on to really be able to pinpoint what’s working.

Here are a couple of simple best practices to consider as you get started.

#1. Start with segmentation, then optimize with A/B/n testing

The goal of A/B testing is naturally to increase your engagement rates. And if you’re consistently getting double digit engagement rates, A/B testing will definitely help you optimize further.

However, if your messages are performing in the single digits, it’s likely that you haven’t addressed your segmentation strategy. Because mobile is so personal, segmentation is even more important than on other channels. Our research shows that by segmenting on as little as one user attribute, your engagement rates can increase 4-7x. So start with segmentation first and then build your first test to further optimize.

#2. Include a meaningful variation in your test to get results

This may sound elementary, but all too often we see tests with variants that have very small degrees of variation. For example:

Variant A: The time is now. Set up your free trial today.
Variant B: The time is now. Set up your free trial today…

 

Did you notice the difference? If you didn’t, don’t feel bad. The ellipses on the end of Variant B are hard to catch. More to the point, the change isn’t meaningful enough to make a difference. These types of tests rarely bear statistically significant changes.

Instead, we recommend varying your message with a different way to approach the same goal. For example, a media company recently tested these two variants to notify users of a new edition:

Variant A: New edition ready to view!
Variant B: US economic supremacy is fading

In this test, Variant B dramatically outperformed Variant A. The company ran this test multiple times with multiple content headlines and found that the headline outperformed the generic “new edition” messages the majority of times.

#3. Make sure your tests are valid

Once you have your test in place, the next thing to consider is sample size: is it big enough? In general, an audience of greater than 500 is a good start. There are also sample size calculators available online if you want to measure for statistical significance.

You will want to make sure you measure and compare your variant responses to a control group. Urban Airship provides a handy traffic regulator that allows you to easily define how much of your audience you want to hold for your control group, which will help determine if the results are significant and should be actionable. (You can see it at work from our short demo video.)

It’s Easy to Get Started

That’s it. If you haven’t used A/B testing until now, it’s a great tool to wrap into your mobile engagement repertoire. And, if you’re an A/B testing pro, we invite you try out our new A/B/n capabilities to find your best outcomes.
If you’re new to Urban Airship, A/B/n testing is available on all of our paid plans. Get started today or contact your account rep for more information.