3 Ways You Can A/B Test Your Push Notification Marketing Messages Today

Improve mobile conversions and mobile engagement by testing your push notification marketing messages with our how-to guide.

---

With mobile A/B testing, you can determine the effectiveness of dozens of different tactics to which ones work with your audience.  You can test different app landing page layouts, emoticons, interactive buttons, etc.

But it’s also important to A/B test different variants of the content in your mobile messaging to make sure it’s resonating with your audience and helping you meet and beat your mobile engagement and conversion goals.

In this post, we’re focusing in on A/B testing push notification messaging, but a similar methodology would work for A/B testing in-app messaging, in-app message center content and content in other mobile messaging channels.

Mobile A/B Testing Methodology for Push Notification Content Variants

To set up A/B testing for mobile messaging variants, we recommend the following process, which is based on growth marketing testing principles:  

Step 1: Set a Hypothesis

What type of messaging do you think your audience will be more likely to respond to for the push notification marketing message categories you’ll likely be doing (educational, promotional and transactional)? Should the messaging be serious? Playful? Practical? Emotional?

Step 2: Run the Test

Write push notifications that take different approaches and test them. Airship’s Channels composer allows you to experiment with different variations of a message for a given audience, and provides engagement reporting data for each variant.

Step 3: Document the Results

What worked better? By how much? What was the content of each of the push notifications you tested? Don’t rely on your memory; document results in a spreadsheet so you can focus on the data. Make sure you’re documenting when you started and stopped the A/B test, which group received which variant, what the variants consisted of and what is the impact on your success metric.

Step 4: Take Action

Look at the results carefully. Which variant won, and on which criteria (direct open rates, conversion rates, etc.)? Act on the data and optimize future efforts based on what you’ve learned.  

Three Push Messaging A/B Variants to Test

Here are a few examples of push messaging content options to consider for A/B testing variations:

1) Just the Facts vs. Brand Voice

Do your users prefer pushes with personality or just the facts?

Examples:

A: Just the Facts: “Our new premium cheeseburger is here to stay. Choose #2 on the menu.”

B: Brand Voice: “‘I can has cheezburger?’ Yes, yes you can. See what we’ve got for you.”

2) Clear Call to Action vs. Intrigue

Are your users more likely to engage with a more intriguing, mysterious message or do they prefer a straightforward call to action (CTA)?

Examples:

A: Clear CTA: “You can now enjoy $1 off premium cheeseburgers. Tap to redeem.”

B: Intrigue: “Hungry? We’ve got deals… “

3) Urgency vs. Value

Does urgency drive more engagement than a friendly nudge?

Examples:

A: Urgency: “Your coupon expires today! Tap to redeem!”

B: Value: “You’ve still got time to enjoy $1 off your next premium cheeseburger combo!”

Putting it All Together: Looking at Your Push Messaging A/B Test Results

Let’s say your hypothesis going in was that a push notification with brand voice would get more engagement than one with just the facts. You run the A/B test, look at the report, and find that:

Your fact-forward notification resulted in a 23% direct open rate and a 11.5% conversion rate

Your fun brand voice notification resulted in an 18% direct open rate  and a 13.9% conversion rate

What to make of these results? Well, while the just-the-facts style notification got a higher open rate, it had a lower conversion rate. Consider what this means for your brand and your goals. If conversion is the priority, you have the basis for your next push notification — and your next A/B test.

Don’t Stop There: Advanced Mobile A/B Testing Can Make a Big Impact on Engagement and Conversions

Testing messaging variants is only one facet of testing you can do. You can also test for personalization (individualizing even one aspect of a message can have a major impact on engagement rates), the impact of segmenting your list, location-based push notifications and more.

The key is to follow good growth marketing principles for A/B testing — especially being methodical about how you test, and continuing to apply your learnings, both on mobile and other channels in your stack.

A/B testing is available on all of our paid Channels plans.  Sign up today, or get in touch and we can talk through your goals and your options.