How to do Research for Your App Store Optimization A/B Tests

If you think that simply involves changing the color of your screenshot background or the order of screenshots is good enough for A/B testing, you need to read this post.

While those tests may yield a slightly improved conversion rate, the best teams don’t A/B test random changes and call it a day.

Rather, they use research to generate a strong reasoning or a hypothesis that supports why the new test variant is likely to perform better, and create variations based on that supporting logic, setting themselves up for far more success than failure, and to inform the next test even when they do fail.

Read this post for some actionable tips on how to better target your A/B tests for successful outcomes.

Before diving into the suggestions, it's important to realize that ASO is a zero-sum game where users are comparing your app to several other competing apps, where only one app will win the user’s install at a time (and ultimately the user’s end-loyalty).

For this reason, it’s important not to create A/B variants in isolation; form your hypothesis and create variants after researching how your app can be best positioned relative your competition in order to ensure that your app "shows up" well in the zero sum App Store environment.

Conduct Research to Support Your A/B Tests

Consider a few of the following activities to help inform your A/B tests and generate clear hypotheses that can better inform your creative process.

• Compare your screenshot, feature graphic, video or icon design with the styles of other apps that appear near yours in keyword search results. Focus on the fact that your app will need to capture user attention better than these apps in order to earn that user’s initial tap and/or install.
  • Moreover, consider only the keywords that have the highest volume and for which your app ranks best for (i.e. rank 1-10), as these keywords will have the highest contribution margin to your organic downloads. If your app ranks well in the top charts, research your closest top chart peers.

• Compare the messaging found in your captions/videos and descriptions against that of your competitors, and brainstorm on messaging that gives your app a competitive advantage. What claims can you make that your competitors cannot? What advantages do they possess that you need to overcome in users' eyes, in order to convince them to download your app?
• Read your user reviews to determine the language and key features that users care about and integrate these into your messaging, rather than using other language/selling points that do not resonate with your users (and as such are considered dead weight text).
  • For best results, focus on those reviews that are marked most useful, or the most recent reviews.
• Review your app’s usage data to figure out which features or configurations are the most-used or most-purchased, and use these insights to increase your app listing’s appeal to new users.

 

Here are two examples of clients for whom we used research-informed A/B testing to drive better results:

 

With photo lock app, Keepsafe we based our research on: A) the most popular keywords that Keepsafe ranked for and B) messaging that could position Keepsafe into a stronger stance relative the competition. We began running experiments to:

1. Explore the messaging that most resonated with users (e.g. Keepsafe has 15+ features, Keepsafe has been downloaded by 50 million people, Keepsafe has been used by millions to store over 1 billion photos...).
2. The best visual UI to present that messaging in, in the screenshot (e.g. a circular call-out, a square banner, a dog-ear call-out...)
3. The color which meshed best with the screenshot design and which users most preferred (e.g. orange, blue, purple).

We discovered through A/B testing (which expanded to localized versions in 6 non-USA countries) a winning variant that is still in partial use by the Keepsafe team today, several months later. The winning messaging was that millions of people had saved over a billion photos to Keepsafe, and the blue banner was the best visual call-out.

Source: Keepsafe Google Play live listing.

For local marketplace app, 5miles we based our A/B testing research on a few different research insights:

1. We realized that in 5miles’ Android listing, the traditional caption style of one or two sentences on top of the screenshots underperformed screenshots without any captions at all; but we also saw that one competitor had made use of a caption style that blended captions into the screenshots themselves.
2. We also saw strong trends in products that were listed very frequently and hypothesized that the products shown in the screenshots would have a strong impact on performance if linked to products most popular in the marketplace.
3. We also drew the conclusion based on competitive research that people in buy/sell mode wanted captions focused on features that helped them to buy or sell faster/easier, rather than more generic branding or social proof.

With this information, we set up a set of progressive A/B tests in the Google Play Experiments engine to test several new caption-styles, new caption text, and focused photos on several different popular products being sold on 5miles. The result was a brand-new screenshot style that was not only successful on Android, but also iOS!

Source: 5miles Google Play listing.

 

The next time you create an A/B test, think about how you can use marketplace or back end data to inform your process. Do your research due diligence to ensure that your app uses the right language, positions well against your competition’s messaging, and uses visuals that capture attention better than your competition.

That's all for now, folks! Be sure to bookmark our blog, sign up to our email newsletter for new post updates and reach out if you're interested in working with us.

Incipia is a mobile app development and marketing agency that builds and markets apps for companies, with a specialty in high-quality, stable app development and keyword-based marketing strategy, such as App Store Optimization and Apple Search Ads. For post topics, feedback or business inquiries please contact us, or send an inquiry to hello@incipia.co.