Google Play Experiments Allow More Than 50% Traffic Tests

While Google has traditionally only allowed up to 50% of live Play Store traffic to be testable through its A/B experiments engine, it appears that this limit may be lifting soon. We have confirmed two "in the wild" experiments testing options, including up to 66.6% of traffic and 75% of traffic.

In the feature image, we are running a test for 75% of traffic. Here, Google allows up to 66.6% of traffic to be used for testing.

While running one (50/50) or two (25/25/50) variants in an experiment can more quickly produce results, a 3 variant test reduces traffic for each variant to 16.67%. This means that the subset of traffic that each test receives is very small and can produce large swings in performance day over day as the variation between the different types of user sources sent to each variant can be quite significant.

For instance, consider the fact that people searching different keywords have significantly different search intents and react differently to an app's listing just by virtue of search intent, rather than as a reaction the test being run; or, consider how different search intents interact with the test being run (for better or worse), such as one screenshot catering to one type of searcher vs another.

Other wish list items for Google's A/B test engine include:

1. The ability to run multiple simultaneous tests within one country
2. The ability to run tests within a specific country, rather than by language
3. An ability to auto-apply the winner by selecting a minimum performance threshold 
4. The ability to change confidence level
5. Factoring for purchases
6. A/B testing from a specific URL (e.g. allowing store testing for specific ad campaigns)
7. A category test (not very likely)
8. A/B testing for specific keywords (unlikely)
9. A title test (unlikely)
10. A better way to test the potential impact of keyword ranking from text changes (unlikely)

The Bottom Line: Google Play may soon be offering an ability to increase A/B testing volume as a % of Play Store traffic, which would help experiments to become faster and produce more reliable results.

That's all for now, folks! Be sure to bookmark our blog, sign up to our email newsletter for new post updates and reach out if you're interested in working with us.

Incipia is a mobile app development and marketing agency that builds and markets apps for companies, with a specialty in high-quality, stable app development and keyword-based marketing strategy, such as App Store Optimization and Apple Search Ads. For post topics, feedback or business inquiries please contact us, or send an inquiry to hello@incipia.co.