Multivariate Testing of Native Mobile Applications


C. Holzmann, P. Hutflesz - Multivariate Testing of Native Mobile Applications - Proceedings of the 12th International Conference on Advances in Mobile Computing and Multimedia (MoMM 2014), Kaohsiung, Taiwan, China, 2014


A/B testing has a long history in web development and is used on a daily basis by many companies. Although it is a common test method for web pages, it is hardly used for native mobile applications. The reason seems to be that it is much more difficult to change the user interface of a mobile application, which has been downloaded from an app store, than that of a web page which is fetched from a server by request. In this paper, we present an approach for A/B testing of native mobile applications. Furthermore, it allows for the more flexible multivariate testing, which is based on the same mechanisms as A/B testing, but compares a much higher number of variants by combining variations for different sections of the user interface. Our proposed approach works without redeployment of the mobile application in the app store and thus allows for a seamless integration into the developer's workflow with low effort for creating and deploying new variants. We implemented a prototype solution for the Android platform and compared it against other A/B testing products. It shows that our solution requires less effort and is more convenient to use than related products. Moreover, it is the only one which allows for multivariate testing of native mobile applications.