How Can You Run More Mobile A/B Testing Experiments?
Imagine you and your team just a released a redesign of the login section of your software product. Shortly after the release you notice that the numbers of newly registered users dropped almost to zero. But why? Is it because the change of the register button to a register link? Or is it the new naming of the register element?
Maybe this kind of change should have been tested before with an A/B test.
What is A/B Testing?
A/B testing in the context of digital software products gives a team the power to test and compare ideas like in a science experiment with real users. Basically everything can be tested: the color of buttons, new text for navigation elements or a complete different version of the product. A/B testing will help teams understand the impact of the changes on a subset of real users to get real insights.
Every A/B test starts with the creation of a hypothesis. A strong hypothesis is needed to describe the A/B test upfront and to explain the expected outcome. You can use the pattern:
Changing__________ from__________to__________will__________.
An example:
“Changing the register button from button to link will increase the sign-up rate.”
This is just a theoretical assumption but it will help you to solve the problem.
Once the hypothesis is created, it’s important to define a test user group. Group A is the control group, which still gets the existing product and group B gets the modified version. If one group is performing better according to the hypothesis, then it should be applied.
How Can You Run More A/B Testing Experiments
So how can you start with A/B testing in your mobile project? First, you need to find the A/B testing vendor you want to use in your apps.
All vendors provide an A/B testing SDK (Software Development Kit) that needs to be added to your application. The integration is a one-time task and should not take too much time.
Once the SDK is installed, you can start to create an A/B test on the mobile app using the dashboard of the A/B testing vendor. Most of them offer a web based dashboard, that can be used to configure the A/B test, as well as select the users for the control and testing groups. From the dashboard, you can see the progress of the A/B test as well as the results once the test has been finished.
However, there are two things you should keep in mind when adding an A/B testing SDK.
- Users need to update the mobile app with the included SDK in order to be part of an A/B test. If users don’t have the right version installed, make sure to target the test group based on the app version.
- Adding the SDK will increase the size of the mobile app. Make sure to update the app to the latest version in case the user doesn’t have enough memory left on the device.
Benefits of A/B Testing
By adding an A/B testing SDK to your mobile app, you gain a number of benefits that are worth increasing the app size as well as the one-time integration effort. The following list includes an excerpt of the benefits you gain:
- You can run multiple A/B tests at the same time
- You can extend the A/B to an A/B/C../Z test
- The findings of an A/B test can result in higher conversion rates
- It will be easier to make decisions based on real customer usage data and statistics
- You gain more insights about customer behavior
5 min