The dataset is from GameStream Live, a social app for mobile gamers that lets users watch and interact with live streams and recorded videos of mobile game content.
A common key performance indicator for a mobile application is user retention (i.e., getting users to come back day after day). For a social app like ours, strong user retention is integral to building up a user base and ensuring any investment in user acquisition does not go to waste.
Suppose you are asked to evaluate the effectiveness of a recently conducted 2week A/B test. The hypothesis is that sending an app push notification to a dormant user could revive their interest in the app.
An engineer implements the test as follows:
- Whenever a popular streamer goes live, a check is made for users that have not returned to the app in 3 days, but have push notifications enabled a. Users that were the result of a previous check are excluded to prevent a user from being entered into the experiment twice
- The users are randomly split into two variants: a control and a test variant
- The test group is pushed a notification encouraging them to check out the live stream while the control group is left alone
The engineer responsible for implementing this experiment has provided you with data and you are asked to evaluate the effectiveness of this A/B test.
The CSV file contains data on how many users were assigned to the test and control groups each day and how many then launched the app in each of the subsequent seven days. For example, the Test Launched Day 3 column represents the number of users in the test group that launched the social app 3 days after they were entered into the A/B test. Launched Day 0 means they launched the app the same day they were entered into the test.
Create a mock report with the following:
- Summary of the analysis performed
- How did you evaluate and compare each group’s retention?
- Explain any assumptions that were made in the analysis
- Your conclusion as to whether or not the push notification improved retention for the test group
- Your final recommendation regarding this test
- Should the push notification be rolled out to all users,removed entirely, or is it too early to tell?
- Provide justification for your answer!
- If you had the opportunity to redesign this experiment, what changes would you make?
- What are the most important metrics you would want measured?
We encourage you to write the report as if you were planning to present the results to a product manager who already understands how the A/B test was implemented. Your aim is to explain your analysis in a clear and concise manner and then convince the product manager to go along with your final recommendation. You can use any tools you like to analyze the data, but the report you submit should be a simple document or spreadsheet.
The data you’ll get:
All company names in this question are fictional.