Side Project Time Hacks: Automating Android App Performance Reports

 

Introduction:

The most recent post in my series on Side Project Time Hacks was about how I automated my Android apps’ store listing descriptions with a small program written in an afternoon. While I’ve already saved several hours with that tool since I finished it, I felt that the following side project time hack project needed to be something of larger scale in order to better demonstrate the breadth of what you can create for your own “side project time hacks”.

What I ended up choosing was to write a tool to automate all of my metrics reporting for my Android projects so that I never have to manually collect and manage the data by hand ever again – saving me hours upon hours every single week.

In this post, I’d like to talk about not only the motivations for making this tool, but also go into detail on how it works so that you can make your own.

 

 

Why is this Important:

Keeping performance reports for your projects is one of the most important aspects of maintaining a successful mobile app.

By tracking installs, active users, ad impressions, ad revenue, reviews, and other important metrics over time, you can gain valuable insight as to what’s working/not working with your applications. Perhaps since implementing a new feature graphic two months ago, your installs have increased by 15%. Or maybe since adding a periodic review prompt feature to your app, you’ve gained an extra 10 reviews per month over the previous average.

The analysis of this information can also help you form future plans as to which areas of your applications need improvement or which tasks you might want to prioritize over others. If your app is getting a lot of installs but not retaining many of those users, then it may be safe to conclude that your promotion/marketing is good but some improvements are needed to the app itself. Switched around if you maintain a lot of your users but you don’t seem to get very many downloads per month, you might want to consider focusing on your ASO / marketing.

Simply put, you want to make sure that you keep detailed records of your apps’ performance so that you can periodically evaluate them and create future game plans.

 

 

The Problem:

The main problem with how I previously collected and stored my own app performance reports was that I did it by hand and organized it all in a Google Drive spreadsheet.

There’s nothing at all wrong with using Google Drive to organize yourself because it’s an absolutely amazing and powerful tool, but in my specific case I had issues with both the tedious workflow I had developed to maintain my records by hand and also with how the records themselves were organized in the spreadsheet.

I used to maintain a new sheet for each month and an additional sheet for each Quarter:

 

 

This way, I could flip through all of my metrics over time such as active users, total installs, reviews, etc and compare them to the previous months/quarters (see screenshot below – the previous month’s values are grayed out so I could see them at a glance).

 

 

All sorts of problems came up with this spreadsheet over time. At the start of every new month and quarter, I’d have to manually set up a new sheet and copy/paste all the current values into it. In the “start active” and “start total” columns for example, I’d copy over the ending values of the previous month so I could see growth as the current month progressed – which was messy and time consuming.

Another problem would always come up when I’d go to publish a new app to my portfolio (and thus need to add it to my tracking spreadsheet). I’d have to add a new row for that app, but then that would often mean changing the cell positions of the existing apps – which could mess up any of my references if I ever wanted to access these values in another sheet to create some graphs.

Lastly by far the biggest problem was the sheer overhead of manually going into the Google Play Developer Dashboard, Firebase Console, Admob Dashboard, and Adsense Dashboard to copy and paste the new values presented there one by one, cell by cell, into my spreadsheet for each app. This became absolutely unsustainable as my portfolio grew because each new app I created would add several minutes to the time it would take for me to update the spreadsheet.

If I had to estimate, I would spend roughly 30-45 minutes updating my tracking spreadsheet by hand every time. Admittedly, I’d do this several times per week – totaling to at least 2-4 hours of my time per week just managing a simple spreadsheet.

 

 

The Solution:

The solution to my problem is fairly straightforward – I need to be able to press a button and have all of the historical data for my published Android apps be programmatically retrieved, organized, and exported into a spreadsheet to act as my new app performance report over time.

 

 

While I was at it, I decided to adjust my spreadsheet layout from the original version. Rather than having a new sheet for each month/quarter, I thought it best to have a sheet for each individual app:

 

 

This makes adding new apps to the document extremely simple and also makes my cell positions consistent across every app. This means that the data for say July 2017 for one app is in the exact same cell position as any app in another sheet – so it makes creating summary sheets to view that data a breeze and this was simply not maintainable with my previous spreadsheet layout.

 

 

I also put in a bit more effort with the formatting this time around. If you’ll see the column headings across the top, columns with data that comes from Google Play are highlighted in blue. The columns that come from Admob/Adsense data are highlighted in red. Lastly, each year is a different color, and each quarter is also a different color so I can still easily compare data at a glance through different units of time than just months.

 

 

How it Works:

 

The main logic of the tool as explained earlier is captured in this above method. The main data structure I chose for containing the app historical data is a Map where each key is a custom “App” object. It’s a simple plain old java object that contains any relevant information I might need (see below).

 

 

The values in the resulting Map are Lists of custom “MonthlyEntry” objects. Each “MonthlyEntry” is roughly analogous to a row in the resulting exported spreadsheet and basically represents a month’s worth of data that I want to track – clicks, impressions, installs, etc. The full list that gets returned is all of the App’s historical data over time.

 

 

 

Step 1: Load App List to Process

Step 1 of the tool (loading the App list) is pretty simple. I store the information about each App I want to load in a configuration CSV file, and just read from it to create the list:

 

 

Here is the non-default App constructor which allows me to create a new App object in the above method. All I do is pass in a line from the CSV, which gets split and the appropriate pieces parsed out:

 

Step 2: Fetching / Processing Data

Step 2 (processing and fetching all of the data for each app) is where the bulk of the tool’s code lies. At a high level, I need to pick a date to start my reports from and loop through each month after the next until we reach present day. For each month, I need to build a MonthlyEntry object containing all of that month’s data, which gets added to the resulting List. This method gets called for each App that I need to process:

 

 

If you’ll notice in the screenshot above, I have split the data fetching into 3 different classes. I have one class to fetch the Install metrics, one to fetch the Review metrics, and lastly one class to fetch the Admob metrics. This is simply because the requests and data that gets returned from each are all of slightly different format, so I wanted to keep them separated – however all 3 share as much code as possible between them to fully exploit the benefits of using an OOP language like Java.

 

Retrieving Google Play Metrics

Every day, Google Play will add new report data to monthly CSV files for each app. These monthly CSV files are stored in a private Bucket in Google Cloud Storage under the same account that owns your Google Play Developer account. All you need to do in order to fetch that data is implement the Google Cloud API and then specify the bucket object name (name of the CSV file) that you want to fetch. Then it’s just a matter of parsing the data out of the file that you’ve fetched and formatting it into a usable structure.

The other option you can choose instead of implementing the Google Cloud API is to use the command line Python tool called gsutil. It seems like the easiest option if all you want to do is download your reports, but because I wanted my tool to do a little bit beyond that I just chose to use the Cloud API instead.

Google has provided a good bit of documentation on this whole process and they can almost certainly explain things much better than I can so here is a list of good resources I used when building this project:

• Fetching Google Play Reports
• Google API Client Java Library
• Google Cloud Storage Docs

 

Retrieving Google Adsense / Admob Metrics

The data that comes from Adsense is fetched using the Adsense Management API which you can import with Maven or Gradle. The process for fetching this data is also fairly simple. All you need to do is activate the API under your Google Developer account, authenticate your tool, and then begin making requests.

Here is a list of some good resources I used when Implementing the Adsense metrics piece:

• Admob Reporting Guide
• Adsense Management API Reference
• Adsense Management API Libraries

 

A Quick Note on Caching

I think if I were to go into any more depth on how specifically each of these 3 classes work (and all the details of implementing the two different APIs), I would need to roughly double this already lengthy blog post so I will save that for another day. However one thing that I would like to point out about this piece before proceeding to the export code is the extra layer I wrote which caches the historical CSV data locally.

With nearly twenty apps to process and roughly two dozen months of historical data to fetch per app (which only increases as time goes on), fetching the data from the cloud every single time seemed impractical, slow, and a complete waste of resources. This is especially true when you consider that metrics from previous months won’t change, so there’s no need to download a new version of the file when we know it will be the exact same.

So what I implemented is a system that will save any CSV file that’s older than last month (if it’s not already saved). This way for any MonthlyEntry I go to build that is older than last month, I will first check locally to see if I’ve saved the CSV file before. If I have, I use the local version. If not, I will fetch the file from the cloud, but then save it so I can re-use it later. I don’t cache anything newer than last month because that data still might get changed / settled as the month goes on.

 

Step 3: Excel Export

The Excel export piece was really easy to do using the Apache POI library. All you need to do to build an Excel spreadsheet programatically with this library is create a new Workbook object, create one or more Sheets, and write your data to them. It even allows you to specify cell formatting – which was really important to me with this project.

Here is a snapshot of how I handle the export process:

 

 

Next Steps / Improvements:

If I could add just one thing to make this side project time hack even better for my use case, it would be to integrate it with Google Drive so it can automatically write to my spreadsheet in the cloud as well as exporting a local Excel version.

I think another major way to improve this project would be to add multi threading. I could implement this in a few different ways, but one of the main places it sticks out to me is to process each month on a separate thread. The results of fetching the data for one month is independent of fetching and processing the data for any other month, so it would really speed things up. Alternatively for each month I go to process, I could process the Admob data on one thread, and the Google Play data on another – which would also likely give some significant performance benefits.

While I don’t think either of these would be particularly difficult to do, I’m going to add them to my backlog and perhaps re-visit them in the future as an extension of this blog post. If that’s something you’d like to see, leave a comment down below or contact me directly and let me know.

 

Wrap-Up:

As I mentioned in the beginning of this blog post, I used to spend hours per week maintaining my old metrics spreadsheet. It was an unorganized and completely unsustainable process which I’m glad to be rid of.

Now that this Play Report Tool is written, I’ve reclaimed all of that time for myself and can put it towards more useful things like new apps and projects – which is the whole point of this blog post series. All it takes is to identify a problem that’s costing you time, think about the solution at a high level and what it needs to do, break it down into pieces, and then implement it.

If you have created your own side project time hacks, I’d love to hear about what you’ve built. Leave a comment on the post down below, fill out my contact form, or reach out to me on social media and tell me about it. If you’re interested in more of my side project time hacks, I keep a running list in my introductory post here, or you can view the whole category here.  I also semi-frequently post more content, so stay tuned for more.

 

Thank you for reading!

 

 

 

New To Java or Programming?

Here are the top 5 books I recommend to help you get started. Note – these links are affiliated, so if you buy something I get a small kickback of a few cents that helps me afford some coffee here and there. Thanks for the support!

1. “Effective Java” by Joshua Bloch

Joshua Bloch is a contributor to several major Java classes / APIs including the java.lang and Java Collection framework, so he is just about as reputable as it gets in terms of someone to learn Java from. The book goes into detail about many best-practices, which are all extremely insightful and things I wish I had learned sooner.

 

 

2. “Test Driven” by Lasse Koskela

If there’s one thing that slows down or inhibits project development more than anything else, it’s the result of poor testing. Learning the ins-and-outs of Java (or any programming language for that matter) is only part of the equation – the rest is how you use it, and how you test it. This is the best resource I’ve found to learn test driven development – which you’ll find all over the enterprise world. I highly recommend you give this a read.

 

3. “Java Concurrency in Practice” by Brian Goetz, Tim Peierls, Joshua Bloch, Joseph Bowbeer, David Holmes, and Doug Lea

Another extremely important programming skill once you start to get into some bigger projects is concurrency. Essentially, concurrency is a concept that allows you to run different pieces of code at the same time – which can cause for some scalable performance gains or is useful for any network functions you might need to use. If you aren’t careful however, it can cause some very serious problems with race conditions – so if you’re unfamiliar with the concept in Java, I think this is about the best resource you can find on the subject.

 

4. “Clean Code” by Robert C. Martin

If you’ve ever worked on a large team project, you may know what a struggle it can be to read someone else’s code. Everyone has their own style, habits, and naming conventions. Even how Java should be spaced and tabbed is hotly debated. “Clean Code” will help you learn the best practices to write clean, readable code. Trust me, your future project teams will thank you for it.

 

 

5. “Elements of Programming Interviews in Java: The Insider’s Guide” by Adnan Aziz, Tsung-Hsien Lee, and Amit Prakash

This is one of my go-to coding interview prep resources (alongside of course the ever-popular “Cracking the Coding Interview” by Gayle McDowell). The book will present you with challenging problems and puzzles (similar to what you might come across in a coding interview) and will show you how to think critically to come up with efficient algorithms for your solutions.