GitUpdated – Automating My Git Repository Maintenance with Python

 

GitUpdated: A command line tool for easily managing and updating multiple git repos. Part of my Side Project Time Hack series.

I’ve accumulated quite the menagerie of Git repositories throughout the years and recently I’ve found myself doing more work on-the-go. Nowadays I am frequently switching between a laptop when traveling and a desktop PC or home server when home.

It became a nuisance to work on these codebases from each of my different environments. For example, if I refactor a few files in a project from my laptop while waiting on a train, I might not be able to push anything until I get Wi-Fi access later. Usually this just leads to me forgetting to push the changes altogether, and the next time I resume working on that project from a different device, I’ll be working without those changes.

Even if I did push those changes, I’d have to remember to pull them down when working on the same project on a different machine – and I often don’t. If I were to pick the project back up on my desktop environment a few weeks later, I may have forgotten about my laptop changes altogether and resume working without them.

It’s also a productivity issue when I don’t work on a project from my laptop for a while, forget to keep it synced with master, and then the next time I’m on an airplane without Wi-Fi and decide I want to work on it, I’m not able to because I wouldn’t have the latest version.

All of these pain points hold true not just for my individual side projects, but also for larger team projects that receive multiple new merges to the master branch every day.

Merge conflicts and stale code problems aside, it made me mildly anxious to know that my repositories may not always be synced across my machines.

I decided to write a simple command line tool that loops through all of my repositories in a directory and shows me at a glance everything I need to know about the status of those repositories. This way, no changes ever get left behind.

For example – what’s the repository name? How many branches do I have in this environment, and what are they? How many commits ahead/behind the default branch are each of those local branches? Do I have any untracked or changed files that need to be added? Can I have the tool try to automatically fast forward any lagging branches so I don’t need to do it manually?

I realized this is the perfect opportunity for another Side Project Time Hack – a programming project you hack together that can automate a part of your life somehow or save you time throughout your day.

If this is the first time you’ve come across my blog, I have an entire series dedicated to the concept of Side Project Time Hacks. Some examples of previous SPTH projects that I’ve done include a “Google Play Metrics Report Generator” which automates how I keep track of the metrics for my Android apps over time, and an automated process for updating my store listing text for a my primary apps. You can check out a running list of all the posts in the series here.

 

---

 

The Solution – Overview + Demo:

The solution: GitUpdated, a Python script that addresses all of the above (and then some).

A quick disclaimer – I’ve only tested and used GitUpdated with my repositories hosted on GitHub and Gitlab. Your results may vary if your repositories are hosted on something like BitBucket or SourceForge.

GitUpdated also assumes you have some sort of root directory for your projects, and inside that directory are any number of subdirectories – each a different local Git repository.

GitUpdated basically assumes your file structure looks something like this:

So if you run it with a path like:

/…/user/projects/Android/

GitUpdated will use that as a root folder and then loop over each repository inside – in this case Repository 1 through Repository 3. It calls the appropriate Git commands for each repository to get all the relevant information about their status using a really handy library called GitPython.

GitUpdated’s first step is to determine what the root directory is. I’ve implemented a handful of different ways to get this information. The first option is to specify the path as a command line argument. So if you were to run:

python3 main.py /user/projects/Android/

Then GitUpdated will automatically assume that you want to use /user/projects/Android as your root path. If you don’t specify a command line argument for the path, then it will prompt you for one when the script first starts:

GitUpdated offers a few pre-defined paths to choose from. As you can see from this screenshot, I’ve added a few options that I use most often. You can either enter the number corresponding to your choice of predefined paths, or you enter the fully qualified file path instead if it’s not among the options.

After GitUpdated has a path to work with, it loops through each repository at that location and provides a nice printout of all the information I’m looking for.

The main information I want to know about each project are:

• Repository name
• Folder location
• # local branches, and for each branch:
  • # commits ahead is it from the default branch
  • # commits behind is it from the default branch
  • Branch name
  • Whether the branch is currently checked out
• Repository status can be either:
  • “Clean” (no changed files, no new files)
  • “Dirty”- and if so:
    • # modified files (+ names)
    • # untracked files (+ names)

All of that information is neatly displayed in the terminal window:

This is a snapshot of the GitUpdated output when it analyzed one of my Android projects – CryptoConvert.

It’s showing me that the v3.62 branch is currently checked out and active. It’s also +2 commits ahead of the default branch. If a branch is ever one or more commits behind, it will be indicated as the left number in a red font. I also have a local version of the master branch, but in this example it’s completely synced with the remote version. Also you can see I have a single modified file that I forgot to commit before I stopped working last – activity_main.xml.

GitUpdated’s final requirement is to be able to automatically fast forward all of the branches in each repository (if possible). This way if GitUpdated does show a repository that I need to sync up, I can automatically update it all in one fell swoop.

It’s a relatively simple project, but it’s saved me a lot of time and headache – and that’s what these kinds of projects are all about.

Without further adieu, here’s a brief demo of GitUpdated running for my Android projects folder – which currently just has 2 projects:

You might have noticed this in the demo gif, but I’ve added the full command to run GitUpdated as an alias to my bash_profile for convenience. All I have to do is type gitupdated into the terminal from any directory and the script will run.

 

---

 

GitUpdated Part 2: Code Walkthrough

For the code walkthrough, my goal is to cover enough of the components that you could replicate the script and adapt it for your own needs. If you want to walk line by line through the project, you can always check it out for yourself in the repository here:

https://github.com/ricemitc/Public-GitUpdated

If you have any suggestions on things I can improve on or refactor, feel free to email me or leave a comment. I’d be happy to get to take a look, but I’m not the most active about checking notifications on any platform so no guarantees.

Let’s jump right into it. GitUpdated’s first task is to figure out which path to use as the root directory.

The first option is to simply check to see if a path has been passed in as an argument – in which case use that. Otherwise it will have to present a prompt asking for a path to use before it can proceed.

As mentioned above, I liked the idea of providing some pre-defined paths that I can choose from. Very rarely will I change any paths for my development environments, and if I ever do I can just update this file later.

If you were to build this project, this is where you would configure your own pre-selectable environments and paths. I should’ve implemented this to read from a configuration file, but I purposely use the word “hack” in this series title. So you get what you pay for.

Once GitUpdated eventually receives a path to use, it can proceed with its execution. The root task we need to accomplish is to loop over each directory in this root folder location, assume it’s a repository, and then try to fetch all of the information we need to know about it.

Thankfully Python makes it pretty easy to loop over directories in a location:

folderList = os.listdir(rootLocation)

But of course we wrap that in a few “try” blocks to catch any exceptions:

I delegated all of the output printing for each repository to a method called printRepo():

The printRepo() method outputs 3 main things:

1. Repository name, folder location, URL, and the number of remotes
2. All of the information I need about each local branch
3. Any details about modified or untracked files

The output of calling printRepo() looks like this when called on my repository for CryptoConvert:

One thing I’d like to highlight about my implementation of the printRepo() method is the importance of calling git fetch on any remotes.

If are unfamiliar with the git fetch command, it basically downloads all of the updated information about the remote repository, but won’t integrate the updates into your local working files.

This is important because if the remote repository has been updated elsewhere, you want to ensure that your local repository is aware of those changes. Later in the GitUpdated’s lifecycle when it determines how far ahead or behind a branch is to the default, the information will be accurate.

GitPython makes this pretty easy with a dedicated fetch() method that you can call on a remote:

remote.fetch()

Falling back to the graphic about how I’ve split up the console output, all of section 1 is contained in the printRepo() method. Parts 2 and 3 get delegated into their own methods to keep the code compartmentalized:

• listBranches()
• printUncommittedChanges()

 

Output Section 2 – listBranches()

The information we want to fetch and display with this method are:

• # local branches, and for each branch:
  • # commits ahead is it from the default branch
  • # commits behind is it from the default branch
  • Branch name
  • Whether the branch is currently checked out

* Note: The default branch is the branch that we compare against each local branch we have so we can get the values of how far ahead/behind it is. For my repositories this is usually the master branch.

Our first step is to get a list of all the local branches for this repository. GitPython provides a pretty handy method for this:

branchList = repo.branches

Then, GitUpdated needs to know the name of the default branch. I’ve separated that logic out into it’s own method called getDefaultBranch():

defaultBranch = getDefaultBranch(repo, url)

For the getDefaultBranch() method, we need to come up with some sort of Git command that will reliably tell us which branch is the head branch for the repository. Once we have that, we can use what we have available through GitPython to implement it.

The Git command I eventually settled on that was easily available through GitPython was this:

git remote show origin

Which will tell us which branch is the head, but also gives us way more information than we need:

So I added a few extra commands to cut the output down:

git remote show origin | grep "HEAD branch" | cut -d ":" -f 2

And the default branch name gets returned back to us just like this – “master”:

There may be some edge cases or repository configurations that this implementation will need to be adjusted to account for, but most of my repositories are relatively simple in structure so this works fine for me. Your results may vary.

Now all we have to do is implement this sequence of commands with Python. It’s clunky, but here’s what I came up with:

Once we know what the default branch is, we need to use it as a baseline against each local branch we have so that we can determine how many commits ahead or behind it is.

Again we need to formulate some sort of git command and figure out how to accomplish the same thing using GitPython. What I eventually settled on was this:

git rev-list --left-right --count --branches origin/[default_branch]...[current_local_branch]

Replace [default_branch] and [current_local_branch] with the actual branch names.

This command will compare the commit lists of the two branches. It returns a count of the number of commits present in the default branch, but not in the local working branch — how far “behind”.

It also returns the number of commits present in the local working branch, but not in the default branch — how far “ahead”.

The official Git documentation has a much better explanation of how the rev-list command works than I could write, so I’ll just refer you there.

Here is what that command would look like if I were to run it directly on my repository for my app CryptoConvert when looking at the v3.62 branch I’ve been working on:

git rev-list --left-right --count --branches origin/master...v3.62

Where “origin/master” is the default branch, and “v3.62” is a local feature branch. The output of that command in this case gives me just the two numbers I need:

0 2

In other words, the v3.62 working branch in my example is 0 commits behind, and 2 commits ahead of the default branch for the repository (master) – which we can see reflected in the output of GitUpdated here:

If we take a closer look at the listBranches() method:

You’ll see it’s just about as clunky as my method for getting the default branch, but regardless it gets the job done.

GitPython provides direct access to the “rev-list” git command, but we need to pass in a few extra flags as parameters to get exactly what we are looking for:

• –left-right
• –count
• –branches (the two branch names to compare)

We can just call split() on the result of this method which gives us an array with 2 indices:

behind = output[0]
ahead = output[1]

The last step is to check if the current branch in the loop is active (currently checked out). GitPython provides the perfect interface for this:

repo.active_branch

If the current branch is equal to the active branch for the repository, it’s signified by a yellow asterisk identifier in the output.

And that’s all there is to part 2 of the repository output!

 

Output Section 3 – printUncommittedChanges()

Onto the last piece – compiling a list of any modified or untracked files. I created a method called printUncommitedChanges():

GitPython provides an easy way to check the status of the repository with the is_dirty() method. It returns true or false depending on if any modified or untracked files exist.

If the repository is clean (no changes), it will just print out a simple message:

Otherwise, there’s a little bit of extra work to do. GitPython can retrieve the list of changed files with this line:

changed_files = [item.a_path for item in repo.index.diff(None)]

Then, iterate over that list and print out each file name.

Getting the list of untracked files is even simpler than that:

untracked_files = repo.untracked_files

Loop over that list too, and print out each file name as we go.

With that, we’ve covered all of the major components of GitUpdated!

 

---

 

Ideas for Improvement:

Below are a few ideas to expand and improve GitUpdated. If I do make any significant changes, I’ll write a follow-up to this post discussing them. 

1. Add the ability for GitUpdated to read from a configuration file instead of hardcoding the pre-selectable environment information as strings in the script. This makes the project much easier to update and maintain moving forward, and is generally a good practice.
2. Allow an additional “environment” parameter to be passed in to act as a filter for all of the pre-defined path prompts. Then you’d only be presented the options corresponding to the dev environment you are currently in. Meaning if I ran GitUpdated from my laptop, the only pre-selectable options I’d be presented with are viable paths for that machine.
3. Implement additional Git commands that add more data, such as notifications of any local branches that are not present in the remote and perhaps some metrics about dates/commit activity.

 

---

 

Final Thoughts

If you have any questions, comments, or suggestions for improvement of this post, I’m happy to hear them. Leave a comment down below or contact me and let me know. Fill out the form below to subscribe and get notified to your email whenever I publish a new post.

You might also like some of the other Side Project Time Hack projects I’ve written about. I’ve also written plenty of other content surrounding app development and general programming. I hope this post inspires you to write your own version of GitUpdated for whatever your project needs are.