Subscribe to our newsletter
📬 Receive new lessons straight to your inbox (once a month) and join 40K+ developers in learning how to responsibly deliver value with ML.
Whether we're working individually or with a team, it's important that we have a system to track changes to our projects so that we can revert to previous versions and so that others can reproduce our work and contribute to it. Git is a distributed versional control system that allows us do exactly this. Git runs locally on our computer and it keeps track of our files and their histories. To enable collaboration with others, we can use a remote host (GitHub, GitLab, BitBucket, etc.) to host our files and their histories. We'll use git to push our local changes and pull other's changes to and from the remote host.
Git is traditionally used to store and version small files <100MB (scripts, READMEs, etc.), however, we can still version large artifacts (datasets, model weights, etc.) using text pointers pointing to blob stores. These pointers will contain information such as where the asset is located, it's specific contents/version (ex. via hashing), etc.
Initialize a local repository (.git directory) to track our files:
We can see what files are untracked or yet to be committed:
We can see that we have some files that we don't want to push to a remote host, such as our virtual environment, logs, large data files, etc. We can create a .gitignore file to make sure we aren't checking in these files.
We'll add the following files to the file:
For now, we're going to add data to our .gitignore file as well but this means that others will not be able to produce the same data assets when they pull from our remote host. To address this, we'll push pointers to our data files in our versioning lesson so that the data too can be reproduced exactly as we have it locally.
Tip
Check out our project's .gitignore for a more complete example that also includes lots of other system artifacts that we would normally not want to push to a remote repository. Our complete .gitignore file is based on GitHub's Python template and we're using a Mac, so we added the relevant global file names as well.
If we run git status now, we should no longer see the files we've defined in our .gitignore file.
Next, we'll add our work from the working directory to the staging area.
Now running git status will show us all the staged files:
Now we're ready to commit the files in the staging area to the local repository. The default branch (a version of our project) will be called main.
The commit requires a message indicating what changes took place. We can use git commit --amend to edit the commit message if needed. If we do a git status check we'll see that there is nothing else to commit from our staging area.
Now we're ready to push the updates from our local repository to a remote repository. Start by creating an account on GitHub (or any other remote repository) and follow the instructions to create a remote repository (it can be private or public). Inside our local repository, we're going to set our username and email credentials so that we can push changes from our local to the remote repository.
Next, we need to establish the connection between our local and remote repositories:
Now we're ready to start adding to our project and committing the changes.
If we (or someone else) doesn't already have the local repository set up and connected with the remote host, we can use the clone command:
And we can clone a specific branch of a repository as well:
When we want to add or change something, such as adding a feature, fixing a bug, etc., it's always a best practice to create a separate branch before developing. This is especially crucial when working with a team so we can cleanly merge our work with the main branch after discussions and reviews.
We'll start by creating a new branch:
We can see all the branches we've created with the following command where the * indicates our current branch:
We can easily switch between existing branches using:
Once we're in a branch, we can make changes to our project and commit those changes.
Note that we are pushing this branch to our remote repository, which doesn't yet exist there, so GitHub will create it accordingly.
When we push our new branch to the remote repository, we'll need to create a pull request (PR) to merge with another branch (ex. our main branch in this case). When merging our work with another branch (ex. main), it's called a pull request because we're requesting the branch to pull our committed work. We can create the pull request using steps outlined here: Creating a pull request.
Note
We can merge branches and resolve conflicts using git CLI commands but it's preferred to use the online interface because we can easily visualize the changes, have discussion with teammates, etc.
Once we accepted the pull request, our main branch is now updated with our changes. However, the update only happened on the remote repository so we should pull those changes to our local main branch as well.
Once we're done working with a branch, we can delete it to prevent our repository from cluttering up. We can easily delete both the local and remote versions of the branch with the following commands:
So far, the workflows for integrating our iterative development has been very smooth but in a collaborative setting, we may need to resolve conflicts. Let's say there are two branches (a and b) that were created from the main branch. Here's what we're going to try and simulate:
When we try to merge the second PR, we have to resolve the conflicts between this new PR and what already exists in the main branch.
We can resolve the conflict by choosing which content (current main which merged with the a branch or this b branch) to keep and delete the other one. Then we can merge the PR successfully and update our local main branch.
Once the conflicts have been resolved and we merge the PR, we can update our local repository to reflect the decisions.
Note
We only have a conflict because both branches were forked from a previous version of the main branch and they both happened to alter the same content. Had we created one branch first and then updated main before creating the second branch, we wouldn't have any conflicts. But in a collaborative setting, different developers may fork off the same version of the branch anytime.
A few more important commands to know include rebase and stash.
Git allows us to inspect the current and previous states of our work at many different levels. Let's explore the most commonly used commands.
We've used the status command quite a bit already as it's very useful to quickly see the status of our working tree.
If we want to see the log of all our commits, we can do so using the log command. We can also do the same by inspecting specific branch histories on the Git online interface.
Commit IDs are 40 characters long but we can represent them with the first few (seven digits is the default for a Git SHA). If there is ambiguity, Git will notify us and we can simply add more of the commit ID.
If we want to know the difference between two commits, we can use the diff command.
One of the most useful inspection commands is blame, which allows us to see what commit was responsible for every single line in a file.
Sometimes we may have done something we wish we could change. It's not always possible to do this in life, but in the world of Git, it is!
Sometimes we may just want to undo adding or staging a file, which we can easily do with the restore command.
Now if we already made the commit but haven't pushed to remote yet, we can reset to the previous commit by moving the branch pointer to that commit. Note that this will undo all changes made since the previous commit.
HEAD is a quick way to refer to the previous commit. Both HEAD and any previous commit ID can be accompanied with a ^ or ~ symbol which acts as a relative reference. ^n refers to the nth parent of the commit while ~n refers to the nth grandparent. Of course we can always just explicitly use commit IDs but these short hands can come in handy for quick checks without doing git log to retrieve commit IDs.
But instead of moving the branch pointer to a previous commit, we can continue to move forward by adding a new commit to revert certain previous commits.
Sometimes we may want to temporarily switch back to a previous commit just to explore or commit some changes. It's best practice to do this in a separate branch and if we want to save our changes, we need to create a separate PR. Note that if you do checkout a previous commit and submit a PR, you may override the commits in between.
There so many different works to work with git and sometimes it can became quickly unruly when fellow developers follow different practices. Here are a few, widely accepted, best practices when it comes to working with commits and branches.
Leverage git tags to mark significant release commits. We can create tags either through the terminal or the online remote interface and this can be done to previous commits as well (in case we forgot).
Tag names usually adhere to version naming conventions, such as v1.4.2 where the numbers indicate major, minor and bug changes from left to right.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
To cite this content, please use:
1
2
3
4
5
6 | @article{madewithml,
author = {Goku Mohandas},
title = { Git - Made With ML },
howpublished = {\url{https://madewithml.com/}},
year = {2023}
}
|