Getting started with OpenFOAM and git

I have the general impression from the forum that many OpenFOAM users may not always understand how they should (or could) be using git effectively when they are working with OpenFOAM.

After an initial git clone to set things up, the simplest method would be to simply use a periodic git pull to update things. However, this has a few distinct disadvantages:

• you have to recompile things immediately after the pull
• it makes it somewhat difficult to deal with any local changes

These disadvantages may not seem severe, but with a few minor changes to your workflow it is easy to start making git work for you. This is not a git tutorial per se, but rather a small quick-start for using git more effectively with OpenFOAM. For some good introductory git tutorials, I found these resources useful:

• Randal Schwartz’s google tech talk gives a very good background about the structure and concepts and helps demystify many aspects
• Bart Trojanowski’s “Git the basics”

Of course, a simple man gittutorial or git tutorial --help works fairly well too and there are plenty of other resources as well: learn.github.com, gitready.com, etc.

The Basic Idea(s)

Never work on tracked branches!!

Unless you have write access to the remote repository and really know what you are doing, nobody should be working on a tracked branch. If you used git clone to initialize your OpenFOAM git repository without any other options, then you will be on a local master branch that tracks the remote origin/master branch (the output of git branch -al can be helpful to list all of the branches known to your repository).

Using a remote-tracking branch is not really a terrible thing, but will become a real nuisance when you start wanting to make any local changes. Even seemingly trivial changes like changing the value of foamInstall in the etc/{bashrc,cshrc} or adjusting the docBrowser string in etc/controlDict should be done on a local non-tracked branch rather than on a tracked branch.

Use your own branches for everything

This preserves your modifications across updates and helps with seeing which changes have been introduced from where. If you have previously worked with subversion, you should note the following about git branches:

Local branches are really cheap (40+1 bytes disk storage) – use them!

• branches are not incredibly annoying and confusing (cf. CVS)
• branches do not need a new subdirectory copy (cf. SVN)

Local branches are also a great way when you are developing code that may not work out. If you create a new branch for it, you have a simple means of switching between code versions. If the coding idea works, you can either merge it back into your normal branch, or rename it to make it your new normal branch. If the coding doesn’t work, you can just leave it about as an unused branch that might prove useful for coding ideas later or else just delete it.

Apart from using local non-tracked branches, the only other significant change to your workflow would be to use git fetch followed by git merge instead of the simpler git pull. This has the following advantages:

Finer grained control

• you can fetch and merge from various sources.

Allows asynchronous development

• the time interval between fetch and merge is completely arbitrary.

If you follow the pattern of using fetch and merge separately, you can start to do more interesting things – such as only merging part of the remote branch or only using some of the new commits (called cherry-picking). This can be useful, for example, if you need some of the bugfixes, but can’t afford the time just now for the full recompile that a later commit might entail.

However, in the spirit of a quick-start, we are initially just concerned about getting things going.

Starting from a clean system

Assuming /data/app/OpenFOAM as the applications directory, create a directory for all OpenFOAM-related things (if it doesn’t already exist) and change there:

$ mkdir -p /data/app/OpenFOAM
$ cd /data/app/OpenFOAM

We can use a single clone command to start things off

$ git clone --origin repo git://repo.or.cz/OpenFOAM-1.6.x.git

The leading git:// protocol can also be replaced by http:// if there is a firewall blocking the git port 9418. if you have the choice though, the git:// protocol is more efficient (faster).

For some added clarity, we took the extra time to designate the remote repository as repo (since it is from repo.or.cz) rather just using the default name origin. This isn’t overly important, but can be less confusing if you later clone from the cloned repository.

Make a local branch and remove a tracking branch

If you have just completed an initial clone, you are very likely on the master branch that is set up as a tracking-remote for the repo/master. This is probably also true if you have already been working with git pull. This is the very first thing that we wish to change, but to be on the safe side we’ll first find out where we are.

List all branches (local and remote):

$ git branch -a -l
  * master
    repo/master

the branch with the ‘*’ is the current (active) branch.

Before we create a new local branch and remove the tracking branch, we should also be careful and verify that the tree and the staging area are clean:

$ git status

If you have been working cleanly, there shouldn’t be anything showing up here as being staged or needing committing. If some files show up as being modified, you should check what has changed before deciding if you want to keep these changes or not:

$ git diff

If you have inadvertently made changes to a particular file that you would like to discard, using git checkout HEAD some/path/fileName will retrieve a fresh copy from the repository.

Assuming everything is clean, we can simply start our userName branch (in my case olesen as the user-name) from the current position:

$ git checkout -b userName HEAD

We can also do the same thing if we want to have a branch point starting at the current position of repo/master, except here we would need to instruct git not to track the remote:

$ git checkout --no-track -b userName repo/master

When you start sharing repositories with coworkers, you will soon appreciate that each user has given their branch a unique name.

As before, git branch -a -l can be used to list the branches. To ensure we aren’t tempted to start working on a tracking branch again, we’ll delete it immediately. Note that this only removes the branching information, not any of the OpenFOAM files themselves. If we wish, we can re-create a tracking branch at any time.

$ git branch -d master

We can now develop asynchronously between branches with our changes on userName and all of the upstream changes on repo/master. If we want to see which commits exist on one branch but not on the other, it is now a bit easier:

$ git log --no-merges userName ^repo/master
$ git log --no-merges repo/master ^userName

Next Steps

The next step to have a working OpenFOAM system is to set up the environment variables. Rather than editing the etc/{bashrc,cshrc} files or hard-coding a particular path in our ~/.bashrc file, it is more versatile to use the environment variable FOAM_INST_DIR as the following snippet illustrates:

FOAM_INST_DIR=/data/app/OpenFOAM   # where to look
vers="1.6.x 1.5.x"                 # versions to look for
vers=${profile_foam:-$vers}        # check for default

if [ -d $FOAM_INST_DIR ]
then
    export FOAM_INST_DIR

    # search for the correct version
    for foamVersion in $vers
    do
        foamDotFile=$FOAM_INST_DIR/OpenFOAM-$foamVersion/etc/bashrc
        if [ -f $foamDotFile ]
        then
            export WM_PROJECT_VERSION=$foamVersion
            . $foamDotFile
            break
        fi
    done
else
    unset FOAM_INST_DIR
fi

unset vers profile_foam foamDotFile foamVersion

The extra environment variable profile_foam can be used to specify a particular version of OpenFOAM should be used instead of searching through the list and makes it easier to write a corresponding alias.

Now that all the environment has been setup, the usual Allwmake can be used. The most difficult part of compiling OpenFOAM is usually resolving issues with building ParaView (eg, figuring out which QT version works properly on your system, or having missing QT bits like webkit etc.).

We are now ready to work, work, commit, work, work, commit, …

Better Practices

Use git commit --amend whenever it makes sense. This can help avoid this sort of thing:

• “fixed X”
• “really fixed X”
• “fixed silly typo in previous fix”
• “this is embarrassing, really, really fixed X”
• “finally fixed (I hope)”

If you haven’t already published your repository, it can often make sense to use --amend for simple repairs to the last commit instead. If you need, you can also use git rebase --interactive to rearrange/squash/drop commits, but this is not something I would advise initially (it is quite easy to make a complete mess of it). In any case, never use commit --amend or rebase on pushed branches!

Housekeeping

To improve how things work, there are several git configuration parameters that can be used. The command git config can be used to get/set them, but I’ve never taken the time to figure out how that command works and just edit the INI-style files directly with a text editor instead.

The ~/.gitconfig contains global configuration parameters. This is the place for everything that should apply to all your repositories. For example,

[alias]
    log1 = log --pretty=oneline
    logs = log --stat
[user]
    name = Mark Olesen
    email = Mark.Olesen@some.domain
[core]
    editor = xjed
    pager = less -FRX
[diff]
    color = auto
[merge]
    tool = kdiff3

NOTE the editor must be a foreground process so that commands like git commit can wait for it.

The .git/config contains repository-specific configuration parameters. This is a useful place for many of your aliases:

[alias]
    getrepo = fetch repo
    logrepo = log --stat --no-merges repo/master ^HEAD
    loghome = log --stat --no-merges bundle/home ^HEAD
    logwork = log --stat --no-merges HEAD ^bundle/home
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "repo"]
    url = http://repo.or.cz/r/OpenFOAM-1.6.x.git
    fetch = +refs/heads/*:refs/remotes/repo/*
[remote "bundle"]
    url = /media/TOSHIBA/transfer/home2work.gitbundle
    fetch = +refs/heads/home:refs/remotes/bundle/home

Workflow

On any local branch:

• Work, work, work, commit, work, work, work, commit

Periodically fetch changes:

$ git fetch repo
$ git getrepo   # using the git alias

See what has changed that we don’t already have:

$ git log --stat --no-merges repo/master ^HEAD
$ git logrepo   # using the git alias

Or see what changed at all:

$ git show repo/master
$ git show repo/master~1
$ git diff HEAD repo/master

Or just what changed:

$ git whatchanged repo/master

Continue to work/commit, or merge now (only on a clean tree):

$ git merge repo/master

Merging without any local changes just fast-forwards the branch.

Working at home

For the case where you have an OpenFOAM git repository and some coding at work and have a similar setup at home, but a firewall or equivalent between them, it is still possible to fetch and merge between them using a so-called git bundle. The transport method is termed “sneaker net”, which usual translates to using a USB stick to connect the two repositories.

The command git bundle is used to pack changes. After the USB stick has been transported to its new location, it can be added as a new remote repository that we’ll call bundle for the sake of convenience:

$ git remote add bundle /media/TOSHIBA/transfer/home2work.gitbundle
$ git remote show bundle

The fetch/merge is identical to any other remote repository:

$ git fetch bundle
$ git merge bundle/home

Examining the .git/config file reveals that the bundle remote is indeed fetching from a particular file on the USB stick:

[remote "bundle"]
    url = /media/TOSHIBA/transfer/home2work.gitbundle
    fetch = +refs/heads/home:refs/remotes/bundle/home

The setup at work and home are similar but not identical. At work, we have the current branch being userName and at home we have the current branch simply being home. We can work asynchronously on some code bits at work and some code bits at home and use git to merge the branches whenever required.

Adding some simple home2work and work2home scripts makes for easier creation of the git bundles. The work2home script is shown here as an example, with the corresponding home2work bits shown in the comments:

#!/bin/sh
cd ${0%/*} || exit 1    # run from this directory
# work -> home
bundle=16x-work2home.gitbundle
here=olesenm
there=bundle/home
#
# home -> work
# bundle=16x-home2work.gitbundle
# here=home
# there=bundle/olesen
#
if [ -f $bundle ]
then
    echo
    echo "remove existing bundle: $bundle"
    echo
    rm $bundle
fi
set -x
git bundle create $bundle $here ^$there

cat <<EOF
verifying bundle contents
# ----------------------------------------------------------------------
EOF
git bundle verify $bundle

Note that even for cases in which an electronic connection exists between the two repositories, it is still advantageous to use different branch names on each repository.

Closure

I hope this short description proves useful for OpenFOAM users working with git. Git may certainly seem confusing and cryptic to a new user, but with some minor learning effort it proves to be a trusty and very useful tool. In this regard it is similar to my own experience with C++, Perl and Unix.

In general, git fits into the usual Unix tools paradigm (eg, ls, cat, …) and as such it is a command-line program. There are some graphical user interfaces, but the difficult posed by git is with understanding the underlying concepts and not the commands themselves. Nonetheless, for viewing the history or the branching relationships a graphical view can be fairly useful. Here are some of the common graphical tools and plugins:

gitk

graphical branch viewer (Tcl/Tk) included with git

qgit

same idea, but with Qt (looks quite nice) and possible to define command sequences if you figure which ones you want.

egit

eclipse plugin