Merging OpenFOAM versions in git

A question came up recently on this forum thread about how to merge the separate change histories from different OpenFOAM versions. The simplest solution is to merge with an ours strategy, which I’ll present directly.

Of course, you can also question why disconnected git histories exist in the first place and immediately notice that the same type of issues will simply re-occur with future OpenFOAM-1.7.x, OpenFOAM-1.8.x, etc releases. I’ll thus also take the opportunity to explain why I think the current OpenFOAM development model is perhaps not as bad at it might first appear and might indeed be the only practical development model at the moment.

The quick solution

This is literally a quick solution if you are using exising local files.

The merge operations are safe to use within your normal OpenFOAM directories (provided they are in a clean state), but it is also quite reasonable to use a new temporary directory for the operations – either for safety or just as a sandpit to play around a bit. Using a new temporary directory also nicely illustrates the distributed nature of git – each repository being its own centre of the universe.

Starting from a completely empty directory:

    mkdir /tmp/OpenFOAM-merge
    cd /tmp/OpenFOAM-merge

We initialize a git repository:

    git init

Since there is obviously nothing there yet, we’ll add some repository content. In this case from the current OpenFOAM version (1.6.x):

    git remote add foam16x /SOME/PATH/OpenFOAM-1.6.x/.git
    git fetch foam16x

We can reuse the remote and fetch commands to add any number of other repository references. We’ll also add in the previous OpenFOAM version (1.5.x):

    git remote add foam15x /SOME/PATH/OpenFOAM-1.5.x/.git
    git fetch foam15x

Using git branch -a -l you can display all the known branches.

Since OpenFOAM-1.6.x supercedes OpenFOAM-1.5.x, this is where we should be when conducting an ours merge. We’ll select the current topic branch (usually master) from the current OpenFOAM version (which we added as foam16x in this example):

    git checkout -b master foam16x/master

warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of foam16x/master
Branch master set up to track remote branch refs/remotes/foam16x/master.
Already on "master"

To keep things straight, and to avoid accidentally working on a tracked branch, we’ll make sure that we are working on a new branch that doesn’t currently exist in the remote repository. The name is fairly arbitrary, we’ll take merged for the example:

    git checkout -b merged HEAD

Switched to a new branch "merge"

Using git branch -a -l again, you can verify that you are indeed on the correct branch. Now that everything is prepared, the merge itself can be conducted:

    git merge --strategy ours foam15x/master

Merge made by ours.

Here we’ve merged in the foam15x/master branch and instructed git that the current branch (ie, the master from foam16x) always has precedence. The two histories are now joined as can be seen in the log information:

    git log --graph

Keeping the merged branch

Even although we worked in a temporary directory to create the merged branch, we may wish to also have a copy back in our original repositories. This is easily done by push-ing the new branch back to a particular repository:

    git push foam16x merged

To /SOME/PATH/OpenFOAM-1.6.x/.git
 * [new branch]      merged -> merged

Changing back to the original OpenFOAM-1.6.x directory, you can checkout the new branch:

    cd /SOME/PATH/OpenFOAM-1.6.x
    git checkout merged

Since the directory and file contents were resolved as ours there are no differences between the previous master branch and the new merged branch. The subsequent checkout merged thus shouldn’t affect any file modification times.

Exploring what has changed

Now that both repositories have been united, it is possible to explore what has changed. For the sake of clarity, I’ll use foam16x/master and foam15x/master to refer to the respective branches. The astute reader will notice that the following operations didn’t actually require the merge operation at all, only that both repositories be accessible as remotes from the same repository.

From the usual palette of git commands at our disposal, the most useful at this point is diff (and to a certain extent diff-tree as well).

To get a general idea of where most of the changes occurred:

    git diff --stat foam15x/master foam16x/master

which we can also restrict to a particular path:

    git diff --stat foam15x/master foam16x/master src/OpenFOAM/db

Or look for differences that contain a particular change. For example, to see all changes to files that use the new argsList::optionFound method:

    git diff -SoptionFound --stat foam15x/master foam16x/master

For a more detailed view, the --stat option is simply dropped. Again, a path can be specified to restrict the view. For example,

    git diff foam15x/master foam16x/master etc/bashrc

Why are there separate repositories / histories anyhow?

I must admit that this annoyed me quite a bit for a while. Why have completely separate repositories for 1.6.x and 1.5.x when separate branches would suffice? It helps to turn the question around:

How would it look like with everything in a single repository?

The real issue is actually about how the development should proceed. From an existing development line, from which the 1.5.x version initially stems, the development continues after a release:

o---o---o---o---o  development
    \
     o  1.5.x

The changes in development should normally not find their way into the stable 1.5.x branch unless they are clearly necessary bugfixes or feature requests with limited impingement on the overall code. Even many seemingly innocent cosmetic changes should probably not be applied to the stable branch – do you really want to recompile everything just because someone tweaked the documentation formatting in some key header?

Thus in short, the development and 1.5.x branches will rapidly diverge in their git histories. Bugfixes will be applied to both the development and the 1.5.x branches, but the development branch will obviously have many more commits:

o---o---o---o---o---o---o---o  development
    \
     o---o---o  1.5.x

When the time comes for an OpenFOAM-1.6.x release, a tested version of the development branch would be taken:

o---o---o---o---o---o---o---o---o---o---o---o  development
    \                           \
     o---o---o  1.5.x            o  1.6.x

For consistency with the previous example, we can use merge ours on the newly created 1.6.x branch to preserve the history (Note: this is functionally equivalent to a normal upstream merge for this example).

o---o---o---o---o---o---o---o---o---o---o---o  development
    \                            \
     A---o---o---o  1.5.x         B---M---o  1.6.x
              \                      /
               ----------------------

in which the ‘A’ and ‘B’ represent the branch points for 1.5.x and 1.6.x, respectively, and the ‘M’ represents the subsequent merge point with ours to rejoin the histories.

Thus while separate development and maintenence branches are indeed desirable and necessary, it isn’t strictly necessary that the various branches be split across separate repositories. However, there are also advantages to having separate repositories:

Pro

No confusion for users about which version they are using – one repository per OpenFOAM version.

The git repositories have fewer changes in them and are thus smaller and faster.

Contra

The git history of the internal development repository is missing from the released versions.

The only remaining difficulty (or irritation) might be that the git history of the internal development repository is missing from the released versions. I don’t know if we really need to know every step of OpenFOAM’s internal development in order to use it. It might help, it might be interesting, but probably isn’t strictly necessary.

On the other hand, I can quite understand why there is no OpenFOAM-dev repository published between official releases. Just imagine the flood of posts on the forum:

I’m using the bleeding edge version and the new class ‘X’ worked last week, but now I get lots of compiler error messages and it won’t compile. What is wrong?

I think you get the idea.