41

Moving Files from one Git Repository to Another, Preserving History

Posted May 17th, 2011 in Development and tagged , , by Greg Bayer
                                

If you use multiple git repositories, it’s only a matter of time until you’ll want to refactor some files from one project to another.  Today at Pulse we reached the point where it was time to split up a very large repository that was starting to be used for too many different sub-projects.

After reading some suggested approaches, I spent more time than I would have liked fighting with Git to actually make it happen. In the hopes of helping someone else avoid the same trouble, here’s the solution that ended up working best. The solution is primarily based on ebneter’s excellent question on Stack Overflow.

Another solution is Linus Torvald’s “The coolest merge, EVER!” Unfortunately, his approach seems to require more manual fiddling than I would like and results in a repository with two roots. I don’t completely understand the implications of this, so I opted for something more like a standard merge.

Goal:

  • Move directory 1 from Git repository A to Git repository B.

Constraints:

  • Git repository A contains other directories that we don’t want to move.
  • We’d like to perserve the Git commit history for the directory we are moving.


Get files ready for the move:

Make a copy of repository A so you can mess with it without worrying about mistakes too much.  It’s also a good idea to delete the link to the original repository to avoid accidentally making any remote changes (line 3).  Line 4 is the critical step here.  It goes through your history and files, removing anything that is not in directory 1.  The result is the contents of directory 1 spewed out into to the base of repository A.  You probably want to import these files into repository B within a directory, so move them into one now (lines 5/6). Commit your changes and we’re ready to merge these files into the new repository.

git clone <git repository A url>
cd <git repository A directory>
git remote rm origin
git filter-branch --subdirectory-filter <directory 1> -- --all
mkdir <directory 1>
mv * <directory 1>
git add .
git commit

Merge files into new repository:

Make a copy of repository B if you don’t have one already.  On line 3, you’ll create a remote connection to repository A as a branch in repository B.  Then simply pull from this branch (containing only the directory you want to move) into repository B.  The pull copies both files and history.  Note: You can use a merge instead of a pull, but pull worked better for me. Finally, you probably want to clean up a bit by removing the remote connection to repository A. Commit and you’re all set.

git clone <git repository B url>
cd <git repository B directory>
git remote add repo-A-branch <git repository A directory>
git pull repo-A-branch master
git remote rm repo-A-branch

Update: Removed final commit thanks to Von’s comment.

                                
  • http://buildengineer.org/ Kate Ebneter

    Very nice writeup, and I’m happy to see that my question/answer was helpful!

  • Von

    Just tried this on a couple quickly whipped up repos and it worked well. Only comment was there was nothing to commit after the pull, so I’m not sure you need the last two steps.

  • http://GBayer.com Greg Bayer

    Great catch! The commit is not required with the pull approach. Will update accordingly.

  • Anonymous

    Thank you, for explaining this!. Your post will be a big help. I have wanted to do this for some time, but kept putting it off!

  • http://www.facebook.com/joechiu Joseph Chiu

    hi, based on what I read on http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-repo-to-another-not-a-clone-preserving-history, I wonder if your line 5 should be “mkdir -p <directory >” and line 6 should be “git mv * <directory 1>” ?

  • http://GBayer.com Greg Bayer

    Thanks for the feedback. In this case I believe the results should be similar either way.

    1) “mkdir -p” is only required if the new directory is more than one level deep.

    2) There shouldn’t be much difference between “mv *” and “git mv *” in this case.

  • Mguyre

    Need to add a git fetch between steps 4 and 5 for the new repository to retrieve the tags and history

  • Adam Monsen

    Great post, thank you. One suggestion: change the title as follows: s/file/one directory/.

  • http://adammonsen.com Adam Monsen

    Great post, thank you. One suggestion: change the title as follows: s/Files/one directory/.

  • http://adammonsen.com Adam Monsen

    Sorry, I meant: change “Files” to “one directory” or “a directory”.

  • http://GBayer.com Greg Bayer

    Thanks for the suggestion. You can actually use this approach to move an arbitrary set of files by first moving them into a temporary directory. Because of this, the current title seems to be appropriate and more general.

  • Navitf

    Hi, But when files are moved into a temporary directory, the command: “git filter-branch –subdirectory-filter” extract history that is relevant only to the temporary directory and thus real history logs are not preserved. Any idea how to overcome this?

  • http://GBayer.com Greg Bayer

    Thats an interesting point. This wasn’t a problem in my case, so I haven’t looked into it. Maybe another reader can suggest a solution?

  • 123456

    I get an error message when I run the get filter-branch command:
    $ git filter-branch –subdirectory-filter mt — –all
    C:Program Files (x86)Git/libexec/git-core/git-filter-branch: line 289: /libexe
    c/git-core/git: Bad file number
    Could not get the commits

    In typical git fashion, the error message is incomprehensible to me. Any idea what’s going wrong?

  • http://GBayer.com Greg Bayer

    I think that means git can’t access one of your files. Based on a few posts I see on stackoverflow.com, this could be caused by a bad network connection or proxy configuration.

  • 123456

    Don’t understand how this could be the case. I cloned the repo to local disk.. and to my understanding, “git remote rm origin” severs the link between my local repo and the remote one.. so I don’t see where networks/proxies would enter into it.

  • Xavier MARTIN

    Really really helpful post…
    Thanks much for sharing :)

  • http://twitter.com/SteveALee SteveALee

    if you don’t use git mv * the deleted files will not be staged for a commit. Also use -k to skip error of moving the dir itself

  • Robbie Van Gorkom

    We had the same issue at the office, I wrote a script to exactly this. https://github.com/vangorra/git_split

  • Rasheed Barnes

    Good stuff. worked like a charm.

  • Bernd

    Thaaaanks a lot. This post saved me a shit ton of hours. :)

  • devguy

    this is cool, but it seems that you can only see the old history if you do git log –follow [file] , which is kinda inconvenient, especially in a large project. am i missing something? is there a way to modify this process so that the –follow is not required?

  • http://GBayer.com/ Greg Bayer

    You should be able to see the history in all the normal ways. Personally I tend to use gitk or github to view the history for old files.

    How do you want to be able to view it?

  • Bastian Krol

    Thanks a bunch for this guide.

    Here is my version:

    To move some directories from repository A to repository B without losing history:

    1. git clone tmp-repo
    2. cd tmp-repo
    3. git checkout
    4. git remote rm origin # not really needed
    5. git filter-branch –subdirectory-filter — –all
    6. mkdir -p
    7. git mv -k *
    8. git commit
    9. cd # clone it, if you didn’t do already
    10. Create a new branch and check it out
    11. git remote add origin-tmp-repo
    12. git pull origin-tmp-repo
    13. rm -rf

    Repeat all steps with every that needs to be moved. You’ll need a new tmp-repo for every directory, because “git filter-branch –subdirectory …” can only take one directory as an argument and the repo is largely unusable after executing the command. That’s why there is a rm -rf in step 13. When transferring subsequent directories, steps 10 and 11 can be omitted.

    When you are done with all directories, you should do
    git remote rm origin-tmp-repo and git push in local repository B.

  • http://webdevguy.myopenid.com/ webdevguy

    This was very helpful. I had a much simpler requirement. I needed to pull one directory and all it’s history out of one repo and create a new repo for just that directory. Here are the steps that worked for me:

    1. git clone <git repository A url>
    2. cd <new git repository>
    3. git remote rm origin
    4. git filter-branch –subdirectory-filter <directory 1> — –all

  • http://www.facebook.com/nathan.whitehead.3551 Nathan Whitehead

    Thanks for putting this up, worked like a charm. I just set up git as a deployment mechanism, this helped me split out the bits I needed from my main repository.

  • http://twitter.com/sleberrig Samuel Le Berrigaud

    using “git mv *” worked better for me. Somehow the history is ‘better’ kept that way… Not sure why and how though.

  • Arthur Taborda

    Easier: to merge files to new repository, simply do:

    git push :master

  • zupeanut

    “It goes through your history and files, removing anything that is not in directory 1″

    The problem with this is if files that are *currently* in directory 1 but previously were not will have history before the move lost.

  • Cat Lookabaugh

    I was reading this post, and it doesn’t seem to quite be what I’m after. Caveat: I’m new to git, so I just may not understand.

    Here’s what I want:

    I’m using git to store documentation. We have different docbooks for each of our products and some content that fits more than one book. So, we have a common repository with that content (in a directory) and other repos for each product. The current process to get at the shared content involves some scripts and the git subtree command, with the shared content ending up in the product repo in a directory called shared-files. It’s a bit convoluted. Currently the shared content is only used for two products.

    I need to add another set of shared content that will be used by all product docbooks (stored in its own directory in common). I want to be able to pull one or both shared content directories into a product repo. I don’t need any history. I just want the current up-to-date common files in my product repo in their own directory. If I want to update the shared content, I will do so directly in the common repo.

    One way to accomplish this would be to clone the common repo and the product repo to my pc and then literally copy the directory I want from common repo into the product repo.

    Is there a relatively simple way to do this without cloning the common repo? I’d appreciate your advice!

  • http://GBayer.com/ Greg Bayer

    Have you considered adding the common repo as a submodule of each of the product repos?

  • Cat Lookabaugh

    My research thus far seems to indicate that subtree is preferred over submodule, though i’m not sure why. would submodule option allow me to pick and choose which directory or directories i want from the common repo?

  • http://GBayer.com/ Greg Bayer

    It wouldn’t let you choose certain directories, but it might be simpler overall if thats not a hard requirement. You could also combine the submodule approach with a simple script that deletes directories you don’t need from the local clone of the common submodule in each product repo.

    You could also consider creating separate common repos for each directory and only pulling in those you need for each product.

  • Cat Lookabaugh

    food for thought…thanks :)

  • gitnewbie

    Used the exact commands listed above and I see only the first 2 entries with git log although git log on the source shows me many more. Looks like partial history is moved.

  • Eliyahu

    Thanks!

  • Brandon Mintern

    Thanks! This was very helpful.

    Note that if your repository B branch has `rebase = true` set, then you will almost certainly want to use `git pull –no-rebase repo-A-branch master`.

  • Dave

    This helped … a lot!!! Thank you Greg!!!

    Motivated by your *excellent* posting (I can’t stress that enough), I dug around some more, since I also had to go the other direction … that is, once I did this, and had my desired subdirectory now as the entire project, I then wanted to submerge it into a subdirectory, still keeping the history, of course (i.e. no –follow required).

    I found http://stackoverflow.com/questions/4042816/how-can-i-rewrite-history-so-that-all-files-are-in-a-subdirectory

    Which had:

    git filter-branch –prune-empty –tree-filter ‘
    if [[ ! -e foo/bar ]]; then
    mkdir -p foo/bar
    git ls-tree –name-only $GIT_COMMIT | xargs -I files mv files foo/bar
    fi’

    This worked for me, pretty much literally.

    If this helps anyone, I dedicate the good will to Greg, who exemplifies what these postings should be all about imho.

    Thanks again Greg.

  • romu

    Hi, bit late to come here now, but I’ve discovered this article because of a similar need.

    I ran this instructions on the code I needed to move, and everything was fine…except a little problem, the history is not preserved at all. The files are well moved but with no history.

    Any idea? Thanks.

  • Karthik T

    I am using this to extract some stuff out as a gem and it works like a … gem! I didnt realize the first set of commands were destructive.. but no worries!

  • Carlos Vinicius

    perfect <3