28

Moving Files from one Git Repository to Another, Preserving History

Posted May 17th, 2011 in Development and tagged , , by Greg Bayer
      

If you use multiple git repositories, it’s only a matter of time until you’ll want to refactor some files from one project to another.  Today at Pulse we reached the point where it was time to split up a very large repository that was starting to be used for too many different sub-projects.

After reading some suggested approaches, I spent more time than I would have liked fighting with Git to actually make it happen. In the hopes of helping someone else avoid the same trouble, here’s the solution that ended up working best. The solution is primarily based on ebneter’s excellent question on Stack Overflow.

Another solution is Linus Torvald’s “The coolest merge, EVER!“ Unfortunately, his approach seems to require more manual fiddling than I would like and results in a repository with two roots. I don’t completely understand the implications of this, so I opted for something more like a standard merge.

Goal:

  • Move directory 1 from Git repository A to Git repository B.

Constraints:

  • Git repository A contains other directories that we don’t want to move.
  • We’d like to perserve the Git commit history for the directory we are moving.


Get files ready for the move:

Make a copy of repository A so you can mess with it without worrying about mistakes too much.  It’s also a good idea to delete the link to the original repository to avoid accidentally making any remote changes (line 3).  Line 4 is the critical step here.  It goes through your history and files, removing anything that is not in directory 1.  The result is the contents of directory 1 spewed out into to the base of repository A.  You probably want to import these files into repository B within a directory, so move them into one now (lines 5/6). Commit your changes and we’re ready to merge these files into the new repository.

git clone <git repository A url>
cd <git repository A directory>
git remote rm origin
git filter-branch --subdirectory-filter <directory 1> -- --all
mkdir <directory 1>
mv * <directory 1>
git add .
git commit

Merge files into new repository:

Make a copy of repository B if you don’t have one already.  On line 3, you’ll create a remote connection to repository A as a branch in repository B.  Then simply pull from this branch (containing only the directory you want to move) into repository B.  The pull copies both files and history.  Note: You can use a merge instead of a pull, but pull worked better for me. Finally, you probably want to clean up a bit by removing the remote connection to repository A. Commit and you’re all set.

git clone <git repository B url>
cd <git repository B directory>
git remote add repo-A-branch <git repository A directory>
git pull repo-A-branch master
git remote rm repo-A-branch

Update: Removed final commit thanks to Von’s comment.

VN:F [1.9.22_1171]
Rating: 4.4/5 (17 votes cast)
Moving Files from one Git Repository to Another, Preserving History, 4.4 out of 5 based on 17 ratings       
  • http://buildengineer.org/ Kate Ebneter

    Very nice writeup, and I’m happy to see that my question/answer was helpful!

    VA:F [1.9.22_1171]
    Rating: 5.0/5 (2 votes cast)
  • Von

    Just tried this on a couple quickly whipped up repos and it worked well. Only comment was there was nothing to commit after the pull, so I’m not sure you need the last two steps.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://GBayer.com Greg Bayer

    Great catch! The commit is not required with the pull approach. Will update accordingly.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Anonymous

    Thank you, for explaining this!. Your post will be a big help. I have wanted to do this for some time, but kept putting it off!

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://www.facebook.com/joechiu Joseph Chiu

    hi, based on what I read on http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-repo-to-another-not-a-clone-preserving-history, I wonder if your line 5 should be “mkdir -p <directory >” and line 6 should be “git mv * <directory 1>” ?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://GBayer.com Greg Bayer

    Thanks for the feedback. In this case I believe the results should be similar either way.

    1) “mkdir -p” is only required if the new directory is more than one level deep.

    2) There shouldn’t be much difference between “mv *” and “git mv *” in this case.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Mguyre

    Need to add a git fetch between steps 4 and 5 for the new repository to retrieve the tags and history

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Adam Monsen

    Great post, thank you. One suggestion: change the title as follows: s/file/one directory/.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://adammonsen.com Adam Monsen

    Great post, thank you. One suggestion: change the title as follows: s/Files/one directory/.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://adammonsen.com Adam Monsen

    Sorry, I meant: change “Files” to “one directory” or “a directory”.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://GBayer.com Greg Bayer

    Thanks for the suggestion. You can actually use this approach to move an arbitrary set of files by first moving them into a temporary directory. Because of this, the current title seems to be appropriate and more general.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Navitf

    Hi, But when files are moved into a temporary directory, the command: “git filter-branch –subdirectory-filter” extract history that is relevant only to the temporary directory and thus real history logs are not preserved. Any idea how to overcome this?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://GBayer.com Greg Bayer

    Thats an interesting point. This wasn’t a problem in my case, so I haven’t looked into it. Maybe another reader can suggest a solution?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • 123456

    I get an error message when I run the get filter-branch command:
    $ git filter-branch –subdirectory-filter mt — –all
    C:Program Files (x86)Git/libexec/git-core/git-filter-branch: line 289: /libexe
    c/git-core/git: Bad file number
    Could not get the commits

    In typical git fashion, the error message is incomprehensible to me. Any idea what’s going wrong?

    VA:F [1.9.22_1171]
    Rating: 1.0/5 (1 vote cast)
  • http://GBayer.com Greg Bayer

    I think that means git can’t access one of your files. Based on a few posts I see on stackoverflow.com, this could be caused by a bad network connection or proxy configuration.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • 123456

    Don’t understand how this could be the case. I cloned the repo to local disk.. and to my understanding, “git remote rm origin” severs the link between my local repo and the remote one.. so I don’t see where networks/proxies would enter into it.

    VA:F [1.9.22_1171]
    Rating: 1.0/5 (1 vote cast)
  • Xavier MARTIN

    Really really helpful post…
    Thanks much for sharing :)

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://twitter.com/SteveALee SteveALee

    if you don’t use git mv * the deleted files will not be staged for a commit. Also use -k to skip error of moving the dir itself

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Robbie Van Gorkom

    We had the same issue at the office, I wrote a script to exactly this. https://github.com/vangorra/git_split

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Rasheed Barnes

    Good stuff. worked like a charm.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Bernd

    Thaaaanks a lot. This post saved me a shit ton of hours. :)

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • devguy

    this is cool, but it seems that you can only see the old history if you do git log –follow [file] , which is kinda inconvenient, especially in a large project. am i missing something? is there a way to modify this process so that the –follow is not required?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://GBayer.com/ Greg Bayer

    You should be able to see the history in all the normal ways. Personally I tend to use gitk or github to view the history for old files.

    How do you want to be able to view it?

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Bastian Krol

    Thanks a bunch for this guide.

    Here is my version:

    To move some directories from repository A to repository B without losing history:

    1. git clone tmp-repo
    2. cd tmp-repo
    3. git checkout
    4. git remote rm origin # not really needed
    5. git filter-branch –subdirectory-filter — –all
    6. mkdir -p
    7. git mv -k *
    8. git commit
    9. cd # clone it, if you didn’t do already
    10. Create a new branch and check it out
    11. git remote add origin-tmp-repo
    12. git pull origin-tmp-repo
    13. rm -rf

    Repeat all steps with every that needs to be moved. You’ll need a new tmp-repo for every directory, because “git filter-branch –subdirectory …” can only take one directory as an argument and the repo is largely unusable after executing the command. That’s why there is a rm -rf in step 13. When transferring subsequent directories, steps 10 and 11 can be omitted.

    When you are done with all directories, you should do
    git remote rm origin-tmp-repo and git push in local repository B.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://webdevguy.myopenid.com/ webdevguy

    This was very helpful. I had a much simpler requirement. I needed to pull one directory and all it’s history out of one repo and create a new repo for just that directory. Here are the steps that worked for me:

    1. git clone <git repository A url>
    2. cd <new git repository>
    3. git remote rm origin
    4. git filter-branch –subdirectory-filter <directory 1> — –all

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://www.facebook.com/nathan.whitehead.3551 Nathan Whitehead

    Thanks for putting this up, worked like a charm. I just set up git as a deployment mechanism, this helped me split out the bits I needed from my main repository.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • http://twitter.com/sleberrig Samuel Le Berrigaud

    using “git mv *” worked better for me. Somehow the history is ‘better’ kept that way… Not sure why and how though.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  • Arthur Taborda

    Easier: to merge files to new repository, simply do:

    git push :master

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)