Copy multiple files and dirs from one git repo to another while keeping their original history

Question

My requirement: Break down one git repo, into multiple git repos, preserving the same directory structure as in the original repo, and preserve the commit history for the files that are copied to the new repo. What I have tried already:

1. First I tried git filter-branch --subdirectory-filter based on the suggestions in http://gbayer.com/development/moving-files-from-one-git-repository-to-another-preserving-history/ Result: The history is maintained, but can be viewed only on running git log --follow Also, the original commit history cannot be seen on Github. It just displays my merge commit as the only commit for that file, and does not display any previous commits. I can still live with this limitation and accept it as a solution. But another concern I have with this approach is that, for each folder and each file that I want to copy, I need to clone the original repo multiple times and also repeat all those 12 or 13 steps everytime. I would like to know if there is any simpler way of doing it, since I'm moving a lot of files around. Also, since the post is 5 years old, just wondering if newer easier solutions are available? (Surprisingly Google mostly shows this blog as the first search result)

2. Next thing I tried was a comment on the earlier Greg Bayer's post http://gbayer.com/development/moving-files-from-one-git-repository-to-another-preserving-history/#comment-2685894846 This solution made things a bit simpler by using git subtree split but the results were same as listed in the first case.

3. Then I tried the git log --patch-with-stat and git am option based on this answer https://stackoverflow.com/a/11426261/5497551 Result: This usually gives errors on encountering a merge, while applying the patch. I tried one of the suggestions to this answer of using -m --first-parent This resolved the errors but does not expand any merges into their commits, just lists the merge as a single commit. Hence most of the commit history is lost. So I added another option of --3way. This went over and over through the commits, and did not lead to any acceptable solution.

In conclusion, I would prefer using the 3rd solution, if only there was an option to have all the commits in a merge to be listed in the history of the new repo. Else I have to stick to the first solution which is a bit inconvenient and tedious in my situation. Any advice, help would be greatly appreciated.

Thanks.

Answer 1

Here is what worked for me(combining answers from @AD7six and @Olivier) to split my orig-repo into multiple new repos. I'm listing here steps for creating only one new repo new-repo1. But same have been used to create the others as well.

First create new empty repo on Github with the name new-repo1

git clone [Github url of orig-repo]

git clone --no-hardlinks orig-repo new-repo1
cd new-repo1
git remote rm origin
git checkout -b master  //This step can be skipped. I had to do it since the default branch on my orig-repo was `develop`, but on the new-repo1 I wanted to create it as `master`

//I used a script here to delete files and directories not required in the new-repo1. 
//But if you have very few files/dirs to be deleted then you can do the below.
git rm <path of file 1 to be deleted>   
git rm <path of file 2 to be deleted>
git rm -rf <path of dir 1 to be deleted>

git commit -m "Deleted non-new-repo1 code"

git ls-files > keep-these.txt
git filter-branch --force --index-filter "git rm  --ignore-unmatch --cached -qr . ; cat $PWD/keep-these.txt | xargs git reset -q \$GIT_COMMIT --" --prune-empty --tag-name-filter cat -- --all

rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --prune=now

git init
git remote add origin [Github url of new-repo1]
git push -u origin master

After this, I can view history of files in the new-repo1 on Github as well as through command line using git log

Answer 2

With method 1, do you clone from a local directory or a URL? If you clone from a local directory, you should use the --no-hardlinks option. Otherwise, what you do in one clone might affect the .git directories of the other ones, because git hard-linked files.

Here’s how I do it:

• Clone the local repository:

  git clone --no-hardlinks source_repo detached_repo
  

• In detached_repo, remove the origin (more information here to preserve branches other than the current one):

  git remote rm origin
  

• Remove tags you don’t want to keep. To remove all tags, use git tag -l | xargs git tag -d

• Use filter-branch to exclude the other files, so they can be pruned. Let's also add --tag-name-filter cat --prune-empty to remove empty commits and to rewrite tags (more information here if you have several branches to keep):

  git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter folder/to/keep HEAD
  

• Then delete the backup reflogs so the space can be truly reclaimed (now the operation is destructive):

  git reset --hard
  git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
  git reflog expire --expire=now --all
  git gc --aggressive --prune=now
  

  and now you have a local git repository of the folder/to/keep sub-directory with all its history preserved.

EDIT

Since you need to keep more than one subdirectory, I will assume that you have a list of files to keep in a file called files_to_keep. Then change the git filter-branch step to:

git filter-branch --tag-name-filter cat --prune-empty \
    --index-filter 'git ls-tree -z -r --name-only --full-tree $GIT_COMMIT \
    | grep -z -v -F -f /absolute/path/to/files_to_keep \
    | xargs -0 -r git rm --cached -r' HEAD

You can generate the list of files to keep by running this command:

git log --pretty=format: --name-status | cut -f2- | sort -u > all_files

and removing the files you don’t want to keep.

Answer 3

For such a scenario one might want to give git-import a try.

It basically creates patches from the given file or directory ($object) of one repo and applies them to another while keeping history.

cd old_repo
git format-patch --thread -o "$temp" --root -- "$object"

These patches then get applied to a new repository:

cd new_repo
git am "$temp"/*.patch 

(This procedure can be repeated for different parts of the old repository if needed.)

For details please look up:

• the documented source
• git format-patch
• git am