For my company I have to migrate all our company repositories from Github to our self hosted Gitlab-Server.
The main problem I have is that we use Git-LFS for the most of our repositories.
Therefore when cloning a repository I have to repair the Git-LFS by running
git lfs fetch --all
git lfs push remoteOnGitlab
I'm writing on a script for that - since it are a lot of repositories - using the Github API to work off a list of all our repositories:
#!/bin/bash
###_Adjust those_#######################################################
githubUrlStart="https://"
githubLink="github.com/<orgaName>"
githubApiLink="api.github.com/orgs/<orgaName>/repos"
# maxRepoCount is needed if the organisation has more than 30 repos
# otherwise only the first 30 repos would be mirrored
maxRepoCount="200"
gitlabUrlStart="http://"
gitlabLink="<gitlabURL>/<groupName>"
########################################################################
###_Get Sensible Data at runtime_###
bold=$(tput bold)
normal=$(tput sgr0)
echo -n "${bold}Github Username: ${normal}"
read -e githubName
echo -n "${bold}Github Password: ${normal}"
read -e -s githubPassword
echo ""
echo -n "${bold}Gitlab Username: ${normal}"
read -e gitlabName
echo -n "${bold}Gitlab Password: ${normal}"
read -e -s gitlabPassword
###_Add folder for cloned git repos_###
mkdir gits
###_GET JSON of all REPOS from the Github API_###
# writes output to file test for later use
curl "$githubUrlStart${githubName//\"}:${githubPassword//\"}@${githubApiLink//\"}?per_page=${maxRepoCount//\"}" > test
###_Get URLS and NAME_###
# -> http://xmodulo.com/how-to-parse-json-string-via-command-line-on-linux.html #
NAMES=$(cat test | jq ".[] | .name");
###_Transform into Array_###
NAMES_ARRAY=()
COUNT=0
for name in $NAMES
do
NAMES_ARRAY[$COUNT]=$name
COUNT=$((COUNT+1))
done
###_Clone each repo local and use it to repair git lfs_###
COUNT=0
for reponame in "${NAMES_ARRAY[@]}"
do
###_Build full link for cloning_###
fulllink="$githubUrlStart$githubName:$githubPassword@$githubLink/${reponame//\"}"
###_Clone from github using git lfs_###
git lfs clone "${fulllink//\"}" "gits/${reponame//\"}"
###_Enter folder of cloned repository_###
cd "gits/${reponame//\"}"
###_repair git lfs_###
# get lfs from github
git lfs fetch --all
# add the remote for gitlab
gitlabRemote="$gitlabUrlStart$gitlabName:$gitlabPassword@$gitlabLink/${reponame//\"}.git"
git remote add gitlab "${gitlabRemote//\"}"
# Just for security remove the github remote
git remote remove origin
# Push lfs to the gitlab remote
git lfs push gitlab --all
##_leave folder!_##
cd ../.. #
##_remove folder to save disk space_##
rm -rf gits/${reponame//\"}
COUNT=$((COUNT+1))
done
###_Remove the gits folder_####
rm -rf gits
My problem here:
It seems this works fine only once, but the repositories get not updated to the newest state if I run the script over them again. Also I'm not sure but I guess it will only work on the master branch so far.
What do I have to change in order to allways get the newest state of the git repo from github also on gitlab? (After running it I still have the state of from 2 months ago on Gitlab...)
Alternative
I also have seen another solution here which apparently would handle the multibranch problem:
git clone --mirror path/to/original path/to/dest/.git
cd path/to/dest
git config --bool core.bare false
git checkout anybranch
which I probably also could combine with
git push --mirrow path/to/otherRemote
as mentioned there in the comments.
This would ofcourse be way simplier .. BUT:
Is this compatible with LFS? let's say e.g. as
git lfs clone --mirrow /originOnGithub /localFolder
cd /localFolder
git lfs fetch --all
git lfs push --mirrow /remoteOnGitlab
