78

I have a folder, which was a git repo. It contains some files and .gitmodules file. Now, when I do git init and then git submodule init, the latter command output is nothing. How can I help git to see submodules, defined in .gitmodules file without running git submodule add by hand again?

Update: this is my .gitmodules file:

[submodule "vim-pathogen"]
    path = vim-pathogen
    url = git://github.com/tpope/vim-pathogen.git
[submodule "bundle/python-mode"]
    path = bundle/python-mode
    url = git://github.com/klen/python-mode.git
[submodule "bundle/vim-fugitive"]
    path = bundle/vim-fugitive
    url = git://github.com/tpope/vim-fugitive.git
[submodule "bundle/ctrlp.vim"]
    path = bundle/ctrlp.vim
    url = git://github.com/kien/ctrlp.vim.git
[submodule "bundle/vim-tomorrow-theme"]
    path = bundle/vim-tomorrow-theme
    url = git://github.com/chriskempson/vim-tomorrow-theme.git

and here is listing of this dir:

drwxr-xr-x  4 evgeniuz 100 4096 июня  29 12:06 .
drwx------ 60 evgeniuz 100 4096 июня  29 11:43 ..
drwxr-xr-x  2 evgeniuz 100 4096 июня  29 10:03 autoload
drwxr-xr-x  7 evgeniuz 100 4096 июня  29 12:13 .git
-rw-r--r--  1 evgeniuz 100  542 июня  29 11:45 .gitmodules
-rw-r--r--  1 evgeniuz 100  243 июня  29 11:18 .vimrc

so, definitely, it is in top level. the git directory is not changed, only git init is done

evgeniuz
  • 2,329
  • 5
  • 26
  • 35
  • Are the submodules already present, in the sense that if you change into any submodule directory there are files present, and `git rev-parse --show-toplevel` gives you the submodule rather than "supermodule" directory? – Mark Longair Jun 29 '12 at 09:07
  • no, submodule dirs are not present. imagine this folder is completely empty with just `.gitmodules` file – evgeniuz Jun 29 '12 at 09:09
  • Ah, I see what the problem is - I've updated my answer. – Mark Longair Jun 29 '12 at 09:18

7 Answers7

107

git submodule init only considers submodules that already are in the index (i.e. "staged") for initialization. I would write a short script that parses .gitmodules, and for each url and path pair runs:

git submodule add <url> <path>

For example, you could use the following script:

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        url=$(git config -f .gitmodules --get "$url_key")
        git submodule add $url $path
    done

This is based on how the git-submodule.sh script itself parses the .gitmodules file.

MrTux
  • 28,370
  • 24
  • 91
  • 123
Mark Longair
  • 385,867
  • 66
  • 394
  • 320
  • 1
    do you mean `git submodule update --init`? all of these commands fail silently. It looks like git doesn't see modules defined only in `.gitmodules` – evgeniuz Jun 29 '12 at 08:58
  • After the `git submodule init`, do you see any output from `git config --list | grep submodule`? (As the documentation says, `git submodule init` should "Initialize the submodules, i.e. register each submodule name and url found in .gitmodules into `.git/config`".) – Mark Longair Jun 29 '12 at 09:02
  • no, nothing. `git config --list` gives only standard values, no mention of submodules – evgeniuz Jun 29 '12 at 09:07
  • @Shark: that's strange - could you update your question with the contents of your `.gitmodules` file? Also, is your `.gitmodules` file definitely at the top level of your repository, rather than a subdirectory? – Mark Longair Jun 29 '12 at 09:08
  • Thanks, I thought there was more generic way than script, though :) – evgeniuz Jun 29 '12 at 09:36
  • @Shark: yeah, me too - and I may well be missing something obvious! However, I think the idea of initializing from `.gitmodules` is that it will have been committed in the same commit (or close to the same commit) as one that added the submodules to the tree, so having only `.gitmodules`, but no submodules in the index, is an unusual situation to be in. – Mark Longair Jun 29 '12 at 09:41
  • ```git submodule update --init -f``` - will fork submodules to it`s dir`s permomently. – iegik Jan 18 '13 at 12:24
  • Thank you for the script, @MarkLongair! – James Conroy-Finn Nov 01 '16 at 21:04
  • Thanks for the solution! Usability of git is worst I ever seen! – DenisKolodin Nov 02 '16 at 07:43
  • One of the problems here is that because the original `.git` is not available, you don't know which commit-id the submodule was pinned to, this makes it difficult to acquire the exact revision of the submodule that you had in the original repository. – CMCDragonkai Jan 05 '17 at 11:51
  • Has anyone got a solution for parsing the branch in the `.gitmodules` file and adding the submodule using the correct branch? – Fred Stark Jul 27 '17 at 02:45
  • Unfortunately, if the submodule contains spaces like after `git submodule add URL 'path with spaces'` this solution fails. Perhaps try something like `git config -f .gitmodules -z --get-regexp '^submodule\..*\.path$' | sed -z 's/\n.*$//' | tr '\0' '\n' | while read keys; ..` – Tino Nov 09 '17 at 20:17
  • If some submodules are already in the index, the script will exit. To avoid that: git submodule add $url $path || true – schwart Nov 19 '18 at 09:39
  • This was very useful. Thanks!! – pztrick Apr 16 '21 at 20:42
15

Expanding on @Mark Longair's answer, I wrote a bash script to automate steps 2 & 3 of the following process:

  1. Clone a 'boilerplate' repo to begin a new project
  2. Remove the .git folder and re-initialize as a new repo
  3. Re-initialize the submodules, prompting for input before deleting folders
#!/bin/bash

set -e
rm -rf .git
git init

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' > tempfile

while read -u 3 path_key path
do
    url_key=$(echo $path_key | sed 's/\.path/.url/')
    url=$(git config -f .gitmodules --get "$url_key")

    read -p "Are you sure you want to delete $path and re-initialize as a new submodule? " yn
    case $yn in
        [Yy]* ) rm -rf $path; git submodule add $url $path; echo "$path has been initialized";;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac

done 3<tempfile

rm tempfile

Note: the submodules will be checked out at the tip of their master branch instead of the same commit as the boilerplate repo, so you'll need to do that manually.

Piping the output from git config into the read loop was causing problems with the prompt for input, so it outputs it to a temp file instead. Any improvements on my first bash script would be very welcome :)

Big thanks to Mark, https://stackoverflow.com/a/226724/193494, bash: nested interactive read within a loop that's also using read, and tnettenba @ chat.freenode.net for helping me arrive at this solution!

Community
  • 1
  • 1
Kevin C.
  • 2,409
  • 1
  • 27
  • 40
7

Extending excellent @Mark Longair's answer to add submodule respecting branch and repo name.

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        name=$(echo $path_key | sed 's/\submodule\.\(.*\)\.path/\1/')
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        branch_key=$(echo $path_key | sed 's/\.path/.branch/')
        url=$(git config -f .gitmodules --get "$url_key")
        branch=$(git config -f .gitmodules --get "$branch_key" || echo "master")
        git submodule add -b $branch --name $name $url $path || continue
    done
mauron85
  • 1,084
  • 12
  • 22
  • 1
    This worked great, however it also duplicated each entry in the .gitmodules file which needs some manual cleanup after the fact. – Marcus Ottosson Aug 18 '20 at 07:02
3

I had a similar issue. git submodule init was failing silently.

When I did:

git submodule add <url> <path>

I got:

The following path is ignored by one of your .gitignore files: ...

I'm thinking that .gitignore(d) paths might be the cause.

Henry
  • 6,843
  • 2
  • 32
  • 35
2

An updated version of the script by @mark-longair . This one also supports branches, handles the case where some of the submodules already exist in .git/config, and when necessary backs up existing directories of the same name as the submodule paths.

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/');
        branch_key=$(echo $path_key | sed 's/\.path/.branch/');
        # If the url_key doesn't yet exist then backup up the existing
        # directory if necessary and add the submodule
        if [ ! $(git config --get "$url_key") ]; then
            if [ -d "$path" ] && [ ! $(git config --get "$url_key") ]; then
                mv "$path" "$path""_backup_""$(date +'%Y%m%d%H%M%S')";
            fi;
            url=$(git config -f .gitmodules --get "$url_key");
            # If a branch is specified then use that one, otherwise
            # default to master
            branch=$(git config -f .gitmodules --get "$branch_key");
            if [ ! "$branch" ]; then branch="master"; fi;
            git submodule add -f -b "$branch" "$url" "$path";
        fi;
    done;

# In case the submodule exists in .git/config but the url is out of date

git submodule sync;

# Now actually pull all the modules. I used to use this...
#
# git submodule update --init --remote --force --recursive
# ...but the submodules would check out in detached HEAD state and I 
# didn't like that, so now I do this...

git submodule foreach --recursive 'git checkout $(git config -f $toplevel/.gitmodules submodule.$name.branch || echo master)';
bokov
  • 3,205
  • 2
  • 28
  • 45
1

I know its been a while, but I want to share this version that calls git config only once, doesn't requires a script and also handles branches:

git config -f .gitmodules --get-regexp '^submodule\.' | perl -lane'
$conf{$F[0]} = $F[1]}{
@mods = map {s,\.path$,,; $_} grep {/\.path$/} keys(%conf);
sub expand{$i = shift; map {$conf{$i . $_}} qw(.path .url .branch)}
for $i (@mods){
    ($path, $url, $branch) = expand($i);
    print(qq{rm -rf $path});
    print(qq{git submodule add -b $branch $url $path});
}
'

The only side effect is the output of the commands, nothing gets executed, so you can audit before committing to them.

This works with a simple copy and paste at the console, but should be trivial to put in a shell script.

example output:

rm -rf third-party/dht
git submodule add -b post-0.25-transmission https://github.com/transmission/dht third-party/dht
rm -rf third-party/libutp
git submodule add -b post-3.3-transmission https://github.com/transmission/libutp third-party/libutp
rm -rf third-party/libb64
git submodule add -b post-1.2.1-transmission https://github.com/transmission/libb64 third-party/libb64
rm -rf third-party/libnatpmp
git submodule add -b post-20151025-transmission https://github.com/transmission/libnatpmp third-party/libnatpmp
rm -rf third-party/miniupnpc
git submodule add -b post-2.0.20170509-transmission https://github.com/transmission/miniupnpc third-party/miniupnpc
rm -rf third-party/libevent
git submodule add -b post-2.0.22-transmission https://github.com/transmission/libevent third-party/libevent
alexgirao
  • 815
  • 7
  • 9
0

For zsh users, try my function which has DRY_RUN=1 support to see what commands will be ran and only uses git to parse the file instead of sed.

gsub_file() {(
  set -eu

  cd "$(git rev-parse --show-toplevel)"

  submodule_paths=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "path" | awk '{ print $2 }')}"
  )
  submodule_urls=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "url" | awk '{ print $2 }')}"
  )
  submodule_branches=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "branch" | awk '{ print $2 }')}"
  )

  sh_c() {
    echo + "$*"
    if [ "${DRY_RUN-}" ]; then
      return
    fi
    eval "$@"
  }

  for (( i=1; i <= ${#submodule_paths[@]}; i++ )) do
    p="${submodule_paths[$i]}"
    if [ -d "$p" ]; then
      continue
    fi

    url="${submodule_urls[$i]}"
    unset b
    if [ "${submodule_branches[$i]-}" ]; then
      b="-b ${submodule_branches[$i]}" 
    fi
    sh_c git submodule add "${b-}" "$url" "$p"
  done
)}
nhooyr
  • 885
  • 9
  • 28