255

I'm not clear on what the following means (from the Git submodule update documentation):

...will make the submodules HEAD be detached, unless --rebase or --merge is specified...

How does --rebase/--merge change things?

My main use case is to have a bunch of central repositories, which I will embed via submodules into other repositories. I would like to be able to improve on these central repositories, either directly in their original location, or from within their embedding repositories (the ones that use them via submodule).

  • From within these submodules, can I create branches/modifications and use push/pull just like I would in regular repositories, or are there things to be cautious about?
  • How would I advance the submodule referenced commit from say (tagged) 1.0 to 1.1 (even though the head of the original repository is already at 2.0), or pick which branch's commit is used at all?
Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
deepblue
  • 7,836
  • 12
  • 45
  • 59
  • On the topic of "detached head", see also http://stackoverflow.com/questions/964876/head-and-orighead-in-git and http://stackoverflow.com/questions/237408/pushing-an-existing-git-repository-to-github-only-sends-about-half-the-commits for a practical example (not submodule-related, but still) – VonC Dec 30 '09 at 10:39
  • "you cannot modify the contents of the submodule from within the main project": yes, true. And I have edited my answer to shed some light on that apparent contradiction (non-modifiable submodule, that you still can modify from the main project repo!) – VonC Dec 30 '09 at 14:04

4 Answers4

319

This GitPro page does summarize the consequence of a git submodule update nicely

When you run git submodule update, it checks out the specific version of the project, but not within a branch. This is called having a detached head — it means the HEAD file points directly to a commit, not to a symbolic reference.
The issue is that you generally don’t want to work in a detached head environment, because it’s easy to lose changes.
If you do an initial submodule update, commit in that submodule directory without creating a branch to work in, and then run git submodule update again from the superproject without committing in the meantime, Git will overwrite your changes without telling you. Technically you won’t lose the work, but you won’t have a branch pointing to it, so it will be somewhat difficult to retrieve.

Note March 2013:

As mentioned in "git submodule tracking latest", a submodule now (git1.8.2) can track a branch. 

# add submodule to track master branch
git submodule add -b master [URL to Git repo];

# update your submodule
git submodule update --remote 
# or (with rebase)
git submodule update --rebase --remote

See "git submodule update --remote vs git pull".

MindTooth's answer illustrate a manual update (without local configuration): 

git submodule -q foreach git pull -q origin master

In both cases, that will change the submodules references (the gitlink, a special entry in the parent repo index), and you will need to add, commit and push said references from the main repo.
Next time you will clone that parent repo, it will populate the submodules to reflect those new SHA1 references.

The rest of this answer details the classic submodule feature (reference to a fixed commit, which is the all point behind the notion of a submodule).

To avoid this issue, create a branch when you work in a submodule directory with git checkout -b work or something equivalent. When you do the submodule update a second time, it will still revert your work, but at least you have a pointer to get back to.

Switching branches with submodules in them can also be tricky. If you create a new branch, add a submodule there, and then switch back to a branch without that submodule, you still have the submodule directory as an untracked directory:

So, to answer your questions:

can I create branches/modifications and use push/pull just like I would in regular repos, or are there things to be cautious about?

You can create a branch and push modifications.

WARNING (from Git Submodule Tutorial): Always publish (push) the submodule change before publishing (push) the change to the superproject that references it. If you forget to publish the submodule change, others won't be able to clone the repository.

how would I advance the submodule referenced commit from say (tagged) 1.0 to 1.1 (even though the head of the original repo is already at 2.0)

The page "Understanding Submodules" can help

Git submodules are implemented using two moving parts:

  • the .gitmodules file and
  • a special kind of tree object.

These together triangulate a specific revision of a specific repository which is checked out into a specific location in your project.

From the git submodule page

you cannot modify the contents of the submodule from within the main project

100% correct: you cannot modify a submodule, only refer to one of its commits.

This is why, when you do modify a submodule from within the main project, you:

  • need to commit and push within the submodule (to the upstream module), and
  • then go up in your main project, and re-commit (in order for that main project to refer to the new submodule commit you just created and pushed)

A submodule enables you to have a component-based approach development, where the main project only refers to specific commits of other components (here "other Git repositories declared as sub-modules").

A submodule is a marker (commit) to another Git repository which is not bound by the main project development cycle: it (the "other" Git repo) can evolves independently.
It is up to the main project to pick from that other repo whatever commit it needs.

However, should you want to, out of convenience, modify one of those submodules directly from your main project, Git allows you to do that, provided you first publish those submodule modifications to its original Git repo, and then commit your main project refering to a new version of said submodule.

But the main idea remains: referencing specific components which:

  • have their own lifecycle
  • have their own set of tags
  • have their own development

The list of specific commits you are refering to in your main project defines your configuration (this is what Configuration Management is all about, englobing mere Version Control System)

If a component could really be developed at the same time as your main project (because any modification on the main project would involve modifying the sub-directory, and vice-versa), then it would be a "submodule" no more, but a subtree merge (also presented in the question Transferring legacy code base from cvs to distributed repository), linking the history of the two Git repo together.

Does that help understanding the true nature of Git Submodules?

VonC
  • 1,042,979
  • 435
  • 3,649
  • 4,283
  • by publishing you mean commiting? – deepblue Dec 30 '09 at 09:18
  • @deepblue: yes, pushing. I have added the link to the tutorial which illustrates both the normal workflow and the incorrect one. – VonC Dec 30 '09 at 09:25
  • interesting, the git-submodule man page says that the contents of the git submodule directory cannot be modified "you cannot modify the contents of the submodule from within the main project" – deepblue Dec 30 '09 at 11:28
  • 81
    Wow. That long an explanation for something that is principally that simple ought to be enough to scare any newcomer into just sticking with their svn:externals. ;-) – conny May 27 '10 at 09:26
  • 2
    @conny: but, as I detail in " [Why are git submodules incompatible with svn externals?](http://stackoverflow.com/questions/3131912/why-are-git-submodules-incompatible-with-svn-externals/3132221#3132221) ", submodules are *fundamental* different and not compatible with `svn:externals`. – VonC Jan 15 '11 at 14:48
  • So, if each submodule is tied to the specific commit that _was_ the current one at the time when I added the submodule: how would I go about updating (pull/fetch) that submodule to include the latest changes from upstream? For example, I'm keeping my dotfiles in a repository and inside that, I've added things like vim-plugins as submodules, which I rarely intend to tamper with, but nevertheless want to update on regular intervals. – Henrik Mar 10 '11 at 01:05
  • 1
    Sorry, to answer my own question, I gather cd'ing into the submodule and git checkout a sha, or git pull/fetch will do fine. Then commiting the update in the local repository. – Henrik Mar 10 '11 at 02:02
  • 2
    @hced: You can also hit all submodules at once using `git submodule foreach` – Dav Clark Jul 26 '11 at 00:57
  • 2
    .. still not getting it. is there an easier explanation on the web somewhere? – eugene Jan 10 '13 at 02:29
  • @Eugene it can help realizing this (git submodules) is really suited for component-based development (http://stackoverflow.com/a/933735/6309): what is the list of sources (with their exact revision) you need for your program to work? That list of precise point in history (http://stackoverflow.com/a/2300638/6309) of different groups of sources is represented by git submodules. – VonC Jan 10 '13 at 06:41
  • @IulianOnofrei Since git 1.8.2, `git submodule update --remote`, provided the submodule is configured to follow a branch. – VonC Dec 18 '17 at 08:51
  • So this answer is incomplete then. – Iulian Onofrei Dec 18 '17 at 08:53
  • 1
    @IulianOnofrei I have edited the answer to make that option more visible. – VonC Dec 18 '17 at 09:00
  • I would delete this answer in favor of his, instead of incorporating it here, as it doesn't seem fair. Also, his comment is the correct solution, not his answer. – Iulian Onofrei Dec 18 '17 at 09:03
  • 1
    @IulianOnofrei no, I would not delete that answer! The all point of this answer is to point out what a submodule actually is. But yes, if you are after the git command to blindly copy-paste, without realizing what you are doing, then MindTooh's answer is the right one for you. – VonC Dec 18 '17 at 09:05
139

To update each submodule, you could invoke the following command (at the root of the repository):

git submodule -q foreach git pull -q origin master

You can remove the -q option to follow the whole process.

Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
MindTooth
  • 4,272
  • 3
  • 17
  • 15
  • 15
    If you just run `git submodule update --init --recursive` from the root, it will get all of them recursively and initialize them if they aren't already. – Sam Soffes Oct 03 '13 at 05:23
  • 10
    @SamSoffes That serves a different purpose entirely. Updating submodules will check out the submodules at the commit they are **currently** pointing to (not necessarily the latest commit). The solution in the above answer updates the commit of each submodule to the latest HEAD from origin/master. – indragie Oct 04 '13 at 04:03
  • 8
    My new method: `git submodule update --rebase --remote` – MindTooth Feb 19 '15 at 16:22
19

To address the --rebase vs. --merge option:

Let's say you have super repository A and submodule B and want to do some work in submodule B. You've done your homework and know that after calling

git submodule update

you are in a HEAD-less state, so any commits you do at this point are hard to get back to. So, you've started work on a new branch in submodule B

cd B
git checkout -b bestIdeaForBEver
<do work>

Meanwhile, someone else in project A has decided that the latest and greatest version of B is really what A deserves. You, out of habit, merge the most recent changes down and update your submodules.

<in A>
git merge develop
git submodule update

Oh noes! You're back in a headless state again, probably because B is now pointing to the SHA associated with B's new tip, or some other commit. If only you had:

git merge develop
git submodule update --rebase

Fast-forwarded bestIdeaForBEver to b798edfdsf1191f8b140ea325685c4da19a9d437.
Submodule path 'B': rebased into 'b798ecsdf71191f8b140ea325685c4da19a9d437'

Now that best idea ever for B has been rebased onto the new commit, and more importantly, you are still on your development branch for B, not in a headless state!

(The --merge will merge changes from beforeUpdateSHA to afterUpdateSHA into your working branch, as opposed to rebasing your changes onto afterUpdateSHA.)

Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
robinspb
  • 341
  • 3
  • 4
9

Git 1.8.2 features a new option ,--remote, that will enable exactly this behavior. Running

git submodule update --rebase --remote

will fetch the latest changes from upstream in each submodule, rebase them, and check out the latest revision of the submodule. As the documentation puts it:

--remote

This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote-tracking branch.

This is equivalent to running git pull in each submodule, which is generally exactly what you want.

(This was copied from this answer.)

Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
Iulian Onofrei
  • 7,489
  • 8
  • 59
  • 96
  • If you decide to answer an older question that has well established and correct answers, adding a new answer late in the day may not get you any credit. If you have some distinctive new information, or you're convinced the other answers are all wrong, by all means add a new answer, but 'yet another answer' giving the same basic information a long time after the question was asked usually won't earn you much credit. There is no explanation of what this does — not even a link to external documentation (which wouldn't be sufficient). – Jonathan Leffler Dec 17 '17 at 21:15
  • 2
    It's not "yet another answer", as *NO* other answer has this command (prove me wrong). Other answers didn't work for me, this *comment* did, so I decided to post it as an answer while giving credit to the original owner. So consider removing your downvote. – Iulian Onofrei Dec 18 '17 at 08:44
  • There's a [comment](https://stackoverflow.com/questions/1979167/git-submodule-update/47836590?noredirect=1#comment45526594_9103113) by [MindTooth](https://stackoverflow.com/users/182868/mindtooth) from 2015 saying this is what they do now. You give no explanation of what this does (though you do mention MindTooth, but there's no real explanation of what you mean by that -— embedding URLs, as in this comment, would help). You don't say why it is a good idea. You don't give any caveats. This isn't, in my view, a helpful answer because it raises more questions than it resolves. – Jonathan Leffler Dec 18 '17 at 08:49
  • 1
    By that, I meant that **it works** instead of not working. Believe me, if more people were to see this answer, they would be glad, because **it works**. For things like this, most people just want to know the command that updates a git submodule, not how it's implemented. – Iulian Onofrei Dec 18 '17 at 08:52
  • I edited the answer to prove you wrong, also, https://stackoverflow.com/questions/1979167/git-submodule-update/1979194?noredirect=1#comment82694557_1979194 !!! – Iulian Onofrei Dec 18 '17 at 08:53
  • You've dramatically improved the answer — it would be sensible to do the editing before making comments so that I see your improvements, not the status quo ante. I'd prefer to see a discussion of the `--rebase` too. Reassuring that it works is good; explaining what it does helps too. – Jonathan Leffler Dec 18 '17 at 08:53
  • But you realise that is was clear for everybody that as of almost 2018, this was the correct command that achieves this simple task, right? – Iulian Onofrei Dec 18 '17 at 08:55
  • Definitively not "clear for everybody" — I don't realize that at all. I didn't know it. I'm not sure how big of a minority I'm in (if, indeed, I'm in a minority). The answer was not helpful to me because there was zero explanation of what it does. It is better; it still ain't perfect. It ain't my down-vote, so I can't undo it. It is my delete vote; I'm trying to remember/discover how to cancel that. – Jonathan Leffler Dec 18 '17 at 08:56