git trying to clone a submodule's ref that doesn't exist

Question

I'm trying to clone a project with submodules, and I get a message like this one:

fatal: reference is not a tree: da3d99e1e6c554231cba76066a7b3149592e597e
Unable to checkout 'da3d99e1e6c554231cba76066a7b3149592e597e' in submodule path 'module_path'

Now, I'm not a first timer with git modules, and first I checked if the co-worker who pushed the main project, forgot somehow to push the module as well, and this is not the case.

Point is, not only the module in module_path doesn't have the specific commit anywhere (I've checked all remotes and all branches in each remote), but I can't find a single reference to said commit anywhere.

This is a (partial) list of the things I tried (I admit that I don't know much about the innards of a git repository, so my attempt are pretty general in scope):

1. checked the module in question, and unexpectedly is correctly checked out to the ref I would expect it to be (not the one that's giving me the error), which is the last commit that was made for the submodule
2. grepped the whole tree (including .git subdirs) for a file containing the offending reference
3. find on the tree (including .git and submodules) for a filename containing the offending ref.
4. git verify-pack on each and every pack file, grepping for the offending ref (nope: I do find the one I expect, though)
5. entered the submodule in question in .git/modules/<module name>, and done the aforementioned operations there too (not that I expected to find anything, as all previous operations where recursive)
6. mirrored the remote locally, cloned it, verified that the problem can be reproduced in this way too, and replicated all tests from 1 to 5 in the bare repository as well
7. tried dtruss (it's like strace, but on a Mac), without finding anything relevant on a cursory look
8. checked the reflog (not that I did expect anything). No dice.

I guess I could try force-pushing the current commit upward on the remote, but I would really like to understand what's going on, specifically: where does that ref comes from? Why do I get that error?

Thanks for the help!

UPDATE found a new "toy":

$ git cat-file -t da3d99e1e6c554231cba76066a7b3149592e597e
fatal: git cat-file da3d99e1e6c554231cba76066a7b3149592e597e: bad file

this happens both in the main git repository, and in the submodule's one (same message)

UPDATE2 found the command ls-tree and it doesn't report the offending ref for the specific submodule

$ git ls-tree master:<plugin_location> |grep <plugin_name>
160000 commit afa6b85e6aac7dfa13b74fb26fe68055864cdebe  <plugin_name>

but the one is actually checked out in the submodule :(

UPDATE3 turns out that the refs actually existed on my coworker's machine (they where in the reflog, that's why I didn't noticed them at first glance), and after a bit of fiddling I fixed the problem. The question still stands, though: where does git save the ref of the submodules?

Answer 1

where does git save the ref of the submodules?

git log --patch <submodule path> will show you changes to the submodule.

An example from one of my repos:

$ git log --patch tools

commit 3adfe4b33e4ade9da82983d268e60ca2e52268f9
Author: Alexander Groß <agross@therightstuff.de>
Date:   Wed Oct 22 16:50:12 2014 +0200

    The big Paket conversion

diff --git a/tools b/tools
index 84c72d3..c84d813 160000
--- a/tools
+++ b/tools
@@ -1 +1 @@
-Subproject commit 84c72d3cbe98e28d3b5c7643e7290085ec1551f7
+Subproject commit c84d813ebdae2fc9f2f14d5821cf820fbed42534

Git submodules store three bits of information:

1. The submodule URL in .gitmodules
2. The submodule path in .gitmodules
3. The submodule commit in your superproject commits (the one shown above)