Why git is hanging on git commit with directory of ~1000 files

Question

I have a large nested directory with < 1000 files throughout. They are on an external hard drive exthd.

$ cd /Volumes/exthd/project
$ tree 10/44/
├── 0
│   └── record.yaml
├── 00
│   ├── 11
│   │   └── record.yaml
│   ├── 18
│   │   └── record.yaml
│   ├── 37
│   │   └── record.yaml
│   ├── 41
│   │   └── record.yaml
│   ├── 43
│   │   └── record.yaml
│   ├── 47
│   │   └── record.yaml
│   ├── 77
│   │   └── record.yaml
│   └── 83
│       └── record.yaml
├── 01
│   ├── 02
│   │   └── record.yaml
│   ├── 07
│   │   └── record.yaml
│   ├── 27
│   │   └── record.yaml
│   ├── 61
│   │   └── record.yaml
│   ├── 63
│   │   └── record.yaml
│   ├── 67
│   │   └── record.yaml
│   ├── 68
│   │   └── record.yaml
│   ├── 73
│   │   └── record.yaml
│   └── record.yaml
├── 02
│   ├── 17
│   │   └── record.yaml
│   ├── 33
│   │   └── record.yaml
│   ├── 38
│   │   └── record.yaml
│   ├── 47
│   │   └── record.yaml
│   ├── 58
│   │   └── record.yaml
│   ├── 63
│   │   └── record.yaml
│   ├── 77
│   │   └── record.yaml
│   └── 88
│       └── record.yaml
├── 02
│   ├── 01
│   │   └── record.yaml
│   ├── 41
│   │   └── record.yaml
│   ├── 48
│   │   └── record.yaml
│   ├── 73
│   │   └── record.yaml
│   ├── 83
│   │   └── record.yaml
│   └── record.yaml
├── 04
│   ├── 27
│   │   └── record.yaml
│   ├── 33
│   │   └── record.yaml
│   ├── 48
│   │   └── record.yaml
│   ├── 61
│   │   └── record.yaml
│   ├── 67
│   │   └── record.yaml
│   ├── 73
│   │   └── record.yaml
│   └── record.yaml
├── 05
│   ├── 08
│   │   └── record.yaml
│   ├── 11
│   │   └── record.yaml
│   ├── 17
│   │   └── record.yaml
│   ├── 38
│   │   └── record.yaml
│   ├── 41
│   │   └── record.yaml
│   ├── 53
│   │   └── record.yaml
│   ├── 58
│   │   └── record.yaml
│   ├── 71
│   │   └── record.yaml
│   └── 87
│       └── record.yaml
├── 06
│   ├── 07
│   │   └── record.yaml
│   ├── 11
│   │   └── record.yaml
│   ├── 31
│   │   └── record.yaml
│   └── 41
│       └── record.yaml
├── 07
│   ├── 43
│   │   └── record.yaml
│   ├── 67
│   │   └── record.yaml
│   ├── 78
│   │   └── record.yaml
│   ├── 88
│   │   └── record.yaml
│   └── record.yaml
├── 08
│   ├── 08
│   │   └── record.yaml
│   ├── 51
│   │   └── record.yaml
│   ├── 53
│   │   └── record.yaml
│   ├── 63
│   │   └── record.yaml
│   ├── 68
│   │   └── record.yaml
│   └── record.yaml
├── 08
│   ├── 01
│   │   └── record.yaml
│   ├── 67
│   │   └── record.yaml
│   ├── 73
│   │   └── record.yaml
│   ├── 77
│   │   └── record.yaml
│   └── 78
│       └── record.yaml
├── 1
│   └── record.yaml
├── 10
│   ├── 27
│   │   └── record.yaml
│   ├── 37
│   │   └── record.yaml
│   ├── 48
│   │   └── record.yaml
│   └── 88
│       └── record.yaml
├── 11
│   ├── 21
│   │   └── record.yaml
│   ├── 33
│   │   └── record.yaml
│   └── 47
│       └── record.yaml
├── 12
│   ├── 31
│   │   └── record.yaml
│   ├── 37
│   │   └── record.yaml
│   ├── 53
│   │   └── record.yaml
│   ├── 58
│   │   └── record.yaml
│   ├── 67
│   │   └── record.yaml
│   ├── 73
│   │   └── record.yaml
│   ├── 83
│   │   └── record.yaml
│   └── 87
│       └── record.yaml
├── 13
│   ├── 37
│   │   └── record.yaml
│   ├── 38
│   │   └── record.yaml
│   ├── 57
│   │   └── record.yaml
│   ├── 68
│   │   └── record.yaml
│   ├── 73
│   │   └── record.yaml
│   └── 88
│       └── record.yaml
├── 14
│   ├── 11
│   │   └── record.yaml
│   ├── 23
│   │   └── record.yaml
│   ├── 27
│   │   └── record.yaml
│   ├── 33
│   │   └── record.yaml
│   ├── 38
│   │   └── record.yaml
│   ├── 47
│   │   └── record.yaml
│   ├── 63
│   │   └── record.yaml
│   └── 88
│       └── record.yaml
├── 15
│   ├── 23
│   │   └── record.yaml
│   ├── 31
│   │   └── record.yaml
│   ├── 47
│   │   └── record.yaml
│   ├── 48
│   │   └── record.yaml
│   ├── 78
│   │   └── record.yaml
│   ├── 88
│   │   └── record.yaml
│   └── 87
│       └── record.yaml
├── 16
│   ├── 01
│   │   └── record.yaml
│   ├── 18
│   │   └── record.yaml
│   ├── 31
│   │   └── record.yaml
│   ├── 33
│   │   └── record.yaml
│   ├── 87
│   │   └── record.yaml
│   └── record.yaml
├── 17
│   ├── 27
│   │   └── record.yaml
│   ├── 87
│   │   └── record.yaml
│   ├── 88
│   │   └── record.yaml
│   └── record.yaml
├── 18
│   ├── 43
│   │   └── record.yaml
│   ├── 53
│   │   └── record.yaml
│   ├── 61
│   │   └── record.yaml
│   ├── 67
│   │   └── record.yaml
│   ├── 81
│   │   └── record.yaml
│   └── 87
│       └── record.yaml
├── 18
│   ├── 02
│   │   └── record.yaml
│   ├── 08
│   │   └── record.yaml
│   ├── 38
│   │   └── record.yaml
│   ├── 57
│   │   └── record.yaml
│   ├── 67
│   │   └── record.yaml
│   ├── 78
│   │   └── record.yaml
│   ├── 87
│   │   └── record.yaml
│   └── 88
│       └── record.yaml
├── 2
│   └── record.yaml
├── 20
│   ├── 27
│   │   └── record.yaml
│   ├── 53
│   │   └── record.yaml
│   └── 83
│       └── record.yaml
├── 21
│   ├── 18
│   │   └── record.yaml
│   └── 87
│       └── record.yaml
├── 22
│   ├── 02
│   │   └── record.yaml
│   ├── 08
│   │   └── record.yaml
│   ├── 13
│   │   └── record.yaml
│   ├── 31
│   │   └── record.yaml
│   ├── 38
│   │   └── record.yaml
│   ├── 61
│   │   └── record.yaml
│   ├── 73
│   │   └── record.yaml
│   ├── 87
│   │   └── record.yaml
│   └── 83
│       └── record.yaml
├── 23
│   ├── 02
│   │   └── record.yaml
│   ├── 17
│   │   └── record.yaml
│   ├── 47
│   │   └── record.yaml
│   ├── 58
│   │   └── record.yaml
│   └── 63
│       └── record.yaml
...~1000 or so files

I am trying to commit them to a large git repo:

$ git commit -v 10/44/ -m 'Autocommit'

But the spinner thing goes away in the terminal after about 3 seconds, and it just sits there for > 10 minutes (before I cancelled the process). No verbose output was seen either, even with specifying -v. Wondering how to get git to commit these files other than committing them one-by-one. Wondering why it is hanging without doing any processing, or if something is happening in the background. 1000 files should take at most 5 minutes in my experience, but this goes significantly longer. If I shorten the directory, same thing.

Is that less than 1000 files in the entire repo, or just this directory? If the latter, how many files are in the entire repo? Also, what's the speed of the drive on the external hard disk and what interface (e.g. USB 2.0) is being used to connect it? And what operating system and file system are being used? — bk2204, May 29 '19 at 01:12
There are 100 million files in the repo (most aren't added or committed, only about 10000 are committed because I'm trying to find out better ways of committing this many objects). 1000 files nested in this particular example directory. This is Mac, WD external hard drive USB 3.0. — Lokasa Mawati, May 29 '19 at 01:13
Every commit saves every file that's in the *index*. The stuff in your work-tree is irrelevant. How many files are in the index? (`git ls-index | wc -l`) Meanwhile, to see what your machine is doing, open the Activity Monitor and look at disk and process activity. — torek, May 29 '19 at 01:17
Oops, sorry, I do that all the time: it's `git ls-files` (which lists the contents of the index by default, hence my mental substitution). — torek, May 29 '19 at 02:42
Just to be sure : you wouldn't happen to have some hooks around the `commit` action ? (`pre-commit` or `post-commit`) — LeGEC, May 29 '19 at 07:08

score 2 · Answer 1 · answered May 29 '19 at 04:50

The first things to check is your Git version:

Git 2.20 added optimization in tree traversal which could impact your commit.
the commit graph introduced in GIt 2.18 can speed up the retrieval on latest commits to build upon the next one
the diff --historigram has been optimized in Git 2.19
Git pack-object size is smaller with Git 2.21
etc...

Why git is hanging on git commit with directory of ~1000 files

1 Answers1