Setting up Gitlab using Docker on Windows host, issue with shared folders

Question

TLDR;

Does anyone know how to solve the "Failed asserting that ownership of "/var/opt/gitlab/git-data" was git" error?

Background:

I want to set up the Gitlab Docker on WindowsServer2012R2 running Docker toolbox, version 17.04.0-ce, build 4845c56.

Issue/Question

I can't get the shared folder to work properly on the D drive of the server. I read that I needed to add the folder to the VirtualBox VM, which I did via the settings/shared folder menu in the VB GUI. I set a name "gitlab" to the path "D:\data\gitlab" then checked auto-mount, make permanent, and set it to full access.

I started the docker machine and ran "docker-machine ssh $machine-name". I noticed that there was no /media directory and so I added a folder at the home directory (/home/docker/gitlab) and then mounted the shared folder using the following command I found in several forums:

sudo mount -t vboxsf gitlab /home/docker/gitlab

At this point I can add files to the Windows host directory or the Docker VM and it seems to work fine and the test files show up.

Now when I spin up the Gitlab Docker image, I use the following command modified from their documentation:

docker run --detach --hostname gitlab.example.com --publish 80:80 --name gitlab --volume /home/docker/gitlab:/etc/gitlab:Z --volume /home/docker/gitlab/logs:/var/log/gitlab:Z --volume /home/docker/gitlab/data:/var/opt/gitlab:Z gitlab/gitlab-ce

Now I know that it appears to be writing to the shared drive, because all of these files are generated, but then it crashes after a few seconds and I receive the following error log.

Error Log:

Thank you for using GitLab Docker Image!
Current version: gitlab-ce=9.3.6-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab vim /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

Installing gitlab.rb config...
Generating ssh_host_rsa_key...
Generating public/private rsa key pair.
Your identification has been saved in /etc/gitlab/ssh_host_rsa_key.
Your public key has been saved in /etc/gitlab/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:GyFlf9tl7ZuEbuE+dwZUYiyahdsRzpC1T7kwyUvoD+o root@gitlab.example.com
The key's randomart image is:
+---[RSA 2048]----+
|        o .+oo   |
|       o .o*+o+.o|
|      . . o*@+oo+|
|       . o+o.Oo= |
|        S o o++..|
|         + oo + o|
|        o   .+ + |
|       .    o. .o|
|        E    .o..|
+----[SHA256]-----+
Generating ssh_host_ecdsa_key...
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/gitlab/ssh_host_ecdsa_key.
Your public key has been saved in /etc/gitlab/ssh_host_ecdsa_key.pub.
The key fingerprint is:
SHA256:Kb99jG8EtMuTSdIuqBT3GLeD1D0wwTEcQhKgVJUlBjs root@gitlab.example.com
The key's randomart image is:
+---[ECDSA 256]---+
| .o+=*=+=+       |
|..  oo..=..      |
|.  E   . * .     |
|    o + +.B      |
|     +.BS* *     |
|    . +o= B .    |
|   . .  .o =     |
|    .    o. +    |
|        . .+.    |
+----[SHA256]-----+
Generating ssh_host_ed25519_key...
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/gitlab/ssh_host_ed25519_key.
Your public key has been saved in /etc/gitlab/ssh_host_ed25519_key.pub.
The key fingerprint is:
SHA256:lVxpu0UoyNPWVY6D9c+m/bUTyvKP6vuR4cTOYwQ0j+U root@gitlab.example.com
The key's randomart image is:
+--[ED25519 256]--+
|       . o +.=o..|
|        +.=o@o.+ |
|         o+=.Eo o|
|         .  + .o.|
|        S    B  +|
|            B o= |
|            .Oo +|
|           ..o+.+|
|          .+*+.oo|
+----[SHA256]-----+
Preparing services...
Starting services...
Configuring GitLab package...
/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 34: ulimit: max user processes: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
Configuring GitLab...

================================================================================
Error executing action `run` on resource 'ruby_block[directory resource: /var/opt/gitlab/git-data]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Failed asserting that ownership of "/var/opt/gitlab/git-data" was git
---- Begin output of set -x && [ "$(stat --printf='%U' $(readlink -f /var/opt/gitlab/git-data))" = 'git' ] ----
STDOUT:
STDERR: + readlink -f /var/opt/gitlab/git-data
+ stat --printf=%U /var/opt/gitlab/git-data
+ [ UNKNOWN = git ]
---- End output of set -x && [ "$(stat --printf='%U' $(readlink -f /var/opt/gitlab/git-data))" = 'git' ] ----
Ran set -x && [ "$(stat --printf='%U' $(readlink -f /var/opt/gitlab/git-data))" = 'git' ] returned 1

Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:124:in `validate_command'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:112:in `block in validate'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:111:in `each_index'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:111:in `validate'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:87:in `validate!'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:35:in `block (3 levels) in from_file'

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb

 26:   ruby_block "directory resource: #{params[:path]}" do
 27:     block do
 28:       # Ensure the directory exists
 29:       storage_helper.ensure_directory_exists(params[:path])
 30:
 31:       # Ensure the permissions are set
 32:       storage_helper.ensure_permissions_set(params[:path])
 33:
 34:       # Error out if we have not achieved the target permissions
 35:       storage_helper.validate!(params[:path])
 36:     end
 37:     not_if { storage_helper.validate(params[:path]) }
 38:   end
 39: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:26:in `block in from_file'

ruby_block("directory resource: /var/opt/gitlab/git-data") do
  params {:path=>"/var/opt/gitlab/git-data", :owner=>"git", :group=>nil, :mode=>"0700", :name=>"/var/opt/gitlab/git-data"}
  action [:run]
  retries 0
  retry_delay 2
  default_guard_interpreter :default
  block_name "directory resource: /var/opt/gitlab/git-data"
  declared_type :ruby_block
  cookbook_name "gitlab"
  recipe_name "gitlab-shell"
  block #<Proc:0x000000054a99a8@/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:27>
  not_if { #code block }
end

Platform:
---------
x86_64-linux

Does anyone know how to solve the "Failed asserting that ownership of "/var/opt/gitlab/git-data" was git" error? I'm still somewhat new to Docker/setting up Gitlab, so it's very possible I could have overlooked something simple. I've spent several hours Googling this, and it seems that others also have a lot of issues getting shared folders to work from Windows using the Docker Toolbox, so hopefully this will help others as well.

Answer 1

Background

One solution (maybe not the best) for those of us stuck in a world without native docker, is to use vdi drives and shared folders. The vdi drive can live on an drive we want (which is important if you don't want to use the C drive) and is used to allow the Gitlab docker the ability to chown anything it wants, so this is where we'll store the persistent volumes. The downside is that a vdi is not as transparent as a simple shared folder, thus for backups, a shared folder makes things a little bit easier/transparent.

Disclaimer

I'm not an expert on any of this, so please use caution and take what I say with a grain of salt.

Steps to perform

Create a new vdi drive and shared folder on any drive you'd like

1. Turn off your docker machine you want to use for gitlab
2. In virtualbox go into the settings on your docker-machine, then Storage, and click Add Hard Disk icon, then Create new disk
3. Select VDI (VirtualBox Disk Image) and click Next
4. Select Dynamically allocated and click Next
5. Select the name and location you want to store the vdi by clicking the folder with green carrot symbol, then select the max size the vdi can grow to, and click Create
6. Now in the settings menu, switch to Shared Folders and click Adds new shared folder icon
7. Create a gitlabbackups folder to where ever you want and select Auto-mount and Make Permanent

Now partition and format the drive

1. Start/enter the docker machine (either use VBox window or docker-machine ssh <your docker machine name> from cmd prompt)
2. Run fdisk -l to list the available drives, and if you've only mounted the one extra vdi drive, you should see something like /dev/sdb
3. The next steps are irreversible, so perform it at your own discretion: enter command fdisk /dev/sdb then n for new partition, p for primary, and 1
4. Now format the new partition (you might need sudo as well): mkfs.ext4 /dev/sdb1

Run docker with persistent volumes on second vdi and backups in shared folder

Sample Dockerfile:

FROM gitlab/gitlab-ce:latest

RUN apt-get update
RUN apt-get install -y cron

# Add a cron job to backup everyday
RUN echo "0 5 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create STRATEGY=copy CRON=1" | crontab -
# For an unknown reason, the cron job won't actually run unless cron is restarted
CMD service cron restart && \
    /assets/wrapper

Sample docker-compose.yml:

version: "3.0"
services:
  gitlab:
    build: .
    restart: always
    ports:
      - "80:80"
    volumes:
      # These volumes are on the vdi we created above
      - "/mnt/sdb1/etc/gitlab/:/etc/gitlab"
      - "/mnt/sdb1/var/log/gitlab:/var/log/gitlab"
      - "/mnt/sdb1/var/opt/gitlab:/var/opt/gitlab"
      # This volume sits in the shared folder defined above
      - "/gitlabbackups:/var/opt/gitlab/backups"
    cap_add:
      # These seem to be necessary for the mounted drive to work properly
      # https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
      - SYS_ADMIN
      - DAC_READ_SEARCH

Because there seems to be an issue with auto mounting the vdi, use a startup script, for example (assuming you used a D drive, just replace anything inside <...> as needed), sample run.bat:

@cd /d D:\<path to docker-compose.yml, assuming it's on the D drive>
@docker-machine start <docker machine name>
@FOR /f "tokens=*" %%i IN ('docker-machine env <docker machine name>') DO @%%i
@docker-machine ssh <docker machine name> sudo mount /dev/sdb1 /mnt/sdb1
@docker-compose build
@docker-compose up -d
@REM If the docker machine was completely off, running only 'docker-compose up -d' will 
@REM not mount the volumes properly. Stopping and restarting the container results in 
@REM the volumes mounting properly.
@docker stop <gitlab container name>
@docker start <gitlab container name>
@pause

Note: the gitlab container name can be found by running docker-compose up once and then docker ps -a to check it, but it's usually follows the convention <directory compose file is in>_<name in the compose file, e.g. gitlab here>_1

Assuming all went well and you change the stuff in the <...>'s above for your situation, you should be able to run the batch file and have gitlab up and running in such a way that it stores everything on the alternate drive, persistent working files in the vdi (to get around VBox POSIX limitations), and backups transparently stored in the shared folder.

Hope this helps other poor souls that don't have access to native docker yet.