8

I'm trying to invoke docker on my OSX host running Docker for Mac 17.06.0-ce-mac17 from inside a running jenkins docker container (jenkins:latest), per the procedure described at http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/.  

I mount /var/run/docker.sock into the container, I stick a ubuntu docker binary inside it, and it's able to execute - but from inside the container as user "jenkins" when I run e.g. "docker ps" I get

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.30/containers/json?all=1: dial unix /var/run/docker.sock: connect: permission denied.  

If I connect to the container as root (docker exec -u 0) it works though.

I need the jenkins user to be able to run this. I tried adding a docker group and adding jenkins to it inside the ubuntu container but that didn't help, since it's got nothing to do with the outside and Docker for Mac doesn't work like running this on linux where you can do semi easy uid/gid matching. I want to distribute this container so answers that go and hack part of my Docker for Mac install won't really help me. I'd rather not run the whole jenkins setup as root if I can help it. (I also tried running the container as privileged, that didn't help.)

Per the advice in Permission Denied while trying to connect to Docker Daemon while running Jenkins pipeline in Macbook I chowned the /var/run/docker.sock file inside the container manually to jenkins and now jenkins can run docker. But I'm having trouble coming up with a solution for a distributable container - I can't do that chown in the Dockerfile because the file doesn't exist yet, and shimming in into the entrypoint doesn't help because that runs as jenkins.

What do I need to do in order to build and run an image that will run external docker containers on my Mac as a non-root user from inside the container?

mxyzplk
  • 204
  • 1
  • 2
  • 9
  • Docker for Mac runs an Alpine Linux VM, and that's where everything you run actually is running. On my system, the user "docker" inside that VM is uid 1001. If you make Jenkins uid 1001 in your VM, does it work? – Dan Lowe Jul 09 '17 at 18:29
  • Actually I found a similar question from yesterday (!): https://stackoverflow.com/questions/44978017/permission-denied-while-trying-to-connect-to-docker-daemon-while-running-jenkins?rq=1 and I chowned /var/run/docker.sock to jenkins from inside the container and now jenkins can run docker commands on the host. It seems to just change the file ownership inside the container, not outside, which isn't what I expected, I was used to uids/gids matching up but I guess Docker for Mac, you so crazy. Just have to shim this into the entrypoint now I guess... Will self post an answer once it works. – mxyzplk Jul 09 '17 at 18:41
  • Hmm, this isn't going well. I can't do it in the Dockerfile because /var/run/docker.sock doesn't exist prior to VOLUME creation and you can't change it after VOLUME creation either because docker. And the entrypoint runs as jenkins so shimming it in there doesn't help. – mxyzplk Jul 09 '17 at 19:13
  • @DanLowe I also did try running it as user 1001 in the container, that didn't work though. – mxyzplk Jul 09 '17 at 19:16
  • 2
    Turns out on my Mac, /var/run/docker.sock is symlinked down into /Users//Library/Containers/com.docker.docker/Data/s60 and is uid and gid staff - so adding "RUN gpasswd -a jenkins staff" in the dockerfile makes it work - on a Mac running Docker for Mac at least, probably not on any other config, so I'll leave this open looking for a more general solution (so anyone on a different config can consume the image). – mxyzplk Jul 10 '17 at 14:04
  • I forgot about this quirk of osxfs permissions. See my answer here for more information on how osxfs deals with volume permissions. It's not the same way it behaves on Linux systems. https://stackoverflow.com/a/43213455/2449905 – Dan Lowe Jul 10 '17 at 14:22
  • Also there are docs on the docker web site for this - https://docs.docker.com/docker-for-mac/osxfs/#ownership – Dan Lowe Jul 10 '17 at 14:23
  • Yeah I found that, turns out the chowned ownership lasts even through deleting images and volumes but of course, not through docker daemon restarts since the file gets removed and readded :-( – mxyzplk Jul 10 '17 at 17:08
  • A solution I found in a blogpost is to spin up a socat container that creates a bi directional stream between endpoints https://mingheng.medium.com/solving-permission-denied-while-trying-to-connect-to-docker-daemon-socket-from-container-in-mac-os-600c457f1276 – Arne Jenssen Dec 04 '20 at 16:10

3 Answers3

1

I got this working, at least automated but currently only working on docker for Mac. Docker for Mac has a unique file permission model. Chowning /var/run/docker.sock to the jenkins user manually works, and it persists across container restarts and even image regeneration, but not past docker daemon restarts. Plus, you can't do the chown in the Dockerfile because docker.sock doesn't exist yet, and you can't do it in the entrypoint because that runs as jenkins.

So what I did was add jenkins to the "staff" group, because on my Mac, /var/run/docker.sock is symlinked down into /Users//Library/Containers/com.docker.docker/Data/‌​s60 and is uid and gid staff. This lets the jenkins user run docker commands on the host.

Dockerfile:

FROM jenkins:latest

USER root

RUN \
    apt-get update && \
    apt-get install -y build-essential && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

COPY docker /usr/bin/docker

# To allow us to access /var/run/docker.sock on the Mac
RUN gpasswd -a jenkins staff

USER jenkins

ENTRYPOINT ["/bin/tini", "--", "/usr/local/bin/jenkins.sh"]

docker-compose.yml file:

version: "3"
services:
  jenkins:
    build: ./cd_jenkins
    image: cd_jenkins:latest
    ports:
      - "8080:8080"
      - "5000:5000"
    volumes:
      - ./jenkins_home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock

This is, however, not portable to other systems (and depends on that docker for mac group staying "staff," which I imagine isn't guaranteed). I'd love suggested improvements to make this solution work across host systems. Other options suggested in questions like Execute docker host command inside jenkins docker container include:

  • Install sudo and let jenkins sudo and run all docker commands with sudo: adds security issues
  • "Add jenkins to the docker group" - UNIX only and probably relies on matching up gids from host to container right?
  • Setuid'ing the included docker executable might work, but has the same security elevation issues as sudo.
mxyzplk
  • 204
  • 1
  • 2
  • 9
1

Follow this: https://forums.docker.com/t/mounting-using-var-run-docker-sock-in-a-container-not-running-as-root/34390

Basically, all you need to do is to change /var/run/docker.sock permissions inside your container and run the docker with sudo.

I've created a Dockerfile that can be used to help:

FROM jenkinsci/blueocean:latest

USER root
# change docker sock permissions after moutn
RUN if [ -e /var/run/docker.sock ]; then chown jenkins:jenkins /var/run/docker.sock; fi
Tomer
  • 890
  • 9
  • 15
1

Another approach that worked for me - set the uid argument to the uid that owns /var/run/docker.sock (501 in my case). Not sure of the syntax for Dockerfile, but for docker-compose.yml, it's like this:

version: 3
services:
  jenkins:
    build:
      context: ./JENKINS
      dockerfile: Dockerfile
      args:
        uid: 501
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ...

Note this is based on using a Dockerfile to build the jenkins image, so many details left out. The key bit here is the uid: 501 under args.

ash
  • 4,078
  • 1
  • 17
  • 29