Getting permission denied even as root inside the docker container

Question

Connecting to running docker container as a root still gets Operation not permitted error when trying to apt-get update, yet I can still see sensitive file like /etc/passwd. Below are my configurations and also the error message from apt-get update. My host operating system is Ubuntu 18.04.3. My docker version is Docker version 19.03.5, build 633a0ea838

I create a container with the following Dockerfile

FROM python:3.8-slim-buster
RUN useradd -ms /bin/bash andrej
WORKDIR /home/andrej
COPY . /home/andrej/

RUN apt-get update && \
    apt-get install -y gcc && \
    pip install -r requirements.txt && \
    apt-get remove -y gcc && apt-get -y autoremove

RUN chown andrej:andrej pycurl && \
    chmod 0744 pycurl

USER andrej
ENTRYPOINT ["uwsgi"]
CMD ["--ini", "uwsgi.ini"]

starting it with docker compose looking like this:

version: "3.3"

services:

  andrej-cv:
    build: ./andrej_cv
    container_name: andrej-cv
    restart: always
    security_opt:
      - no-new-privileges
    expose:
      - 5000
    healthcheck:
      test: ./pycurl --host=127.0.0.1 --port=5050 --uri=/health_check
      interval: 1m30s
      timeout: 10s
      retries: 3

My docker daemon config:

{
    "icc": false,
    "userns-remap": "default",
    "log-driver": "syslog",
    "live-restore": true,
    "userland-proxy": false,
    "no-new-privileges": true
}

I connect to the container with following command (as root): docker exec -it -u root <container_hash> /bin/bash but when I try to update I got the following:

root@ed984abff684:/home/andrej# apt-get update 
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Hit:1 http://deb.debian.org/debian buster InRelease
Ign:2 http://deb.debian.org/debian buster-updates InRelease
Err:4 http://deb.debian.org/debian buster-updates Release
  Could not open file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - open (13: Permission denied) [IP: 151.101.36.204 80]
Hit:3 http://security-cdn.debian.org/debian-security buster/updates InRelease
rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
W: Download is performed unsandboxed as root as file '/var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - PrepareFiles (13: Permission denied)
E: The repository 'http://deb.debian.org/debian buster-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Problem unlinking the file /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease - PrepareFiles (13: Permission denied)

In the container /etc/subuid and /etc/subgid look like this (both):

andrej:100000:65536

On the host /etc/subuid and /etc/subgid look like this (both):

andrej:100000:65536
dockremap:165536:65536

Apparmor is running on Ubuntu host with following status (only docker-default profile):

andrej@machine:/etc/apparmor.d$ sudo aa-status 
apparmor module is loaded.
38 profiles are loaded.
36 profiles are in enforce mode.
   /sbin/dhclient
   /snap/core/8268/usr/lib/snapd/snap-confine
   /snap/core/8268/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/evince
   /usr/bin/evince-previewer
   /usr/bin/evince-previewer//sanitized_helper
   /usr/bin/evince-thumbnailer
   /usr/bin/evince//sanitized_helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/cups/backend/cups-pdf
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/cups-browsed
   /usr/sbin/cupsd
   /usr/sbin/cupsd//third_party
   /usr/sbin/ippusbxd
   /usr/sbin/tcpdump
   docker-default
   libreoffice-senddoc
   libreoffice-soffice//gpg
   libreoffice-xpdfimport
   man_filter
   man_groff
   snap-update-ns.core
   snap-update-ns.gnome-calculator
   snap-update-ns.gnome-characters
   snap-update-ns.gnome-logs
   snap-update-ns.gnome-system-monitor
   snap.core.hook.configure
   snap.gnome-calculator.gnome-calculator
   snap.gnome-characters.gnome-characters
   snap.gnome-logs.gnome-logs
   snap.gnome-system-monitor.gnome-system-monitor
2 profiles are in complain mode.
   libreoffice-oopslash
   libreoffice-soffice
17 processes have profiles defined.
14 processes are in enforce mode.
   docker-default (1101) 
   docker-default (1102) 
   docker-default (1111) 
   docker-default (1600) 
   docker-default (1728) 
   docker-default (1729) 
   docker-default (1730) 
   docker-default (1731) 
   docker-default (1732) 
   docker-default (1798) 
   docker-default (1799) 
   docker-default (1800) 
   docker-default (1801) 
   docker-default (1802) 
0 processes are in complain mode.
3 processes are unconfined but have a profile defined.
   /sbin/dhclient (491) 
   /usr/sbin/cups-browsed (431) 
   /usr/sbin/cupsd (402)

Selinux seems to be disabled as there is not /etc/selinux/config file and getenfoce and sestatus command are not available.

Also su andrej command run as root (where andrej is unprivileged user in the container) errors out with su: cannot set groups: Operation not permitted

Check updated question. Also I tried to `systemctl stop apparmor`, `systemctl disable apparmor`, `reboot`, but the issue persists. — scagbackbone, Feb 04 '20 at 08:41
Nope. I just removed `userns-remap` to make it work. But was not quite happy with it as it is not solution. — scagbackbone, Nov 30 '20 at 10:30
May have something to do with the security option `no-new-privileges`. See my post below for more details. — yaobin, May 10 '21 at 16:15

score 2 · Answer 1 · answered Nov 27 '20 at 09:03

I had exatly the same issues when running Ubuntu-16.04-based container in rootless Podman with Manjaro as the host system.

TL;DR: try to rebuild the image. That helped in my case.

The issue is likely that Docker cannot map the /var/lib/apt/lists/package directory's owner (_apt) UID to host's UID namespace. This might happen if the /etc/sub{u,g}id is modified after the image is pulled/built.

This is only a guess but the reason might be that Docker performs the UID map first for the image and then modifies /etc/sub{u,g}id resulting in different UID map rules -> Docker cannot map the user inside the container.

You can verify this by running docker inspect <image name> and checking the directories in "LowerDir" part. In one of those there should exist a directory var/lib/apt/lists/package with UID outside of the range specified for dockremap in /etc/sub{u,g}id. The exact command for podman was podman image inspect <image name> --format '{{.GraphDriver.Data.LowerDir}}' but the CLI APIs of Podman and Docker should be identical so the same command should work with Docker also.

E.g. I had an entry tlammi:100000:65536 in /etc/sub{u,g}id but /var/lib/apt/lists/package was owned by UID 165538 in host side which is outside of range [100000, 100000 + 65536).

score 1 · Answer 2 · answered Feb 02 '21 at 12:47

We run into the same issue with rootless podman.

We changed the subuid/subgid range of the user. This means one would need to fix the files stored with the old ranges or just delete the temporary files from the container storage directory:

$ podman info|grep GraphRoot
GraphRoot: /opt/tmp/container_graphroot/.local/share/containers/storage
$ rm -rf /opt/tmp/container_graphroot/.local/share/containers/storage/*

score 0 · Answer 3 · answered May 10 '21 at 16:14

This is not an answer yet (as of 2021-05-10) but my current research on this issue so far. Hopefully, it would give others hints about where to look further. Maybe I'll come back to edit this post to make it a real answer.

As far as I see, the "issue" is caused by the use of the security option no-new-privileges. Note that it is specified in OP's docker-compose file and the Docker daemon's configuration file.

Here is its description in the Docker's doc:

--security-opt="no-new-privileges:true" Disable container processes from gaining new privileges

...

If you want to prevent your container processes from gaining additional privileges, you can execute the following command:

$ docker run --security-opt no-new-privileges -it centos bash

no-new-privileges is a flag that was added to Linux kernel since 3.5. Here is its doc:

The no_new_privs bit (since Linux 3.5) is a new, generic mechanism to make it safe for a process to modify its execution environment in a manner that persists across execve. Any task can set no_new_privs. Once the bit is set, it is inherited across fork, clone, and execve and cannot be unset. With no_new_privs set, execve() promises not to grant the privilege to do anything that could not have been done without the execve call. For example, the setuid and setgid bits will no longer change the uid or gid; file capabilities will not add to the permitted set, and LSMs will not relax constraints after execve.

Notice the last sentence of "the setuid and setgid bits will no longer change the uid or gid". This may be why you would see the following error messages:

E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)

I found an article that talks about it with examples clearly: Running Docker application containers more securely.

My current thoughts:

I don't call the failed "apt-get update" an "issue" or a "problem" because that should be an intentional behavior for security consideration. In other words, it's a good thing.
Because the quoted kernel doc says "Once the bit is set, it ... cannot be unset", I believe you won't be able to "fix" it in the existing containers.
I don't think removing no-new-privileges is the right solution. At least it's not right before you fully discuss it with your team.
Alternatively, create a container without no-new-privileges for testing purpose only.

Getting permission denied even as root inside the docker container

3 Answers3