Docker runs out of disk space even though containers are small

Question

I have installed Docker Toolbox for Mac OSX and running several containers inside. First two I created were with Cassandra and were running fine. After that I've created 2 Debian containers, connected to bash through docker terminal with the purpose to install Oracle JDK8.

At the point when I was about to extract java from the tarball - I've got a ton of "Cannot write: No space left on device" error messages during the execution of "tar" command.

I've checked the space:

$ docker ps -s

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                         NAMES               SIZE

9d8029e21918        debian:latest       "/bin/bash"              54 minutes ago      Up 54 minutes                                                     deb-2               620.5 MB (virtual 744 MB)

49c7a0e37475        debian:latest       "/bin/bash"              55 minutes ago      Up 55 minutes                                                     deb-1               620 MB (virtual 743.5 MB)

66a17af83ca3        cassandra           "/docker-entrypoint.s"   4 hours ago         Up 4 hours          7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp   node-2              40.16 MB (virtual 412.6 MB)

After seeing that output I noticed that one of my nodes with cassandra is missing. In went to check to Kitematic and found out that it is in the DOWN state and I can't start it: "Cannot write node . No space left on device" - error message shown for this attempt.

Are there any limits that Docker has to run the containers?

When I remove all my cassandra ones and leave just a couple of Debian - java is able to be extracted from the tar. So the issue is definitely in some Docker settings related to sizing.

What is the correct way to resolve the issue with space limits here?

UPDATE.

$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE cassandra latest 13ea610e5c2b 11 hours ago 374.8 MB debian jessie 23cb15b0fcec 2 weeks ago 125.1 MB debian latest 23cb15b0fcec 2 weeks ago 125.1 MB

The output of df -hi

$ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on none 251K 38K 214K 15% / tmpfs 251K 18 251K 1% /dev tmpfs 251K 12 251K 1% /sys/fs/cgroup tmpfs 251K 38K 214K 15% /etc/hosts shm 251K 1 251K 1% /dev/shm

`df -h 
Filesystem Size Used Avail Use% Mounted on 
none 1.8G 1.8G 0 100% 
/ tmpfs 1002M 0 1002M 0% 
/dev tmpfs 1002M 0 1002M 0% 
/sys/fs/cgroup tmpfs 1.8G 1.8G 0 100% 
/etc/hosts shm 64M 0 64M 0% /dev/shm`

Appreciate help.

please do not add additional information via 'comments'; instead add them to the question itself (and yes: please go and put that additional info into the question now and remove your comment). — umläute, Dec 23 '15 at 08:59

score 2 · Answer 1 · edited May 23 '17 at 10:29

2

Issue 18869 refers to a docker-machine memory allocation problem.

This can be tested on the fly with

vboxmanage controlvm default 4096

Since drivers/virtualbox/virtualbox.go#L344-L352 reloads the settings from HOME/.docker/machine/machines/default/config.json, it is best to record that new value in that file (as mentioned in this answer).

That "No space left on device" was seen in docker/machine issue 2285, where the vmdk image created is a dynamically allocated/grow at run-time (default), creating a smaller on-disk foot-print initially, therefore even when creating a ~20GiB vm, with --virtualbox-disk-size 20000 requires on about ~200MiB of free space on-disk to start with.
And the default memory is quite low.

Make sure you that don't have :

any more exited container that you could remove:

docker rm -v $(docker ps --filter status=exited -q 2>/dev/null) 2>/dev/null

any dangling images

docker rmi $(docker images --filter dangling=true -q 2>/dev/null) 2>/dev/null

(Those are the result of rebuild which makes intermediate images unused)

See also "How to remove old and unused Docker images"

Then make sure you don't have an inode exhaustion problem, as in issue 10613.
Check df -hi (with i for inodes)

connected to bash through docker terminal with the purpose to install Oracle JDK8.

Try instead to specify the installation in a Dockerfile and build an image with the JDK installed.

edited May 23 '17 at 10:29

Community

1
1

answered Dec 23 '15 at 07:05

VonC

1,042,979
435
3,649
4,283

I have just installed docker today and I don't have any unused and unnecessary containers... This is very weird because you can see on the snippet that I have pasted directly from the command line, that all the containers are very small. – experimenter Dec 23 '15 at 07:08
@experimenter all *running* containers. You need do to a `docker ps -a` to see them all. – VonC Dec 23 '15 at 07:09
I have that container, that doesn't show up by the command "docker ps -s", in the "exited" state. It is down. And I can't start it. But I do need it. I think the issue isn't in some unused containers that take space, the issue is more in the amount of space that should be increased.. What do you think? – experimenter Dec 23 '15 at 07:12
@experimenter can you do a `docker images`, to see the space use by storing those images? – VonC Dec 23 '15 at 07:14
@experimenter Can you check the ouput of `df -hi`? See https://github.com/docker/docker/issues/10613#issuecomment-119679265 – VonC Dec 23 '15 at 07:45
tried to specify the Dockerfile and build an image with JDK installed. However, it doesn't start even, because "no space left on device" – experimenter Dec 23 '15 at 18:21
@experimenter strange indeed. I follow https://github.com/docker/docker/issues/18869 closely. – VonC Dec 23 '15 at 20:21
anyways, thanks for your assistance, will try to find out from the github issue conversation... – experimenter Dec 24 '15 at 00:31

score 2 · Accepted Answer · answered Dec 27 '15 at 23:12

I have resolved this issue in docker somehow.

By default the memory for the docker is set to be 2048M by default.

First step I performed is stopping my docker machine:

$ docker-machine stop default

Then I went to the $HOME/.docker/machine/machines/default/config.json file and changed the "Memory" setting to be higher, i.e. 4096.

{
    "ConfigVersion": 3,
    "Driver": {
        "VBoxManager": {},
        "IPAddress": "192.168.99.102",
        "MachineName": "default",
        "SSHUser": "docker",
        "SSHPort": 59177,
        "SSHKeyPath": "/Users/lenok/.docker/machine/machines/default/id_rsa",
        "StorePath": "/Users/lenok/.docker/machine",
        "SwarmMaster": false,
        "SwarmHost": "tcp://0.0.0.0:3376",
        "SwarmDiscovery": "",
        "CPU": 1,
        "Memory": 4096, 
        "DiskSize": 204800,
        "Boot2DockerURL": "",
        "Boot2DockerImportVM": "",
        "HostDNSResolver": false,
        "HostOnlyCIDR": "192.168.99.1/24",
        "HostOnlyNicType": "82540EM",
        "HostOnlyPromiscMode": "deny",
        "NoShare": false,
        "DNSProxy": false
    },
    "DriverName": "virtualbox",
    "HostOptions": {
        "Driver": "",
        "Memory": 0,
        "Disk": 0,
        "EngineOptions": {
            "ArbitraryFlags": [],
            "Dns": null,
            "GraphDir": "",
            "Env": [],
            "Ipv6": false,
            "InsecureRegistry": [],
            "Labels": [],
            "LogLevel": "",
            "StorageDriver": "",
            "SelinuxEnabled": false,
            "TlsVerify": true,
            "RegistryMirror": [],
            "InstallURL": "https://get.docker.com"
        },
        "SwarmOptions": {
            "IsSwarm": false,
            "Address": "",
            "Discovery": "",
            "Master": false,
            "Host": "tcp://0.0.0.0:3376",
            "Image": "swarm:latest",
            "Strategy": "spread",
            "Heartbeat": 0,
            "Overcommit": 0,
            "ArbitraryFlags": [],
"config.json" [noeol] 75L, 2560C
            "Overcommit": 0,
            "ArbitraryFlags": [],
            "Env": null
        },
        "AuthOptions": {
            "CertDir": "/Users/lenok/.docker/machine/certs",
            "CaCertPath": "/Users/lenok/.docker/machine/certs/ca.pem",
            "CaPrivateKeyPath": "/Users/lenok/.docker/machine/certs/ca-key.pem",
            "CaCertRemotePath": "",
            "ServerCertPath": "/Users/lenok/.docker/machine/machines/default/server.pem",
            "ServerKeyPath": "/Users/lenok/.docker/machine/machines/default/server-key.pem",
            "ClientKeyPath": "/Users/lenok/.docker/machine/certs/key.pem",
            "ServerCertRemotePath": "",
            "ServerKeyRemotePath": "",
            "ClientCertPath": "/Users/lenok/.docker/machine/certs/cert.pem",
            "ServerCertSANs": [],
            "StorePath": "/Users/lenok/.docker/machine/machines/default"
        }
    },
    "Name": "default" 
}

Finally, started my docker machine again:

$ docker-machine start default

Great! +1. I have updated my own answer accordingly. – VonC Dec 28 '15 at 08:15 — VonC, Dec 28 '15 at 08:15

Docker runs out of disk space even though containers are small

2 Answers2

Linked