139

I got some docker containers running on AWS EC2, the /var/lib/docker/overlay2 folder grows very fast in disk size.

I'm wondering if it is safe to delete its content? or if docker has some kind of command to free up some disk usage.

UPDATE:

I actually tried docker system prune -a already, which reclaimed 0Kb.

Also my /docker/overlay2 disk size is much larger than the output from docker system df

After reading docker documentation and BMitch's answer, I believe it is a stupid idea to touch this folder and I will try other ways to reclaim my disk space.

Uwe Keim
  • 36,867
  • 50
  • 163
  • 268
qichao_he
  • 2,009
  • 2
  • 13
  • 20
  • 2
    did you find any answer to this? I am still getting the same issue. – Saurabh_Jhingan Jun 01 '20 at 13:28
  • 2
    I ran `docker image prune --all` and then `docker system prune -a`. It has reclaimed my disk space by around 50 GB, which was being used up by files under /var/lib/docker/overlay2. But, `docker system prune -a` would have been enough. Also, my configuration specifics are : `OS: Ubuntu 20`, `Docker : 19.03.12` – Binita Bharati Sep 21 '20 at 10:23

12 Answers12

105

Docker uses /var/lib/docker to store your images, containers, and local named volumes. Deleting this can result in data loss and possibly stop the engine from running. The overlay2 subdirectory specifically contains the various filesystem layers for images and containers.

To cleanup unused containers and images, see docker system prune. There are also options to remove volumes and even tagged images, but they aren't enabled by default due to the possibility of data loss:

$ docker system prune --help

Usage:  docker system prune [OPTIONS]

Remove unused data

Options:
  -a, --all             Remove all unused images not just dangling ones
      --filter filter   Provide filter values (e.g. 'label=<key>=<value>')
  -f, --force           Do not prompt for confirmation
      --volumes         Prune volumes

What a prune will never delete includes running containers, logs on those containers, and filesystem changes made by those containers. Additionally, anything created outside of the normal docker folders may not be seen by docker during this garbage collection. This could be from some other app writing to this directory, or a previous configuration of the docker engine (e.g. switching from AUFS to overlay2, or possibly after enabling user namespaces).

What would happen if this advice is ignored and you deleted a single folder like overlay2 out from this filesystem? The container filesystems are assembled from a collection of filesystem layers, and the overlay2 folder is where docker is performing some of these mounts (you'll see them in the output of mount when a container is running). Deleting some of these when they are in use would delete chunks of the filesystem out from a running container, and likely break the ability to start a new container from an impacted image.

To completely refresh docker to a clean state, stop the docker engine (systemctl stop docker) and delete the entire directory (not just the overlay2 folder) with rm -rf /var/lib/docker, and restart docker (systemctl start docker). The engine will restart without any images, containers, volumes, user created networks, or swarm state.

BMitch
  • 148,146
  • 27
  • 334
  • 317
  • I'm not talking about deleting docker folder, but the content of its sub-folder, /overlay2 – qichao_he Oct 10 '17 at 17:16
  • 1
    The answer is the same (except you can exclude volumes). If you delete things in there and break it, you get to keep both pieces. Use the tools provided for this job. – BMitch Oct 10 '17 at 17:18
  • thanks for your reply, sir. But there is another folder under /docker, called /volumes, I would assume this is where they keep the volumes. So I'm still confused what is exactly in the /docker/overlay2 folder. – qichao_he Oct 10 '17 at 23:08
  • And I only have one volume currently with name 0a022b88970afe01dee3daf66825ea695b46f8785902a1604744d7fb126e7e59 and I do find the folder with the same name under /docker/volumes – qichao_he Oct 10 '17 at 23:10
  • 1
    The overlay2 folder should contain the filesystem layers needed for your images and containers. You're free to ignore this advice, but please don't ask me for advice on how to recover a failed system after you break it, particularly since I gave you a supported way to cleanup your filesystem. – BMitch Oct 10 '17 at 23:18
  • 33
    I tried `docker system prune -a`, which recovered 0kb space. Right now the case for me is that /docker/overlay2 disk size is much larger than the output from `docker system df`. That's the reason why I keep digging this issue. Again, thanks for your reply sir. I guess I need to read more about docker documentation or probably erase the docker entirely and restart it. I only have a postgres database need to keep, and I did mount it – qichao_he Oct 10 '17 at 23:23
  • `docker system prune -a -f ` worked for me on `Docker version 17.09.0-ce` but only after stopping all my containers which is a real bummer. Had anyone luck with a newer version? – Bruno Oct 28 '18 at 18:11
  • 11
    I will say that the "supported" way isn't working for me neither. Doing all the docker system prune -a, docker volume prune, docker image prune and docker container prune still leaves me with 80% of my disk being used by Docker. That's with all containers stopped. – Craig Brett Jan 07 '19 at 10:52
  • What if I need a fresh start and I really want to clear everything and start my apps again? – Francisco Souza Feb 13 '20 at 14:20
  • 1
    @franzisk to clear everything you would delete all of `/var/lib/docker` rather than a single sub-directory that would leave it in an inconsistent state. – BMitch Feb 13 '20 at 14:26
  • I get that this is the "official" answer, but it's quite simply buggy as hell and leaves behind huge amounts of data, particularly overlay2 diff folders for images and containers that are no longer around. My docker should be using about 100G, but it's using over 200G. – Novaterata May 07 '21 at 12:56
  • If you don't have any containers (including stopped), then a full prune should leave 0G. If you're seeing otherwise, I'd open an issue with docker showing no containers in a `docker container ls -a`, the prune command, and what is still using disk space. – BMitch May 07 '21 at 13:40
  • For an example of what happens when this advice is ignored, see https://stackoverflow.com/q/67591341/596285 – BMitch May 18 '21 at 22:38
71

I found this worked best for me:

docker image prune --all

By default Docker will not remove named images, even if they are unused. This command will remove unused images.

Note each layer in an image is a folder inside the /usr/lib/docker/overlay2/ folder.

Sarke
  • 2,009
  • 14
  • 19
  • 3
    'image prune' worked much better than 'system prune'. Thanks! – DavidG Apr 15 '20 at 13:24
  • 1
    Warning! This is pretty desctructive, as it removes all images of non-running containers. You will be re-building them for hours if they are yours and not yet pushed to the registry. But it still cannot go beyond what `docker system df` shows (you may be still out of space and that evil `overlay2` garbage dump needs to be nuked manually. – mirekphd Apr 18 '20 at 21:35
  • 2
    Well, yeah, it removes the images. – Sarke Apr 20 '20 at 03:16
  • Attention: In my case where I used docker swarm it also deleted all tagged images, even for running containers – velop Oct 18 '20 at 20:17
  • This worked but buyer beware this removes EVERYTHING that is not in a container. – jimh Nov 13 '20 at 00:39
  • @jimh how in the world this can remove EVERYTHING outside a container? such a wording is definitely wrong. my host OS is still there and it's definitely outside any docker container. – whyer Dec 02 '20 at 11:15
  • @whyer i assume this is sardonic wit. – jimh Dec 02 '20 at 21:04
  • @jimh no it isn't. you just haven't specified at all what exactly gets removed. obviously you didn't mean literally what you have said. but what actually you did mean - remains mistery. be specific – whyer Dec 03 '20 at 13:51
  • I'd argue the implication is clearly for all images with the command `image prune` – jimh Dec 04 '20 at 14:02
37

I had this issue... It was the log that was huge. Logs are here :

/var/lib/docker/containers/<container id>/<container id>-json.log

You can manage this in the run command line or in the compose file. See there : Configure logging drivers

I personally added these 3 lines to my docker-compose.yml file :

my_container:
  logging:
    options:
      max-size: 10m
Tristan
  • 371
  • 3
  • 3
  • 2
    Can you add a few lines from the link to the answer? – RtmY Apr 09 '19 at 07:49
  • Would be nice to also get info on how to identify which container is the one with the giant log file. I have a bunch of containers and log files, some are huge some are tiny. – Micah Zoltu Apr 18 '19 at 05:13
  • How's that answers the OP question?! – Slavik Meltser Oct 20 '19 at 11:39
  • 2
    This answer is a partial answer, especially if 'logs' were the problem (maybe we can improve it with some edits?). Until I saw this answer, I was about to start randomly deleting big directories from my overly full `overlay2`. In my case, the total capacity for `/var/lib/docker` was 50GB and 36GB of it was consumed by one file: `/var/lib/docker/overlay2//diff/var/log/faillog`. On the assumption that this file isn't central to keeping everything running, my short term hack is to just remove it (and maybe I'll also adjust my `docker-compose` too). – D. Woods Nov 12 '19 at 21:55
26

also had problems with rapidly growing overlay2

/var/lib/docker/overlay2 - is a folder where docker store writable layers for your container. docker system prune -a - may work only if container is stopped and removed.

in my i was able to figure out what consumes space by going into overlay2 and investigating.

that folder contains other hash named folders. each of those has several folders including diff folder.

diff folder - contains actual difference written by a container with exact folder structure as your container (at least it was in my case - ubuntu 18...)

so i've used du -hsc /var/lib/docker/overlay2/LONGHASHHHHHHH/diff/tmp to figure out that /tmp inside of my container is the folder which gets polluted.

so as a workaround i've used -v /tmp/container-data/tmp:/tmp parameter for docker run command to map inner /tmp folder to host and setup a cron on host to cleanup that folder.

cron task was simple:

  • sudo nano /etc/crontab
  • */30 * * * * root rm -rf /tmp/container-data/tmp/*
  • save and exit

NOTE: overlay2 is system docker folder, and they may change it structure anytime. Everything above is based on what i saw in there. Had to go in docker folder structure only because system was completely out of space and even wouldn't allow me to ssh into docker container.

user2932688
  • 1,157
  • 9
  • 20
  • Thank for this answer, we put in containers an old data-base/app that generate a lot of `/var/log/apache2/error.log`. I reset error.log and access.log and add a new volume to allow easiest management – bcag2 Oct 12 '20 at 12:15
  • 1
    Just a small nitpick: you should edit the crontab using the `crontab -e` command, which checks validity before saving. – programagor Apr 13 '21 at 01:31
  • Your answer was very useful for my investigations on this problem. Thanks. – EM90 May 12 '21 at 09:40
8

Backgroud

The blame for the issue can be split between our misconfiguration of container volumes, and a problem with docker leaking (failing to release) temporary data written to these volumes. We should be mapping (either to host folders or other persistent storage claims) all of out container's temporary / logs / scratch folders where our apps write frequently and/or heavily. Docker does not take responsibility for the cleanup of all automatically created so-called EmptyDirs located by default in /var/lib/docker/overlay2/*/diff/*. Contents of these "non-persistent" folders should be purged automatically by docker after container is stopped, but apparently are not (they may be even impossible to purge from the host side if the container is still running - and it can be running for months at a time).

Workaround

A workaround requires careful manual cleanup, and while already described elsewhere, you still may find some hints from my case study, which I tried to make as instructive and generalizable as possible.

So what happened is the culprit app (in my case clair-scanner) managed to write over a few months hundreds of gigs of data to the /diff/tmp subfolder of docker's overlay2

du -sch /var/lib/docker/overlay2/<long random folder name seen as bloated in df -haT>/diff/tmp

271G total

So as all those subfolders in /diff/tmp were pretty self-explanatory (all were of the form clair-scanner-* and had obsolete creation dates), I stopped the associated container (docker stop clair) and carefully removed these obsolete subfolders from diff/tmp, starting prudently with a single (oldest) one, and testing the impact on docker engine (which did require restart [systemctl restart docker] to reclaim disk space):

rm -rf $(ls -at /var/lib/docker/overlay2/<long random folder name seen as bloated in df -haT>/diff/tmp | grep clair-scanner | tail -1)

I reclaimed hundreds of gigs of disk space without the need to re-install docker or purge its entire folders. All running containers did have to be stopped at one point, because docker daemon restart was required to reclaim disk space, so make sure first your failover containers are running correctly on an/other node/s). I wish though that the docker prune command could cover the obsolete /diff/tmp (or even /diff/*) data as well (via yet another switch).

It's a 3-year-old issue now, you can read its rich and colorful history on Docker forums, where a variant aimed at application logs of the above solution was proposed in 2019 and seems to have worked in several setups: https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604

mirekphd
  • 1,539
  • 12
  • 22
6

WARNING: DO NOT USE IN A PRODUCTION SYSTEM

/# df
...
/dev/xvda1      51467016 39384516   9886300  80% /
...

Ok, let's first try system prune

#/ docker system prune --volumes
...
/# df
...
/dev/xvda1      51467016 38613596  10657220  79% /
...

Not so great, seems like it cleaned up a few megabytes. Let's go crazy now:

/# sudo su
/# service docker stop
/# cd /var/lib/docker
/var/lib/docker# rm -rf *
/# service docker start
/var/lib/docker# df
...
/dev/xvda1      51467016 8086924  41183892  17% /
...

Nice! Just remember that this is NOT recommended in anything but a throw-away server. At this point Docker's internal database won't be able to find any of these overlays and it may cause unintended consequences.

Ravi Luthra
  • 127
  • 1
  • 3
  • 1
    Completely hosing the `/var/lib/docker` directory (while the daemon is stopped and assuming the directory contains no special filesystem mounts or similar) in fact is a valid quick-and-dirty way to get back to square one. I'm not sure why you're getting all the downvotes. Docker tries to be self-healing, and it will recognize when all hope is lost and reinitialize the `/var/lib/docker` directory as needed. – L0j1k Sep 09 '19 at 19:52
  • Holy **** finally a working answer. I've been pruning and doing stuff for 4 hours but i should have just stopped the docker service, put everything in the trash and the restarting it. – Osi Oct 09 '20 at 18:04
  • It _works_, but it also just deletes EVERYTHING that Docker has produced. So not a _good_ solution, per sé. – Akito Nov 23 '20 at 11:52
2

DON'T DO THIS IN PRODUCTION

The answer given by @ravi-luthra technically works but it has some issues!

In my case, I was just trying to recover disk space. The lib/docker/overlay folder was taking 30GB of space and I only run a few containers regularly. Looks like docker has some issue with data leakage and some of the temporary data are not cleared when the container stops.

So I went ahead and deleted all the contents of lib/docker/overlay folder. After that, My docker instance became un-useable. When I tried to run or build any container, It gave me this error:

failed to create rwlayer: symlink ../04578d9f8e428b693174c6eb9a80111c907724cc22129761ce14a4c8cb4f1d7c/diff /var/lib/docker/overlay2/l/C3F33OLORAASNIYB3ZDATH2HJ7: no such file or directory

Then with some trial and error, I solved this issue by running

(WARNING: This will delete all your data inside docker volumes)

docker system prune --volumes -a

So It is not recommended to do such dirty clean ups unless you completely understand how the system works.

Shankar Thyagarajan
  • 482
  • 5
  • 18
1

adding to above comment, in which people are suggesting to prune system like clear dangling volumes, images, exit containers etc., Sometime your app become culprit, it generated too much logs in a small time and if you using an empty directory volume (local volumes) this fill the /var partitions. In that case I found below command very interesting to figure out, what is consuming space on my /var partition disk.

du -ahx /var/lib | sort -rh | head -n 30

This command will list top 30, which is consuming most space on a single disk. Means if you are using external storage with your containers, it consumes a lot of time to run du command. This command will not count mount volumes. And is much faster. You will get the exact directories/files which are consuming space. Then you can go to those directories and check which files are useful or not. if these files are required then you can move them to some persistent storage by making change in app to use persistent storage for that location or change location of that files. And for rest you can clear them.

Dean Meehan
  • 2,321
  • 17
  • 33
Amit Bondwal
  • 103
  • 1
  • 2
  • 12
1

Friends, to keep everything clean you can use de commands:

  • docker system prune -a && docker volume prune.
Tiago Barreto
  • 19
  • 1
0

Everything in /var/lib/docker are filesystems of containers. If you stop all your containers and prune them, you should end up with the folder being empty. You probably don't really want that, so don't go randomly deleting stuff in there. Do not delete things in /var/lib/docker directly. You may get away with it sometimes, but it's inadvisable for so many reasons.

Do this instead:

sudo bash
cd /var/lib/docker
find . -type f | xargs du -b  | sort -n

What you will see is the largest files shown at the bottom. If you want, figure out what containers those files are in, enter those containers with docker exec -ti containername -- /bin/sh and delete some files.

You can also put docker system prune -a -f on a daily/weekly cron job as long as you aren't leaving stopped containers and volumes around that you care about. It's better to figure out the reasons why it's growing, and correct them at the container level.

Jason Hughes
  • 1,008
  • 7
  • 6
0

I recently had a similar issue, overlay2 grew bigger and bigger, But I couldn’t figure out what consumed the bulk of the space.

df showed me that overlay2 was about 24GB in size.

With du I tried to figure out what occupied the space… and failed.

The difference came from the fact that deleted files (mostly log files in my case) where still being used by a process (Docker). Thus the file doesn’t show up with du but the space it occupies will show with df.

A reboot of the host machine helped. Restarting the docker container would probably have helped already… This article on linuxquestions.org helped me to figure that out.

Omar Al-Howeiti
  • 790
  • 5
  • 14
mhe
  • 1
  • 1
-7

I used "docker system prune -a" it cleaned all files under volumes and overlay2

    [root@jasontest volumes]# docker system prune -a
    WARNING! This will remove:
            - all stopped containers
            - all networks not used by at least one container
            - all images without at least one container associated to them
            - all build cache
    Are you sure you want to continue? [y/N] y
    Deleted Images:
    untagged: ubuntu:12.04
    untagged: ubuntu@sha256:18305429afa14ea462f810146ba44d4363ae76e4c8dfc38288cf73aa07485005
    deleted: sha256:5b117edd0b767986092e9f721ba2364951b0a271f53f1f41aff9dd1861c2d4fe
    deleted: sha256:8c7f3d7534c80107e3a4155989c3be30b431624c61973d142822b12b0001ece8
    deleted: sha256:969d5a4e73ab4e4b89222136eeef2b09e711653b38266ef99d4e7a1f6ea984f4
    deleted: sha256:871522beabc173098da87018264cf3e63481628c5080bd728b90f268793d9840
    deleted: sha256:f13e8e542cae571644e2f4af25668fadfe094c0854176a725ebf4fdec7dae981
    deleted: sha256:58bcc73dcf4050a4955916a0dcb7e5f9c331bf547d31e22052f1b5fa16cf63f8
    untagged: osixia/openldap:1.2.1
    untagged: osixia/openldap@sha256:6ceb347feb37d421fcabd80f73e3dc6578022d59220cab717172ea69c38582ec
    deleted: sha256:a562f6fd60c7ef2adbea30d6271af8058c859804b2f36c270055344739c06d64
    deleted: sha256:90efa8a88d923fb1723bea8f1082d4741b588f7fbcf3359f38e8583efa53827d
    deleted: sha256:8d77930b93c88d2cdfdab0880f3f0b6b8be191c23b04c61fa1a6960cbeef3fe6
    deleted: sha256:dd9f76264bf3efd36f11c6231a0e1801c80d6b4ca698cd6fa2ff66dbd44c3683
    deleted: sha256:00efc4fb5e8a8e3ce0cb0047e4c697646c88b68388221a6bd7aa697529267554
    deleted: sha256:e64e6259fd63679a3b9ac25728f250c3afe49dbe457a1a80550b7f1ccf68458a
    deleted: sha256:da7d34d626d2758a01afe816a9434e85dffbafbd96eb04b62ec69029dae9665d
    deleted: sha256:b132dace06fa7e22346de5ca1ae0c2bf9acfb49fe9dbec4290a127b80380fe5a
    deleted: sha256:d626a8ad97a1f9c1f2c4db3814751ada64f60aed927764a3f994fcd88363b659
    untagged: centos:centos7
    untagged: centos@sha256:2671f7a3eea36ce43609e9fe7435ade83094291055f1c96d9d1d1d7c0b986a5d
    deleted: sha256:ff426288ea903fcf8d91aca97460c613348f7a27195606b45f19ae91776ca23d
    deleted: sha256:e15afa4858b655f8a5da4c4a41e05b908229f6fab8543434db79207478511ff7

    Total reclaimed space: 533.3MB
    [root@jasontest volumes]# ls -alth
    total 32K
    -rw-------  1 root root  32K May 23 21:14 metadata.db
    drwx------  2 root root 4.0K May 23 21:14 .
    drwx--x--x 14 root root 4.0K May 21 20:26 ..
Paul Richter
  • 10,414
  • 8
  • 47
  • 77
user1681424
  • 43
  • 5