On our docker implementation on GCE, we are running out of space on the root file system.
Since images themselves are stored on a separate 1TB volume, the images themselves shouldn't be the problem.
One candidate are the centralized logfiles that Docker itself stores (a json file somewhere?), does anyone know where those files/file are/is located, and how we can logrotate/truncate them?
First, I'm using docker 1.1.2 for both client and server, this answer may be obsolete for newer versions of docker as docker evolve quickly.
Find your docker directory. On systems that use apt/debian style system, the package installed by the docker repository https://get.docker.com/ubuntu use /var/lib/docker. Chances are that directory is in the same place on other systems (can't confirm).
under containers/**CONTAINER_ID** you'll find infos about the container.
In the file **CONTAINER_ID**-json.log in that folder, you'll find a file with all the logs for that container. It may look like a json file, it's not. It's a flow a json structures, one per line, each containing one log line (each line ends by a } and the next one starts with a {, thus it's not a valid json as a whole).
Example location:
- /var/lib/docker/containers/05b6053c41a2130afd6fc3b158bda4e605b6053c41a2130afd6fc3b158bda4e6/05b6053c41a2130afd6fc3b158bda4e605b6053c41a2130afd6fc3b158bda4e6-json.log
I suggest you to use that path to see wether or not it's the reason why you're running out of space, but not to log rotate them.
I would rather make sure the container doesn't log too much lines (by using a CMD in the dockerfile that either redirect the output of you process to a file in a volume or to /dev/null - with logs enabled with configuration - and I would then logrotate the log files with another container)
Here a simple one-liner to clean container log file:
echo "" > $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)
if you run sudo lsof | grep '(deleted)', you might find that there are a lot of deleted container log files left open by docker
docker 17511 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 2879 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 6191 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 8713 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 8726 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 9907 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 9918 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11093 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11096 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11110 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11885 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11890 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11973 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11976 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 11983 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 13105 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 16737 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17139 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17516 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17517 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17518 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17519 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17520 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17525 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17574 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17584 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17586 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17637 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17643 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17644 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17659 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17666 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17669 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17680 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17884 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 17895 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 21126 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 21474 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 21475 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 21519 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 21721 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 23829 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 23830 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 23833 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 24196 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 24197 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 26821 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 28597 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 28608 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 30305 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 30308 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 30594 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker 17511 30595 root 25r REG 202,1 105 456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
These are logs that belong to a deleted container. One way to have these closed is by restarting the docker daemon itself. This is not always feasible.
Another way to free this space is to find the relevant file descriptor and to truncate it:
ls -lh /proc/17511/fd/ | grep e49d81fc
lr-x------ 1 root root 64 Sep 8 08:06 25 -> /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
Where 17511 is the pid of docker and e49d81fc is part of the offending logfilename / container id. Here, the fd is /proc/17511/fd/25, so we truncate it:
cat /dev/null > /proc/17511/fd/25
I am using Docker version 1.13.0, build 49bf474
If you are using the default logging format json-file, then you can specify the option to rotate your logs.
You can set up the docker daemon to do that or set it up per container.
For instance, log file size of 2 MB and maximum of 5 logfiles:
Docker daemon: Create file /etc/docker/daemon.json:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "2m",
"max-file": "5",
}
}
Docker run:
docker run -it --log-driver json-file --log-opt max-size=2m,max-file=5 alpine ash
For details see logging and dockerd configuration
A late answer, but anyone in the same boat might be better off exploring options for exporting their docker logs to somewhere sensible for managing them. logspout is useful for colelcting logs from all your docker containers and sending them off to somewhere. IF you're only just needing to consider this, then sending them to the host's syslog daemon might be a good approach, and you can use the usual mechanisms for arranging those into files and rotating them from there. For larger multi-host environments, you might look at shipping to an "ELK" stack. i.e. Elasticsearch, Logstash and Kibana.
Saw this script in official docker documentation:
sudo find /var/lib/docker/containers/ -type f -name "*.log" -delete
Works well, but docker doesn't release the memory automatically...The script just deletes the log files.
I find the containers with the largest logs:
root@dockerhost:~# cd /var/lib/docker/containers/
root@dockerhost:/var/lib/docker/containers# find . -size +2G
./62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72/62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72-json.log
./295abd50c1680f517508a1a35fd32a1f525ec13edb4c539cb4abb1e5e3c47da2/295abd50c1680f517508a1a35fd32a1f525ec13edb4c539cb4abb1e5e3c47da2-json.log
root@dockerhost:/var/lib/docker/containers# docker ps -a --no-trunc | grep 62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72 62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72 apacheverdaccio_verdaccio "uid_entrypoint /bin/sh -c '$VERDACCIO_APPDIR/bin/verdaccio --config /verdaccio/conf/config.yaml --listen $VERDACCIO_PROTOCOL://0.0.0.0:$VERDACCIO_PORT'" 6 months ago Up 2 days 0.0.0.0:14873->4873/tcp apacheverdaccio
root@dockerhost:/var/lib/docker/containers#
I then cat /dev/null > <the log file> and docker restart the container
Of course, all this is bad practice, what you should do is spin up a new container and remove the old one.
To find the docker location, you can use:
docker info
Containers: 16
Images: 45
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Dirs: 78
Execution Driver: native-0.2
Kernel Version: 3.16.0-31-generic
Operating System: Ubuntu 14.10
WARNING: No swap limit support
If you want to free space, you can do in this way:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2e2977ed2f4c ekino/puppetlabs:12.04.5 "/bin/bash" 15 hours ago Exited (0) 15 hours ago stoic_jones
ded5903df161 ekino/puppetlabs:12.04.5 "/bin/bash" 12 days ago Exited (0) 11 days ago boring_wilson
72a831b7e4ab a964750d8f91 "/bin/bash" 4 weeks ago Exited (127) 3 weeks ago hungry_meitner
79af8331bad9 a964750d8f91 "/bin/bash" 4 weeks ago Exited (0) 4 weeks ago tender_wilson
d3d3ca34b4b2 1af3728b2912 "/bin/bash" 6 weeks ago Exited (0) 5 weeks ago distracted_shockley
Now you can delete a container with:
docker rm ac055c58ab19