Clearing/Deleting Docker Logs

On our docker implementation on GCE, we are running out of space on the root file system.

Since images themselves are stored on a separate 1TB volume, the images themselves shouldn't be the problem.

One candidate are the centralized logfiles that Docker itself stores (a json file somewhere?), does anyone know where those files/file are/is located, and how we can logrotate/truncate them?

First, I'm using docker 1.1.2 for both client and server, this answer may be obsolete for newer versions of docker as docker evolve quickly.

Location of the file

Find your docker directory. On systems that use apt/debian style system, the package installed by the docker repository https://get.docker.com/ubuntu use /var/lib/docker. Chances are that directory is in the same place on other systems (can't confirm).

under containers/**CONTAINER_ID** you'll find infos about the container. In the file **CONTAINER_ID**-json.log in that folder, you'll find a file with all the logs for that container. It may look like a json file, it's not. It's a flow a json structures, one per line, each containing one log line (each line ends by a } and the next one starts with a {, thus it's not a valid json as a whole).

Example location: - /var/lib/docker/containers/05b6053c41a2130afd6fc3b158bda4e605b6053c41a2130afd6fc3b158bda4e6/05b6053c41a2130afd6fc3b158bda4e605b6053c41a2130afd6fc3b158bda4e6-json.log

Editing/Altering that file

I suggest you to use that path to see wether or not it's the reason why you're running out of space, but not to log rotate them.

I would rather make sure the container doesn't log too much lines (by using a CMD in the dockerfile that either redirect the output of you process to a file in a volume or to /dev/null - with logs enabled with configuration - and I would then logrotate the log files with another container)

Here a simple one-liner to clean container log file:

echo "" > $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)

if you run sudo lsof | grep '(deleted)', you might find that there are a lot of deleted container log files left open by docker

docker    17511             root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511  2879       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511  6191       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511  8713       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511  8726       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511  9907       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511  9918       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11093       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11096       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11110       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11885       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11890       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11973       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11976       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 11983       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 13105       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 16737       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17139       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17516       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17517       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17518       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17519       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17520       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17525       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17574       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17584       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17586       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17637       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17643       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17644       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17659       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17666       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17669       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17680       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17884       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 17895       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 21126       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 21474       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 21475       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 21519       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 21721       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 23829       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 23830       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 23833       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 24196       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 24197       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 26821       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 28597       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 28608       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 30305       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 30308       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 30594       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)
docker    17511 30595       root   25r      REG              202,1      105     456624 /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)

These are logs that belong to a deleted container. One way to have these closed is by restarting the docker daemon itself. This is not always feasible.

Another way to free this space is to find the relevant file descriptor and to truncate it:

ls -lh /proc/17511/fd/ | grep e49d81fc
lr-x------ 1 root root 64 Sep  8 08:06 25 -> /var/lib/docker/containers/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc/39ba30ffc0e218ee830576570222284a93ebb458eadf46933d2dd7afe49d81fc-json.log (deleted)

Where 17511 is the pid of docker and e49d81fc is part of the offending logfilename / container id. Here, the fd is /proc/17511/fd/25, so we truncate it:

cat /dev/null > /proc/17511/fd/25

I am using Docker version 1.13.0, build 49bf474

If you are using the default logging format json-file, then you can specify the option to rotate your logs.

You can set up the docker daemon to do that or set it up per container.

For instance, log file size of 2 MB and maximum of 5 logfiles:

Docker daemon: Create file /etc/docker/daemon.json:

{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "2m",
        "max-file": "5",
    }
}

Docker run:

docker run -it --log-driver json-file --log-opt max-size=2m,max-file=5 alpine ash

For details see logging and dockerd configuration

A late answer, but anyone in the same boat might be better off exploring options for exporting their docker logs to somewhere sensible for managing them. logspout is useful for colelcting logs from all your docker containers and sending them off to somewhere. IF you're only just needing to consider this, then sending them to the host's syslog daemon might be a good approach, and you can use the usual mechanisms for arranging those into files and rotating them from there. For larger multi-host environments, you might look at shipping to an "ELK" stack. i.e. Elasticsearch, Logstash and Kibana.

Saw this script in official docker documentation:

sudo find /var/lib/docker/containers/ -type f -name "*.log" -delete

Works well, but docker doesn't release the memory automatically...The script just deletes the log files.

I find the containers with the largest logs:

root@dockerhost:~# cd /var/lib/docker/containers/
root@dockerhost:/var/lib/docker/containers# find . -size +2G
./62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72/62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72-json.log
./295abd50c1680f517508a1a35fd32a1f525ec13edb4c539cb4abb1e5e3c47da2/295abd50c1680f517508a1a35fd32a1f525ec13edb4c539cb4abb1e5e3c47da2-json.log
root@dockerhost:/var/lib/docker/containers# docker ps -a --no-trunc | grep 62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72                                       62cd9d1459f90b5b2e72d192e7ff65aa3cbbdd0fa877135cccd4c847a0e6df72   apacheverdaccio_verdaccio                                                                                        "uid_entrypoint /bin/sh -c '$VERDACCIO_APPDIR/bin/verdaccio --config /verdaccio/conf/config.yaml --listen $VERDACCIO_PROTOCOL://0.0.0.0:$VERDACCIO_PORT'"   6 months ago        Up 2 days                  0.0.0.0:14873->4873/tcp         apacheverdaccio
root@dockerhost:/var/lib/docker/containers#

I then cat /dev/null > <the log file> and docker restart the container

Of course, all this is bad practice, what you should do is spin up a new container and remove the old one.

To find the docker location, you can use:

docker info
Containers: 16
Images: 45
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 78
Execution Driver: native-0.2
Kernel Version: 3.16.0-31-generic
Operating System: Ubuntu 14.10
WARNING: No swap limit support

If you want to free space, you can do in this way:

docker ps -a
CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS                     PORTS               NAMES
2e2977ed2f4c        ekino/puppetlabs:12.04.5   "/bin/bash"         15 hours ago        Exited (0) 15 hours ago                        stoic_jones             
ded5903df161        ekino/puppetlabs:12.04.5   "/bin/bash"         12 days ago         Exited (0) 11 days ago                         boring_wilson           
72a831b7e4ab        a964750d8f91               "/bin/bash"         4 weeks ago         Exited (127) 3 weeks ago                       hungry_meitner          
79af8331bad9        a964750d8f91               "/bin/bash"         4 weeks ago         Exited (0) 4 weeks ago                         tender_wilson           
d3d3ca34b4b2        1af3728b2912               "/bin/bash"         6 weeks ago         Exited (0) 5 weeks ago                         distracted_shockley     

Now you can delete a container with:

docker rm ac055c58ab19