How can I automatically restart my Docker containers with container auto-delete?

english
#1

I have a general purpose VPS, and am Dockerising the apps on it. There will be around 5-6 containers on it, and very little else, so the box can be trivially rebuilt as required.

For each app I have a start script. A WordPress container looks like this:

#!/bin/bash

# Get the host IP address
export DOCKER_HOSTIP=`ifconfig docker0 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1`
echo "Connecting to database on Docker host ${DOCKER_HOSTIP}"

docker run \
    --add-host=docker:${DOCKER_HOSTIP} \
    --network dockernet \
    --network-alias jonblog \
    --detach \
    --restart always \
    --rm \
    jonblog

However, that returns an error:

Conflicting options: --restart and --rm

There are several Docker tickets that say this is sensible, but I don't understand it. I think my meaning is clear: if a container is not running (e.g. on start-up) then I want to start it. If it dies then I expect the container to be removed, and a new fresh one created from the base image. Containers should be immutable anyway - any state I wish to preserve, like media files and logs, will be written to volumes.

So, I thought I should drop the --restart flag, and then use a process manager to stop and start Docker containers. Could I use Monit here? I was hoping to be able to do something like:

CHECK PROCESS jonblog MATCHING jonblog
  START PROGRAM = "/root/docker/jonblog/host-start.sh"
  STOP PROGRAM = "docker stop jon-blog"

However, that checks the system process table, and not docker ps, and so it won't find something matching the specified string. Can I get it to exec docker ps periodically, and match lines in the output?

I'd be happy to use another tool if it proves robust. For example, I find Supervisor a bit heavyweight, but if that is more able to work with Docker, I am willing to use it.

Clarification on --rm

The reason why I want --rm is that during the Dockerisation process, I stop the currently running container, load a new version of the image, and re-run the above script. This means that Docker is being notified of a restart policy for each container. I found that after rebooting the box, I would have 15 or so slightly-differing versions of the app running simultaneously, which is not the intention.

I suppose I could use docker update --restart never on old containers to prevent this happening, but then when my container stops, I am left with it lying around, and I'd just as soon have it auto-delete. I could periodically clean old ones up using some sort of cron job, but that feels a bit hacky given that Docker can do it for me.

Seeking a range of answers

A very helpful comment has suggested I should look into Minikube, which apparently simplifies the setup of Kubernetes, even to the degree a suitably skilled individual can be up and running in five minutes.

I would still quite like to see more lightweight solutions proposed, so I have a range of answers to choose from. As indicated, I would like to find out the answer to whether a process supervisor like Monit would work.

Off the top off my head, I could write a shell loop to write docker ps to a file every five seconds for a minute, and then run that on a Cron every minute. I could then scan that file using grep and the Monit CHECK PROGRAM system check. That's a bit hacky, but is something that I can understand easily if there is a problem with it. Any advances on that suggestion?

#2

I have an answer that is suitable for my current understanding of Docker. I was advised in the comments to try Minikube, and although undoubtedly this can be spun up quickly, I feared that this would be a rabbit-hole of learning that would get me stuck in tar for weeks. One of my engineering principles is to know when one has reached a cognitive limit for stuffing in new information!

Thus, I set out to resolve this problem in a simple fashion. I had two choices:

  1. Use the container auto-delete feature in Docker, and set up my own restart system
  2. Use the Docker restart policy, and set up my own container deletion system

I started on the first of these, with the idea that the process supervisor Monit would be nice to use, partly because it is lightweight, and partly because I am familiar with it. However, it started to feel like the wrong solution, since I'd be working around the core problem that it cannot cleanly get a Docker container process list.

In fact, the second option was much cleaner, and this was amplified by the fact that stopped container clean-up is not actually a priority - it is just to keep things tidy. Of course, I used Docker for this; here's the Dockerfile:

# Docker build script for Docker Tidy

FROM alpine:3.6

RUN apk update
RUN apk add docker

# See this for BusyBox cron schedules
# https://gist.github.com/andyshinn/3ae01fa13cb64c9d36e7
COPY bin/docker-tidy.sh /etc/periodic/daily/
RUN chmod +x /etc/periodic/daily/docker-tidy.sh

# Start Cron in the foreground
ENTRYPOINT ["crond", "-l", "2", "-f"]

And here's bin/docker-tidy.sh:

#!/bin/sh
#
# With thanks to:
# http://www.doublecloud.org/2015/05/simple-script-to-list-and-remove-all-stopped-docker-containers/

docker rm -v $(docker ps -a -q -f status=exited)

Finally, one drawback with my solution is that if the host is rebooted prior to a stopped container cleanup, those containers seem to restart as well. I therefore reset the restart policy on those containers prior to starting new ones.

For example, here is how I start the Docker Tidy container itself, on the host. In practice I've tidied up the policy change code into its own script, but this will give the general idea:

#!/bin/bash

# Removes the restart policy from previous containers
CONTAINER_LABEL=docker-tidy-instance
docker ps --all --filter label=$CONTAINER_LABEL --quiet | xargs --no-run-if-empty docker update --restart no

docker run \
    --label $CONTAINER_LABEL \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    --detach \
    --restart always \
    docker-tidy
#3

Go play with Minikube you can have that up in about five minutes.

#4

Fair enough, @Michael. Can I do that with Monit (on the basis I am somewhat familiar with it)? I looked at the Man page, but that is a monster of a read, and I lost the will to live when wading through that. I wonder though, from the docs, whether the program check would work - if I can sneak a parameter into the executable path, it can return success/fail depending on whether the named container is running.

#5

There is probably a more Docker-oriented way to do this, but my search-fu is not turning up anything particularly interesting.

#6

Honestly if you’re rebuilding containers that frequently, you ought to be using something like Kubernetes to manage things.

#7

You’re right @Michael, and I’ll delve into that when I have time. I am informed by the Docker sub on Reddit that Kube is quite an undertaking (though a valuable one). I will seek out something easier for now, which I have a chance of grokking.

#8

(A comment here five minutes ago, now deleted, said that I just need restart. Yes, this would work, but I would then have to forego --rm - I can have one or the other, but not both, given the error I showed in the question. See the clarification edit on why I think --rm is required).

#9

There is no good way to differentiate between the circumstances when you want the container restarted and the circumstances when you want the container deleted. This is something you need to handle yourself outside of Docker.