The problem is that restart: always policy does not seem to work when I kill the container (simulating app crash using docker kill) and docker-compose does not restart my container, even though the Exit Code is 137. I observe the same behaviour when I use restart: on-failure policy. Versions 2 and 3 of docker-compose behave the same. My system is Ubuntu Server 16.04 x64.
My questions are:
Why docker-compose does not restart crashed (killed) container?
When you use docker kill, this is the expected behavior as Docker does not restart the container: "If you manually stop a container, its restart policy is ignored until the Docker daemon restarts or the container is manually restarted. This is another attempt to prevent a restart loop" (reference)
If you use docker stop or docker kill, you're manually stopping the container. You can do some tests about restart policies: restarting the docker daemon, rebooting your server, using a CMD inside a container and running an exit...
For example if I kill my container deployed with a restart policy, I see that it exited with code 137 but it is not restarted according to docker ps -a, it remains exited:
[root@andromeda ~]# docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
819d1264c30a redis:alpine "docker-entrypoint..." 3 minutes ago Exited (137) 34 seconds ago keepalive_redis_1
But if I restart the daemon...
[root@andromeda ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
819d1264c30a redis:alpine "docker-entrypoint..." 30 minutes ago Up 2 seconds 6379/tcp keepalive_redis_1
The container that was set with restart policy, starts again which is what documentation say, so docker kill is not the way you should test the restart policy as it's assumed that you have deliberately stopped the container and Docker wants to have a way to prevent restarting loops, if you kill it, you really want to kill it.
I found the following links valuable that show the same behavior in different versions (so it's not a bug but the expected behavior):
I was there many times, but as you can see, the documentation is not robust and there is no explanation how this feature works, that’s why I asked question - I would like to see answer from someone with hands-on experience in this field.