How do I expose the docker API over TCP?

english
#1

I am using portainer and am unable to manage remote endpoints. I tried using the command line to connect to remote docker nodes, but got a message Cannot connect to the Docker daemon at tcp://<remote_ip>:<port>. Is the docker daemon running?.

Yes, they are running. I have added myself to the docker group and can access docker by SSHing into the nodes. However I cannot access any docker nodes remotely.

I modified /etc/default to add / uncomment DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock"

I also modified /etc/init.d/docker and /etc/init/docker.conf to include DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock".

I restarted the docker service, logged out and logged in multiple times in the process, but still cannot connect to the remote node. I cannot even connect to the local node by passing the IP.

What did I miss out? What configuration in what file exposes the API over TCP?

user@hostname:~$ docker -H tcp://<REMOTE_IP>:2375 info
Cannot connect to the Docker daemon at tcp://<REMOTE_IP>:2375. Is the docker daemon running?
user@hostname:~$ docker -H tcp://127.0.0.1:2375 info
Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?
user@hostname:~$ docker -H tcp://<LOCAL_IP>:2375 info
Cannot connect to the Docker daemon at tcp://<LOCAL_IP>:2375. Is the docker daemon running?
user@hostname:~$

Edit: Running ps aux | grep -i docker returns this -

root      3581  0.1  0.2 596800 41540 ?        Ssl  04:17   0:35 /usr/bin/dockerd -H fd://
root      3588  0.0  0.0 653576 14492 ?        Ssl  04:17   0:18 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
#2

I found a solution thanks to Ivan Krizsan's post.

I had to edit /lib/systemd/system/docker.service on my Ubuntu 16.04.2 LTS system to modify the line 

ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:

then

sudo systemctl daemon-reload
sudo systemctl restart docker.service

and everything worked :-). The next step is to figure out how to protect the docker daemon form being hijacked.

#3

The /etc/default directory is where distribution maintainers put their configuration files. If you install docker directly from Docker's repositories, this directory will not be used.

The /lib/systemd directory is where packages will install their systemd files, and they will overwrite any changes there on upgrade. If you use this, your changes will be lost.

To make your own changes to a systemd unit file that persist, you can create a unit file in /etc/systemd/system/docker.service.d/, e.g. here is my standard /etc/systemd/system/docker.service.d/override.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

That override simply unsets all of the command line flags to the dockerd daemon from systemd. Once done, you can override every setting from /etc/docker/daemon.json which is used by docker, and depending on the setting, can be reloaded without restarting the daemon. E.g. here's an example /etc/docker/daemon.json:

{
"debug": false,
"experimental": true,
"hosts": ["fd://", "tcp://0.0.0.0:2376"],
"labels": ["foo=bar", "fez=baz"],
"log-driver": "json-file",
"log-opts": {"max-size": "10m", "max-file": "3"},
"storage-driver": "overlay2",
"tlscacert": "/etc/docker/certs/ca.pem",
"tlscert": "/etc/docker/certs/host-cert.pem",
"tlskey": "/etc/docker/certs/host-key.pem",
"tlsverify": true
}

For your purposes, you only need the line in there to set the hosts.

One extremely important part of the above config file are the TLS settings. If you do not configure mutual TLS between client and server, and you open docker to listen on the network, you are running the equivalent of an open telnet server with root logins allowed without a password. If you prefer ssh over telnet, or if you prefer having a password for your root account, then you must configure TLS. The docker API ports are frequently scanned on the internet, and you will find malware installed on your host in a short while if you ever skip this configuration step.

Full details on how to configure the TLS keys for client and server can be found at: https://docs.docker.com/engine/security/https/

Note that with docker versions 18.09 and above on the client (both where you run your command and the remote node), you can use ssh instead of configuring TLS. This involves using a DOCKER_HOST value of ssh://user@host. E.g.

docker -H ssh://user@host container ls
#4

There is an official documentation describes how to Configure where the Docker daemon listens for connections.

systemd vs daemon.json

Configuring Docker to listen for connections using both the systemd unit file and the daemon.json file causes a conflict that prevents Docker from starting.

Configuring remote access with systemd unit file

  1. Use the command sudo systemctl edit docker.service to open an override file for docker.service in a text editor.

  2. Add or modify the following lines, substituting your own values.

    [Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375

  3. Save the file.

  4. Reload the systemctl configuration.

    $ sudo systemctl daemon-reload

  5. Restart Docker.

    $ sudo systemctl restart docker.service

  6. Check to see whether the change was honored by reviewing the output of netstat to confirm dockerd is listening on the configured port.

    $ sudo netstat -lntp | grep dockerd
tcp        0      0 127.0.0.1:2375          0.0.0.0:*               LISTEN      3758/dockerd

Configuring remote access with daemon.json

  1. Set the hosts array in the /etc/docker/daemon.json to connect to the UNIX socket and an IP address, as follows:

    {
"hosts": ["unix:///var/run/docker.sock", "tcp://127.0.0.1:2375"]
}

    Configuring Docker to listen for connections using both the systemd unit file and the daemon.json file causes a conflict that prevents Docker from starting.

    1. Add or modify the following lines, substituting your own values.

      [Service]
ExecStart=
ExecStart=/usr/bin/dockerd

    2. Save the file.

    3. Reload the systemctl configuration.

      $ sudo systemctl daemon-reload

  2. Restart Docker.

  3. Check to see whether the change was honored by reviewing the output of netstat to confirm dockerd is listening on the configured port.

    $ sudo netstat -lntp | grep dockerd
tcp        0      0 127.0.0.1:2375          0.0.0.0:*               LISTEN      3758/dockerd

The Docker client will honor the DOCKER_HOST environment variable to set the -H flag for the client. Use one of the following commands:

$ docker -H tcp://127.0.0.1:2375 ps

or 

$ export DOCKER_HOST="tcp://127.0.0.1:2375"
$ docker ps
#5

If you don't want to reconfigure and restart your docker daemon you can simply bridge the unix socket to a TCP socket using ncat (from the nmap package):

ncat -lknvp 2375 -c "ncat -U /var/run/docker.sock"

As alternative, you can use socat or other tools.

#6

For those looking for this answer in the context of Ubuntu server 20.04 that uses SNAP:

This comment github issue should give you the context you need. In my case I didn't find the $SNAP_DATA environment variable set, so I had to look for all the daemon.json files on the system and used the one with the /var prefix

$ sudo find / -name daemon.json

In my case it had two unrelated entries so I just added mine:

{
  [.....]
  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2375"]
}

The above IP determines from where it will be accessible, and the port can be whatever port you want to expose. In my particular case, I was not able to make it work with a specific IP, instead I had to use 0.0.0.0, otherwise it would fail with the following error when restarting:

aufs aufs_fill_super:918:mount[3724]: no arg
overlayfs: missing 'lowerdir'
aufs aufs_fill_super:918:mount[3772]: no arg
overlayfs: missing 'lowerdir'
aufs aufs_fill_super:918:mount[3820]: no arg
overlayfs: missing 'lowerdir'

Weird enough, when using 0.0.0.0 it actually spits one pair of the above error lines but it does work afterwards. In my case as this is a VM this is acceptable for me.

#7

I tried similar things and suspect that the files /etc/default/docker, /etc/init/docker.conf and /etc/init.d/docker are simply ignored on Ubuntu 16.04 with an installation of docker-ce, can anyone confirm? I think when I run “service docker status” what really happens is “systemctl status docker”, a whole other management system.

#8

Is 2375 listening? ss -ntl

#9

No. There is nothing listening on 2375. And I cannot figure out what configuration in what file affects this. I have included the output of ps aux in my answer if that helps.