Setting up Docker for Remote Deployment
I built a cluster of servers using Raspberry PIs as way of learning about scaleable web infrastructure. One of the goals is to be able to build working web services and then deploy them on any node in the cluster – remotely. That is, without having to log in on any node and configure the service manually.
I’m using docker to deploy services within containers. I want to be able to build a container locally, test it, push it to a local container registry, and then to deploy it on any node in my cluster.
How docker works
Docker is a daemon that creates Linux containers on the local system. The daemon is controlled via a REST web api. The docker program on any system simply issues commands to a docker daemon.
In a stock configuration docker listens on a local linux web socket file: /var/run/docker.sock
It is possible to configure the docker daemon to listen on a real network port, and then to have the local docker program issue commands to the docker daemons on remote systems.
Since the docker daemon has the ability to run arbitrary software, you have to keep network access to the docker daemons very secure. Anyone will network access to the docker daemon has the equivalent of root access on that machine.
The standard way to do this with docker is to configure the daemon (a.k.a server) and the docker program (a.k.a. client) to communicate with each other using TLS ( e.g. over HTTPS ). In regular HTTPS web browsing, the client validates the server’s certificate to validate the server’s identity and set up an encrypted channel. The normal way this is done with docker is that both the client and the server require certificates and validate each other. It is particularly important that the server only respond to validated clients.
An explanation of the docker security model can be found in the docker documentation.
Setting up the Certificates
You can find basic instructions on how to do this, in the docker documentation.
And there are several people online who have posted scripts to generate a set of compatible certificates. I used scripts from James Kyle’s article. He has a larger script for this on github, but I found the dead simple one in his article works well enough for what I need:
#!/bin/bash
set -ex
mkdir certs && cd certs
echo "Creating server keys..."
echo 01 > ca.srl
openssl genrsa -des3 -out ca-key.pem
openssl req -new -x509 -days 3650 -key ca-key.pem -out ca.pem
openssl genrsa -des3 -out server-key.pem
openssl req -new -key server-key.pem -out server.csr
openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem \
-out server-cert.pem
echo "Creating client keys..."
openssl genrsa -des3 -out client-key.pem
openssl req -new -key client-key.pem -out client.csr
echo extendedKeyUsage = clientAuth > extfile.cnf
openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca-key.pem \
-out client-cert.pem -extfile extfile.cnf
echo "Stripping passwords from keys..."
openssl rsa -in server-key.pem -out server-key.pem
openssl rsa -in client-key.pem -out client-key.pem
When I ran this script it asks for a lot of information. Since these are internal self-signed certificates, it doesn’t matter what information you give it. Just make sure that you set the FQDN to “*” or to the fully qualified domain of the server or client machine that will use this certificate.
In my case, I set them all to “*”, and will use the same certificates for all the servers and clients on my cluster. This does mean I’m making in a sense universal keys to my docker infrastructure, so I have to really make sure I keep them secure.
I put all the generated certificates and keys in a directory on each machine where I want to run either a docker daemon, or a docker client:
rafael@node00: ls /etc/docker/ssl ca.pem client-cert.pem client-key.pem server-cert.pem server-key.pem
The ca.pem file is a certificate authority file, which you will need as the root to validate any of the certificates generated. All docker servers and clients program will need access to this file.
Since I’m keeping things simple by using the same certificates for all clients and servers, I copy the same certificates to all my nodes. All servers will need access to server-cert.pem and server-key.pem. All clients will need access to client-cert.pem and client-key.pem.
Configuring the Docker Daemon for TLS
You just have to start the docker daemon with the following parameters:
--tlsverify --tlscacert=/etc/docker/ssl/ca.pem --tlscert=/etc/docker/ssl/server-cert.pem --tlskey=/etc/docker/ssl/server-key.pem -H 0.0.0.0:2376
The H option causes the daemon to listen on network port 2376, which is the standard port for docker over HTTPS/TLS. tlsverify, causes the server to verify any clients. The other options point to the required certificates.
How you pass these options to daemon depends on how docker daemon is launched on your system. For me, I add these parameters to /etc/default/docker:
# Docker Upstart and SysVinit configuration file # Use DOCKER_OPTS to modify the daemon startup options. DOCKER_OPTS="--storage-driver=overlay -D --tlsverify \ --tlscacert=/etc/docker/ssl/ca.pem \ --tlscert=/etc/docker/ssl/server-cert.pem \ --tlskey=/etc/docker/ssl/server-key.pem -H 0.0.0.0:2376"
Once you change the docker daemon’s startup options, you have to restart the daemon:
sudo service docker stop sudo service docker start
Configuring the Docker Clients for TLS
By default when you run docker, it will communicate with the local docker daemon though a named socket (/var/run/docker.sock). To communicate to a remote daemon, you have to use the -H option:
docker -H tcp://node00:2376 images
In this case node00 is the domain name of one of my docker servers. But this won’t do the certificate validation that the server requires. So you have to add a bunch of parameters like the ones used to launch the server:
docker --tlsverify --tlscacert=/etc/docker/ssl/ca.pem \ --tlscert=/etc/docker/ssl/client-cert.pem \ --tlskey=/etc/docker/ssl/client-key.pem -H tcp://node00:2376 images
Since this long command line is not practical, I created a short script to simplify this, /usr/bin/dockr:
#!/bin/bash
docker --tlsverify --tlscacert=/etc/docker/ssl/ca.pem \
--tlscert=/etc/docker/ssl/client-cert.pem \
--tlskey=/etc/docker/ssl/client-key.pem -H tcp://$1:2376 ${@:2}
Which I can use like this:
dockr node00 images
Its just like the docker command, but I can prefix any command by the server I want to talk to.
