I was fortunate enough to attend Container Camp 2019 in Sydney in July. While there were a number of interesting presentations, it was the last couple sessions on security that really caught my attention. The first one from Aleksa Sarai @lordcyphar on securing container run time was quite informative.. and yet somewhat worrying! I promise Aleksa, I will always use user namespaces from now on!
So what exactly was Aleksa talking about?
User namespaces.
Now, I’ve been using Docker for a while and had never heard of this, so I wouldn’t be surprised if you haven’t either. So what exactly is it? According to the Docker docs:
The best way to prevent privilege-escalation attacks from within a container is to configure your container’s applications to run as unprivileged users. For containers whose processes must run as the root user within the container, you can re-map this user to a less-privileged user on the Docker host. The mapped user is assigned a range of UIDs which function within the namespace as normal UIDs from 0 to 65536, but have no privileges on the host machine itself.
Ah ha! Ok so basically when we run a container, by default it will run as root. The root users UID inside the container matches the UID on our host and can lead to container escape vulnerabilities. For example: CVE-2019-5736
So by using user namespaces, this means that even if the application within the container is run as root and uses the UID 0 this no longer correlates to the root user on the host. So now, even if an attacker is able to compromise and escape from your container, they will still need to escalate their privileges in order to take control of other services on your host or take control of the host itself.
So how can we enable this security feature for our Docker hosts?
NOTE: Currently, user name spaces does not appear to be supported with Docker Desktop CE on macOS, so don’t try this at home on your macbook kids. See issue: Daemon fails to start with “userns-remap” enabled. Although if you do find a way to get this working please let me know!
Firstly we need to enable the --userns-remap option for the Docker daemon.
This could be done by simply starting dockerd with the --users-remap flag but lets setup a more permanent solution by configuring the Docker daemon with the daemon.json config file.
Edit /etc/docker/daemon.json and add the userns-remap option as shown below {"userns-remap": "default"}.
Save the file and restart Docker NOTE: By using the value “default”, this will automatically create a dockremap user for you. You could, alternatively, create your own users and groups and IDs that you wish to remap to. This is outside the scope of this post but you can look at the docks here for more information on how to achieve this.
To verify if the docker remap feature is working. on your host check that a dockremap user has been created. $ id dockremap uid=111(dockremap) gid=114(dockremap) groups=114(dockremap)
Now we can also verify that the new dockremap user has been added to /etc/subuid and /etc/subgid $ grep dockremap /etc/subuid dockremap:231072:65536 $ grep dockremap /etc/subgid dockremap:231072:65536 NOTE: If these entries are not present (Perhaps if you are using RHEL or CentOS) you should add these entries to the files. When calculating the UID and GID to use. Pick the highest assigned one plus (check /etc/adduser.conf for the range of UIDs and GIDs that are assigned ) To add the entries its as simple as: echo "dockremap:231072:65536" >> /etc/subuid echo "dockremap:231072:65536" >> /etc/subgid
Verify that Docker is using our new user namespace for our containers a. Run a container docker run hello-world b. Check that there is a namespaced directory in /var/lib/docker named with our dockremap UID and GID $ sudo ls -l /var/lib/docker/ total 52 drwx------ 14 231072 231072 4096 Aug 03 06:18 231072.231072 The directories owned by the remapped user and used instead of the same directories directly beneath /var/lib/docker the unused versions such as /var/lib/docker/tmp can be removed. Docker does not use them while userns-remap is enabled
Final proof! Lets run a container and check what user id the application is running as. $ docker run -it ubuntu bash # ps -eo uid,gid,args | grep bash0 0 bash Here we can see the command bash is running as the root user with UID 0 and if we check from the host $ ps -eo uid,gid,args | grep bash 231072 231072 bash We can see that the bash command being run in the container is actually running as UID 231072 on the host!
GREAT SUCCESS!
Note
One other change that occurs as a result of enabling user namespaces is that you will need to re-pull any images as they will now live in that new namespaced directory mentioned above.
What about Kubernetes
While this all seems to make sense if you are running docker. What if you are using Kubernetes?
Kubernetes has got you covered with Pod Security Policies. Specifically you could create a policy that uses RunAsUser along with AllowPrivilegeEscelation=false
There are many other options as well for securing workloads in Kubernetes but that is outside the scope of this blog post.