Running containers as root is a bad idea for security. This has been shown time and time again. Hackers find new ways of escaping out of the container, and that grants unfettered access to the host or Kubernetes node. The latest such bug is CVE-2022-0492 (Palo Alto Network’s writeup here), but we also had CVE-2022-0185 (Aqua Security’s writeup here) earlier this year.
It’s a security nightmare, and you can read why in the next section.
In the remainder of this article, we show how to stop running containers as root. And what to do if you have specialized software that for some reason still needs some root-like capabilities. Because very little actually needs all of them.
What is “running as root”?
Running a container as root means that the software packaged in a container is set to start as the root, or system administrator, user. This user is special in Linux systems, because it has all permissions needed to administer a system. So the root user can read all files, install new software, open any network connection it wants… you name it.
Software started by a user has the same permissions as the user that started it. So if a normal (“non-privileged”) user starts a piece of software, it will be limited in what it can do. If it tries to read files it does not explicitly have permissions to, it will fail. But if the root user starts the same software, the software has the root user’s super powers.
What is the problem with running containers as root?
Containers are a way to package and run software. A running piece of software is called a process. When one starts a container, the software within is started as a process that is isolated via a Linux feature called cgroups. Containers run on a host, or in Kubernetes words, on a node.
True to their name, containers add some isolation between processes. The idea is that cgroups will isolate containers so well that processes running in containers and cgroups will not be able to interact with each other or other processes on the same host.
The problem is that hackers are actively figuring out ways to break this isolation between containers and the rest of the host or Kubernetes node.
What can happen if you run containers as root on Kubernetes?
The problem is that with unrestricted root access to the host or Kubernetes node, a hacker that breaks out of the container’s isolation can view all kinds of secret information. This includes all information from all other containers running there, and all kinds of files on the harddrive. In many cloud environments, that also means access to cloud credentials.
So a hacker can not just read all kinds of information, like database connection credentials, and steal all that data. They could also perhaps start new servers in your cloud account, racking up huge costs and use it as a platform to launch new attacks against other targets. And you’ll be the owner of those resources that are used for the attack.
HOWTO stop running containers as root
The problem with running as root, starts with how the container is first built.
When software is packaged into a container image, you typically have to install some supporting software first. Installing software is something only the root user can do. So container build systems set you up, by default, with the root user.
Unless you explicitly add a non-privileged user, the containerized software will continue to be set to keep running as root.
Download the open source Kubernetes distribution.
All security features pre-configured from day one with Welkin. Including RBAC, monitoring, logging, intrusion detection and more.
Add a non-privileged user and set it as the process owner
The first mitigation technique is to add a non-privileged user and set it as the process owner. In practical terms, this means adding two lines to your Dockerfile (or Containerfile if you’re not using the Docker toolchain to build container images). Add them after any software installations you need to run as root:
RUN useradd --uid 10000 runner
USER 10000
Thank you to /u/nwmcsween on Reddit for suggesting using a UID >= 10000 to avoid clashing with the usual range for regular users on the host.
The first line adds a user with a specified UID (user ID) set to 10000 and a name called “runner”. It also adds a group with the same GID (group ID) and the same name. The second line sets the Dockerfile to switch to the newly created user. There’s a point in setting it to the UID rather than the username, and we’ll get to it.
With just this change, your container will now run as a non-privileged user!
Explicitly tell Kubernetes to run as your non-privileged user and group
If we don’t explicitly tell Kubernetes to run as our non-privileged user via a SecurityContext, it will run with what the container image says, but the default group ID is going to be set to root’s group ID (0). This is more permissions than your process needs.
Add the following four lines to your Pod’s specification, whether just running as a bare Pod (although you probably never want to do that), or inside a Deployment or StatefulSet:
securityContext:
runAsNonRoot: true
runAsUser: 10000
runAsGroup: 10000
Thank you to /u/ElCucharito on Reddit for suggesting adding runAsNonRoot: true.
With these changes, your software will run as the user and the group we created and made sure are known inside your container.
There’s an additional field we can set in the SecurityContext, called fsgroup. This is an additional and quite optional field if you want to add an additional group as owner of files, typically on Persistent Volumes. You can probably ignore it for now, since the above will work.
Verifying that you’re no longer running containers as root
Deploy your software and use “kubectl exec” to get an interactive shell session in your currently running container (or hit the “play”-like button in Lens). There, type “id” as a command. If it comes back and says that your uid and gid are 1000, you’re done!
“But what if I need to run as root?”
First of all, you might not actually need to! If the reason you “need” to run as root is to open, say, port 80 for a web server, you don’t need to do that on a Kubernetes cluster. Tell the software to open and bind to a non-privileged port (the ones <1024 are “privileged” and only root can open them). You can now start the software with a non-privileged user!
Some software does actually require some of root’s privileges and capabilities. For that, we can run software as non-root, but grant additional privileges via Linux capabilities.
This is quite advanced, and you need to figure out which capabilities are actually needed. Once you have, you can add those particular ones to your container via its SecurityContext. Be aware that this can open you up to container escape bugs like the ones we linked to in the start of this article.
What about software out there that already runs as root?
Some software you pull from the Internet is already configured to run as root. Far too much of it, actually. And you can’t just add a SecurityContext to them, because the user ID has to actually exist in the container. So what do you do?
You can create your own version of the container you’re actually interested in, by using it as your container base image. Then add the non-privileged user account to it like we did above.
This can require some fiddling to get right. Perhaps there are some file system permissions that also need to be updated. But you can do this.
The drawback? Whenever the original software updates, so must your derived image, to pull in those changes. You can automate this, of course, in a CI/CD pipeline. And then push your updated image to your container registry.
Some re-packaged software already exists
Bitnami, among others, are great at packaging and keeping other important software up to date. If you want to run, say, MongoDB, and want to run it as a non-root user, your first internet search should be “bitnami mongodb docker”. You will find that they have a suite of well-maintained container images that all run as non-root users.
Summary
In this article, we have shown what the problem is with running containers as root. That it can provide hackers with full access to the node that hosts the container. We have also shown how to avoid it, by adding a non-privileged user and setting the Kubernetes SecurityContext appropriately. Both of these mitigations strengthen your stronger security posture significantly.
In certain Kubernetes distributions, such as Welkin, you can’t even run containers as root. It’s simply not allowed. In such Kubernetes distributions and platforms, you definitely need to take these measures.
