What is a Kubernetes distribution?

For those unfamiliar with the term, a distribution is to Kubernetes just like a distribution is to Linux. As the saying goes, Kubernetes is a powerful engine, but you need to build a car around it. In other words, a Kubernetes distribution is Kubernetes with a chosen configuration and a chosen set of addons. For example, Kubernetes is not secure by default, nor by itself. Hence, a security-focused Kubernetes distribution will configure Kubernetes with restrictive PodSecurityPolicies and add intrusion detection addons.

Specifically, a Kubernetes distribution consists of three parts:

1. A procedure — either code or documentation — to create infrastructure necessary for a Kubernetes cluster, such as virtual machines, load balancers, network security policies, DNS records, etc. The distribution may impose a certain base operating system (OS) image or may cater to a large variety of base images.
2. A procedure to bootstrap Kubernetes control plane components onto the control plane and worker nodes, in particular the apiserver, scheduler, kubelets and controller manager. Most distributions come with a cloud integration. This means that Kubernetes Services of type LoadBalancer and PersistentVolumeClaims automatically work.
3. A procedure to set up addons, such as Ingress controllers, certificate managers, intrusion detection, custom authentication, to name a few.

First Impressions of EKS Distro

EKS Distro launched with a healthy documentation and code. You have several ways of using it:

• Build it from scratch: The documentation and code are sufficient to build EKS Distro from scratch, specifically, the container images that make up the Kubernetes control plane components.
• Start EKS-D: If you don’t need to customize control plane images, then you can use the pre-build images by AWS from their public container registry.

Our next observation is related to the level of automation of EKS Distro. Let’s take an automation scale with a few levels:

1. Prose-only documentation.
2. Documentation with copy-paste-able code snippets.
3. Command that does everything.
4. API that does everything.

EKS Distro is level 2: It focuses on providing a lot of documentation and copy-paste-able. code snippets. However, if you are the “TL;DR” type, EKS Distro will disappoint you. You need to carefully read the prose to understand how to use the next code snippet. 

While this could be seen as a weakness, we find it a strength. Instead of pre-baking and imposing many decisions on the user, EKS Distro is rather un-opinionated. It allows the user to control which decisions to take at each step, while at the same time offering “it just works” defaults.

Next observation: EKS Distro is built on top of kops. Comparing kops and kubespray — the two dominant open source Kubernetes cluster life-cycle managers — deserves its own article. For now, suffice to say that, correctly setting up EKS Distro requires a fair knowledge of kops for things like setting up AWS IAM roles and the DNS stack accordingly. Even small mistakes with kops — e.g., not setting up the DNS stack correctly — will make you unsuccessful with deploying EKS Distro. On the upside, if you are already familiar with kops, EKS Distro will feel like a fairly thin and unobtrusive layer on top.

What does EKS Distro add on top of Kubernetes?

As said before, EKS Distro is a fairly thin layer on top of kops. It basically adds three things:

1. A bit of automation on top of kops.
2. A few fixes for Kubernetes.
3. The AWS IAM authenticator.

Let us focus on the last two.

The control plane images backport a few Kubernetes fixes that are critical for EKS Distro. These fixes are available in the main branch of the Kubernetes project, however, backporting them allows AWS to provide these without waiting for a new Kubernetes release.

The AWS IAM authentication enables authentication with the Kubernetes cluster using AWS credentials. This does make sense for users who want to leverate AWS IAM roles for everything, including on-prem Kubernetes clusters. However, for users who aim for a cloud-agnostic strategy, OpenID is a more portable Kubernetes authentication solution.

Have I already mentioned how thin EKS Distro is? We were surprised to discover that it does not include an Ingress controller, a certificate manager or a log forwarded. Truly un-opinionated!

Gotchas

While the ride was pretty smooth, we did bump into a few minor issues:

1. EKS Distro assumes the us-west-2 region in several places. Fortunately, due to its light packaging, fixing the scripts to deploy in my favorite region, eu-north-1, was a no-brainer.
2. EKS Distro blocked the first time I tried it out, due to my laptop being entropy starved and /dev/random being stuck. A simple change, to take randomness from /dev/urandom fixed the problem. In fact, our Pull Request (PR) was merged within hours. So another point goes to EKS Distro for being very welcoming.