So you recently decided to increase development speed by adopting Kubernetes. Suddenly you face a lot of questions: Is Kubernetes sufficient by itself? What are all these projects around Kubernetes? What is vanilla Kubernetes? Do I need a Kubernetes distribution? Which one?
This post will highlight all the features that a DevOps team typically needs to add on top of a vanilla Kubernetes cluster to obtain a secure, production-ready Kubernetes cluster. Adding these features takes time and effort, hence, starting from a production-ready, battle-tested Kubernetes distribution can accelerate an organization’s adoption of Kubernetes.
Edited September 27, 2021: this article has been updated, and a section on automated security scanning tools have been added.
What is vanilla Kubernetes?
“Vanilla Kubernetes” is a term used to describe a Kubernetes setup which is as small as it can get. This is the kind of Kubernetes cluster one gets from managed Kubernetes offers, such as AWS EKS, Azure AKS or Google GKE, and Kubernetes installation tools, such as kops or kubespray.
Vanilla is as plain as it gets, and comes with no bells and whistles included. So it’s like buying the most basic car in the lot: It has steering wheel and seat belts, but no additional safety features. So it can be driven immediately, but it won’t help keep you safe with lane departure warnings or auto-breaking.
How minimal vanilla Kubernetes is, is subject to some debate. At the very least, Kubernetes needs the following components:
- On the control-plane nodes: etcd, apiserver, scheduler and controller-manager.
- On the worker nodes: some container runtime and kubelet.
(Without these, can anyone really call the system “a Kubernetes cluster”?)
How vanilla is a vanilla Kubernetes really?
So far, you have a car without fuel. The above Kubernetes setup cannot really run anything yet: Kubernetes does not know how to make Pods, i.e., a collection of one-or-more containers, talk to each other. To this end, even the “most vanilla” Kubernetes comes with a Container Network Interface (CNI) addon, such as Calico. Now Kubernetes needs to know how to bring traffic to your Pods and how to store their data, a function fulfilled by the so-called cloud controller.
Knowing what is happening inside your Kubernetes cluster — also called “observability” — is essential to avoid unplanned downtime and customer disappointment. Therefore, most managed Kubernetes offers also include a few more integrations with the underlying cloud provider. For example, Google GKE sends logs, metrics and traces to Google StackDriver.
Great! Now you can run your application, you can direct traffic within the cluster and you can get traffic from end users into the cluster and to your application. Furthermore, logs and metrics are collected. What would I need more than vanilla Kubernetes?
Add Secure Ingress Controller
Data privacy regulations — such as HIPAA and GDPR — require you to encrypt public Internet traffic. Even without regulatory pressure, it is a good idea to encrypt public Internet traffic, unless you are a fan of bad PR. TLS termination — i.e., the “wall” that separates “encrypted traffic outside” from “unencrypted traffic inside” — could be performed by your cloud provider or your application. However, if you want to stay cloud agnostic and keep your application code focused on business logic, you can do TLS termination in the Kubernetes cluster. Projects, such as Ingress controller and cert-manager, will not only provision TLS certificates for you, but will also rotate them before expiry. Rotating certificates is a best practice, but also a chore that sometimes gets forgotten if it is a manual process. Users would get a scary “Warning: Potential Security Risk Ahead” message and flock to your competitors.
Add Security Hardening
Kubernetes is not secure by default, nor by itself. You should further improve your security posture, to minimize the risk of breaches. Why? If you handle personal data, GDPR requires you to notify the data protection authority and/or your users of any breach. Definitely not what your board wants to read in the newspaper.
Therefore, you should regularly scan for vulnerable container images with projects such as Trivy. You should also consider adding intrusion detection via Falco. And don’t forget about disaster recovery! Projects, such as Velero, can backup your Kubernetes cluster and the application data hosted within. As they say: “Hope for the best, prepare for the worst.”
Scan for insecure default settings
Because Kubernetes is not secure by default, there are a number of key configuration changes you ought to make as soon as possible. Enable authentication, role-based access control, limit permissions given to Pods, and so forth. In August of 2021, NSA and CISA published a tech report on this topic, called “Kubernetes Hardening Guidance“. We have summarized, explained, and contextualized key takeaway messages in our blog post. Using tools such as kube-bench by Aqua Seucurity and kubescape by ARMO security helps identify the most grave misconfigurations. The use of these tools is recommended, simply because it is obvious that any attacker would also scan for such misconfigurations. And you’d rather be the one that finds them first!
Want to keep up with the latest in cloud and Kubernetes?
Let us deliver it straight to your inbox!
Add Multi-Cloud Observability and Authentication
Observability is an essential feature of a platform, to ensure that “2am events” are converted into daily checks. Many cloud providers come with great observability stacks included, such as AWS’s CloudWatch, GCP’s StackDriver, and Azure’s Monitor.
However, a multi-cloud strategy is seen as essential to cope with various risks associated with data privacy, but also with contractual requirements, such as “data should stay within my jurisdiction”. Needless to say, integration with the underlying cloud provider provides a great productivity boost. However, it does not cater well to a multi-cloud strategy.
Fortunately, there are cloud agnostic alternatives:
- Dex, for cloud-agnostic authentication against the Kubernetes cluster (and yes, it can work with your current identity provider, such as Active Directory, LDAP, or Google accounts);
- Prometheus and Grafana, for metrics collection, analysis, visualization and alerting;
- Elasticsearch and Kibana, for log collection, analysis, visualization and alerting.
With all these addons, your Kubernetes cluster should be ready to host your application, in a manner that is secure and observable.
Kubernetes Distributions: a One-Stop Shop
While adding all of the above components on top of a vanilla Kubernetes cluster is not rocket science, it does require careful configuration to make sure they work as a whole. Coming back to the car analogy, you can certainly buy an aftermarket lane departure warning system, but finding the right one and fitting it reliably still takes time. Wouldn’t it be great if it came pre-installed? And if you ever need support, wouldn’t you like it all to come from a single vendor that takes full responsibility for making it all work correctly?
Kubernetes distributions fill this gap: They enhance a vanilla Kubernetes cluster with carefully chosen and configured components to allow you and your team to focus on what is most important to your customers: Rapidly adding features to your application, without worrying about the hosting platform.
