One of the first security features you should add on top of your Kubernetes cluster is Role Based Access Control (RBAC). It empowers you to control *who* gets to do *what* inside your clusters – and your logs are no longer a list of anonymous calls performed by “admin”.
Now you’re about to integrate and configure your CI/CD pipeline with your Kubernetes cluster. While some CI/CD solutions require godly cluster-wide permissions, others have their own notion of access control. Which ones work with your existing RBAC solution and what do you need to do to configure your CI/CD?
In this article, I’ll walk you through different kinds of CI/CDs, and what is required to make them play nice with your RBAC solution.
First, what CI/CD pipelines are out there?
One can distinguish between two “styles” of CI/CD: pull-style and push-style. Pull-styles CI/CD, like ArgoCD or Flux, means that a special controller is installed inside the cluster, which monitors a Git repository. When a change is detected the controller “pulls” changes into the cluster from the Git repository. Push-style CI/CD, like GitLab CI or GitHub Actions, means that a commit will trigger some commands on a CI/CD worker, which will push changes into the your Kubernetes cluster. The CI/CD worker generally runs outside the Kubernetes cluster.
How do I integrate a pull-style CI/CD with Kubernetes?
The special controller in a Pull-style CI/CD often requires considerable permissions and introduces a new notion of access control. Some pull-style CI/CD solutions can be used with your existing RBAC solution, others not. By default, ArgoCD installs a ClusterRole with wide permissions, which can be used to bypass your existing access control. Using it as-is might be non-compliant with various regulations. Instead, edit the default ArgoCD manifest to create a very restricted Role that only operates in the target namespace.
Flux v1 is in maintenance mode and might become obsolete soon. Flux v2 on the other hand, brings its own notion of access control and requires special considerations to ensure it obeys your existing access control. Installing it can only be done after having made a thorough risk-reward analysis. At the time of this writing, due to these special considerations, we discourage the use of Flux v2.
Download the open source Kubernetes distribution.
All security features pre-configured from day one with Elastisys Welkin. Including RBAC, monitoring, logging, intrusion detection and more.
How do I integrate a push-style CI/CD with Kubernetes?
Push-style CI/CD works pretty much as if you would access Welkin from your laptop, running kubectl or helm against the cluster, as required to deploy your application. However, for improved access control, the KUBECONFIG provided to your CI/CD pipeline should employ a ServiceAccount which is used only by your CI/CD pipeline. This ServiceAccount should be bound to a Role which gets the least permissions possible. For example, if your application only consists of a Deployment, Service and Ingress, those should be the only resources available to the Role.
To create a KUBECONFIG for your CI/CD pipeline, proceed as shown below.
Pre-verification
First, make sure you are in the right namespace on the right cluster:
kubectl get nodes
kubectl config view --minify --output 'jsonpath={..namespace}'; echoYou can only create a Role which is as powerful as you (see Privilege escalation prevention). Therefore, check what permissions you have and ensure they are sufficient for your CI/CD:
kubectl auth can-i --list!!!note What permissions you need depends on your application. For example, the user demo creates Deployments, HorizontalPodAutoscalers, Ingresses, PrometheusRules, Services and ServiceMonitors. If unsure, simply continue. RBAC permissions errors are fairly actionable.
Create a Role
Next, create a Role for you CI/CD pipeline. If unsure, start from the example Role that the user demo’s CI/CD pipeline needs.
kubectl apply -f ci-cd-role.yaml!!!important “Dealing with Forbidden or RBAC permissions errors” > Error from server (Forbidden): error when creating “STDIN”: roles.rbac.authorization.k8s.io “ci-cd” is forbidden: user “demo@example.com” (groups=[“system:authenticated”]) is attempting to grant RBAC permissions not currently held:
If you get an error like the one above, then it means you have insufficient permissions on the Welkin cluster. Contact your administrator.Create a ServiceAccount
User accounts are for humans, service accounts for robots. See User accounts versus service accounts. Hence, you should employ a ServiceAccount for your CI/CD pipeline.
The following command creates a ServiceAccount for your CI/CD pipeline:
kubectl create serviceaccount ci-cdCreate a RoleBinding
Now create a RoleBinding to bind the CI/CD ServiceAccount to the Role, so as to grant it associated permissions:
NAMESPACE=$(kubectl config view --minify --output 'jsonpath={..namespace}')
kubectl create rolebinding ci-cd --role ci-cd --serviceaccount=$NAMESPACE:ci-cdExtract the KUBECONFIG
You can now extract the KUBECONFIG of the ServiceAccount:
SECRET_NAME=$(kubectl get sa ci-cd -o json | jq -r .secrets[].name)
server=$(kubectl config view --minify --output 'jsonpath={..cluster.server}')
cluster=$(kubectl config view --minify --output 'jsonpath={..context.cluster}')
ca=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.ca\.crt}')
token=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.token}' | base64 --decode)
namespace=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.namespace}' | base64 --decode)
echo "\
apiVersion: v1
kind: Config
clusters:
- name: ${cluster}
cluster:
certificate-authority-data: ${ca}
server: ${server}
contexts:
- name: default-context
context:
cluster: ${cluster}
namespace: ${namespace}
user: default-user
current-context: default-context
users:
- name: default-user
user:
token: ${token}
" > kubeconfig_ci_cd.yamlThe generated kubeconfig_ci_cd.yaml can then be used in your CI/CD pipeline. Note that, KUBECONFIGs – especially the token – must be treated as a secret and injected into the CI/CD pipeline via a proper secrets handing feature, such as GitLab CI’s protected variable and GitHub Action’s secrets.
