As we discussed in a previous post, monitoring your application and its platform is basic hygiene for running a healthy software-as-a-service business. But how exactly do you monitor your application? In this post, we will illustrate how to achieve that using kube-prometheus.
Kube-prometheus collects all the components necessary for monitoring a Kubernetes cluster in one package. At the core, we find the Prometheus operator that makes it easy to deploy and configure, not only Prometheus, but also Alertmanager. Add to this Grafana, some metrics collectors and a default set of dashboards and alerting rules and you have a complete monitoring solution.
There is no one-size-fits-all solution for monitoring your Kubernets cluster however. But fear not, kube-prometheus is based on Jsonnet and made in the form of a library that you build your own custom solution from. Anything and everything can be changed, replaced or added. And you won’t need to wade through Helm templates in order to do it either.
Prerequisites
You will need a Kubernetes cluster running v1.19 or v1.20. See for example compliantkubernetes.io for how to set up a production ready cluster. If you just want something simple to test things out locally, you may use KIND or Minikube. For more details on kube-prometheus prerequisites see the documentation.
In addition to this, you will need to install Jsonnet, Jsonnet-bundler (aka jb or the Jsonnet package manager) and gojsontoyaml. Alternatively, you may use a container with these tools pre-installed, e.g. quay.io/coreos/jsonnet-ci. If you want to use the container, just prepend the following to any jb or jsonnet command (or scripts involving them): docker run --user $(id -u):$(id -g) --rm -v $(pwd):$(pwd) --workdir $(pwd). Here is an example:
# A normal jsonnet command
jsonnet -J vendor example.jsonnet
# Run the same command in a container using docker
docker run --user $(id -u):$(id -g) --rm -v $(pwd):$(pwd) --workdir $(pwd) quay.io/coreos/jsonnet-ci jsonnet -J vendor example.jsonnetThe --user flag is not strictly needed, but it makes sure that any files generated by the command are owned by your normal user account instead of root.
Installing the kube-prometheus library
There has been significant changes in kube-prometheus since the last release, so to keep up with the latest and greates, we will use the very latest code on main.
$ mkdir monitoring-stack; cd monitoring-stack
$ version=main
$ # Initialize jsonnet (creates empty jsonnetfile.json)
$ jb init
$ # Install a specific version of kube-prometheus
$ jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@${version}
$ # Fetch an example and a build script
$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/${version}/example.jsonnet -O example.jsonnet
$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/${version}/build.sh -O build.shAfter this you should have the following: - A folder called monitoring-stack to work in. - A couple of files in the folder: jsonnetfile.json and jsonnetfile.lock.json. These define the jsonnet dependencies, i.e. kube-prometheus. - A vendor folder where kube-prometheus is installed. - An example on how to use kube-prometheus: example.jsonnet. - A simple build script: build.sh.
Compile the manifests and deploy
To make sure that your setup is working, let’s compile the example and see what we get “by default”.
./build.sh example.jsonnetYou should get another folder manifests with a lot of yaml files ready to be applied in your cluster. If you run into problems, you may want to check the kube-prometheus readme for more details.
Deploy everything with two simple commands:
kubectl apply -f manifests/setup
kubectl apply -f manifestsThe first command will create all the CustomResourceDefinitions and deploy the prometheus-operator. Once it is up and running, the second command will add all the dashboards, alerts, prometheus, grafana and alertmanager. It should look something like this when done:
$ kubectl -n monitoring get pods
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 116m
alertmanager-main-1 2/2 Running 0 116m
alertmanager-main-2 2/2 Running 0 116m
grafana-7767599644-gztq5 1/1 Running 0 116m
kube-state-metrics-544cf87cfd-2m8xd 3/3 Running 0 116m
prometheus-adapter-767f58977c-hrgk5 1/1 Running 0 116m
prometheus-k8s-0 2/2 Running 1 116m
prometheus-k8s-1 2/2 Running 1 116m
prometheus-operator-764cb46c94-wcvln 2/2 Running 0 116mIf you have PodSecurityPolicies enforced in your cluster, you may get CreateContainerConfigError for kube-state-metrics. This is due to this issue and can luckily be easily fixed by adding a securityContext to the relevant container.
Fix kube-state-metrics security context
Jsonnet allow us to easily patch anything and everything that we desire. To fix the issue with kube-state-metrics, we will need to patch the Deployment by adding the following securityContext to the kube-state-metrics container:
securityContext:
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532In Jsonnet we can achieve this by adding a local variable that can be merged with the existing container definition. Add the following at the top of example.jsonnet:
local addSecurityContext = {
containers: std.map(
function(container)
if container.name == "kube-state-metrics" then
container + {
"securityContext": {
"runAsGroup": 65532,
"runAsNonRoot": true,
"runAsUser": 65532
}
}
else
container,
super.containers
)
};This variable is not really useful on its own, but when added to another object, it can merge and patch that object by referencing super.containers (i.e. the containers field of the parent object). The map function form the standard library is then applied to this array. An inline function if used to add the security context only to the container with nameequal to kube-state-metrics. All other containers are left untouched.
In order to use the variable, simply add it to the relevant object further down in example.jsonnet:
{
values+:: {
common+: {
namespace: 'monitoring',
},
},
+ kubeStateMetrics+: {
+ deployment+: {
+ spec+: {
+ template+: {
+ spec+: addSecurityContext
+ }
+ }
+ }
+ },
};Accessing dashboards
The easiest and safest way to access Prometheus, Alertmanager and Grafana is to use port-forwarding:
$ # Prometheus can be accessed at port 9090
$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
$ # Grafana can be accessed at port 3000 using admin:admin as for login
$ kubectl --namespace monitoring port-forward svc/grafana 3000
$ # Alertmanager can be accessed at port 9093
$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093Extending the example
Now that we have seen what is included in the example, let’s add something! A simple alert and a dashboard should be enough to see how things can be added.
Adding an alert
Create the file my-alert.jsonnet with the following content:
{
prometheusRule: {
apiVersion: 'monitoring.coreos.com/v1',
kind: 'PrometheusRule',
metadata: {
name: 'my-prometheus-rule',
namespace: $.values.common.namespace,
},
spec: {
groups: [
{
name: 'misc',
rules: [
{
alert: 'Ringing ear',
expr: 'vector(1)',
labels: {
severity: 'low',
},
annotations: {
description: 'Maybe too many alerts made your ear ring?',
},
},
],
},
],
},
},
}To include this in the example.jsonnnet add the following patch:
values+:: {
common+: {
namespace: 'monitoring',
},
},
+ // Import my-alert and set the namespace for it.
+ extraManifests: (import 'my-alert.jsonnet') +
+ {
+ prometheusRule+: { metadata +: { namespace: $.values.common.namespace } },
+ },The alert will end up in manifests/extra-manifests-prometheusRule.yaml when rendered. It shows up in Prometheus as “Ringing ear”.
Adding a dashboard
Create a file my-dashboard.jsonnet with the following content:
local grafana = import 'grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
local row = grafana.row;
local prometheus = grafana.prometheus;
local template = grafana.template;
local graphPanel = grafana.graphPanel;
{
'my-dashboard.json':
dashboard.new('My Dashboard')
.addTemplate(
{
current: {
text: 'Prometheus',
value: 'Prometheus',
},
hide: 0,
label: null,
name: 'datasource',
options: [],
query: 'prometheus',
refresh: 1,
regex: '',
type: 'datasource',
},
)
.addRow(
row.new()
.addPanel(graphPanel.new('My Panel', span=6, datasource='$datasource')
.addTarget(prometheus.target('vector(1)')))
),
}To include this in the example.jsonnnet add the following patch:
values+:: {
common+: {
namespace: 'monitoring',
},
+ grafana+: {
+ dashboards+:: (import 'my-dashboard.jsonnet'),
+ }
},The dashboard will end up in manifests/grafana-dashboardDefinitions.yaml in a ConfigMap when rendered. It shows up in Grafana as “My Dashboard” in the Default folder, and looks like this:
Complete example file
This is the end result with all examples added to example.jsonnet:
local addSecurityContext = {
containers: std.map(
function(container)
if container.name == "kube-state-metrics" then
container + {
"securityContext": {
"runAsGroup": 65532,
"runAsNonRoot": true,
"runAsUser": 65532
}
}
else
container,
super.containers
)
};
local kp =
(import 'kube-prometheus/main.libsonnet') +
// Uncomment the following imports to enable its patches
// (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
// (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
// (import 'kube-prometheus/addons/node-ports.libsonnet') +
// (import 'kube-prometheus/addons/static-etcd.libsonnet') +
// (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
// (import 'kube-prometheus/addons/external-metrics.libsonnet') +
{
values+:: {
common+: {
namespace: 'monitoring',
},
grafana+: {
dashboards+:: (import 'my-dashboard.jsonnet'),
}
},
// Import my-alert and set the namespace for it.
extraManifests: (import 'my-alert.jsonnet') +
{
prometheusRule+: { metadata +: { namespace: $.values.common.namespace } },
},
kubeStateMetrics+: {
deployment+: {
spec+: {
template+: {
spec+: addSecurityContext
}
}
}
},
};
{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{
['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['extra-manifests-' + name]: kp.extraManifests[name] for name in std.objectFields(kp.extraManifests) }
Conclusion
Kube-prometheus brings a comprehensive package for monitoring your Kubernetes cluster. It is built in the form of a library that can be used to quickly build your own unique rendering. The power of Jsonnet means that it is easy to extend, replace and modify parts as you see fit, without the limitations that comes from solutions that build on templates (e.g. Helm).
On the other hand, fully utilizing the library means that you have to actually learn the library and its abstractions, in addition to all the normal Kubernetes resources.
