Backup Kubernetes – how and why (updated for 2020)

Backup a single control plane Node

As mentioned previously, we need to backup etcd. In addition to that, we need the certificates and optionally the kubeadm configuration file for easily restoring the master. If you set up your cluster using kubeadm (with no special configuration) you can do it similar to this:

# Backup certificates
sudo cp -r /etc/kubernetes/pki backup/
# Make etcd snapshot
sudo docker run --rm -v $(pwd)/backup:/backup \
    --network host \
    -v /etc/kubernetes/pki/etcd:/etc/kubernetes/pki/etcd \
    --env ETCDCTL_API=3 \
    k8s.gcr.io/etcd:3.4.3-0 \
    etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
    snapshot save /backup/etcd-snapshot-latest.db

# Backup kubeadm-config
sudo cp /etc/kubeadm/kubeadm-config.yaml backup/

Note that the contents of the backup folder should then be stored somewhere safe, where it can survive if the control plane is completely destroyed. You perhaps want to use e.g. AWS S3 (or similar) for this.

So what is really going on here? There are three commands in the example and all of them should be run on the control plane Node. The first one copies the folder containing all the certificates that kubeadm creates. These certificates are used for secure communications between the various components in a Kubernetes cluster. The final command is optional and only relevant if you use a configuration file for kubeadm. Storing this file makes it easy to initialize the master with the exact same configuration as before when restoring it.

etcd snapshot explanation

The second command needs a bit more explaining.  First of all, the idea is to create a snapshot of the etcd database. This is done by communicating with the running etcd instance in Kubernetes and asking it to create a snapshot. The reason for the very long command is basically to avoid messing with etcd running in Kubernetes as much as possible. We are launching a separate container using the same docker image that kubeadm used for setting up the cluster (k8s.gcr.io/etcd:3.4.3-0). But in order to communicate with the etcd pod in Kubernetes, we need to:

• Use the host network in order to access 127.0.0.1:2379, where etcd is exposed (--network host)
• Mount the backup folder where we want to save the snapshot (-v $(pwd)/backup:/backup)
• Mount the folder containing the certificates needed to access etcd (-v /etc/kubernetes/pki/etcd:/etc/kubernetes/pki/etcd)
• Specify the correct etcd API version as environment variable (--env ETCDCTL_API=3)
• The actual command for creating a snapshot (etcdctl snapshot save /backup/etcd-snapshot-latest.db)
• Some flags for the etcdctl command
  • Specify where to connect to (--endpoints=https://127.0.0.1:2379)
  • Specify certificates to use (--cacert=..., --cert=..., --key=...)

So we start a docker container with the etcdctl tool installed. We tell it to create a snapshot of the etcd instance running in the Kubernetes cluster and store it in a backup folder that we mount from the host.

Restore a single control plane Node

When the time has come to restore the control plane, just copy everything back from the backup and initiate the control plane again. If you want to simulate a control plane Node failing you can for example run “kubeadm reset” for a “soft” destruction. But if you really want to make sure you can set it up from zero, you should delete the VM or format the disk. In this case you must remember to do all the prerequisites before initializing it again (e.g. install kubeadm).

The restoration may look something like this:

# Restore certificates
sudo cp -r backup/pki /etc/kubernetes/

# Restore etcd backup
sudo mkdir -p /var/lib/etcd
sudo docker run --rm \
    -v $(pwd)/backup:/backup \
    -v /var/lib/etcd:/var/lib/etcd \
    --env ETCDCTL_API=3 \
    k8s.gcr.io/etcd:3.4.3-0 \
    /bin/sh -c "etcdctl snapshot restore '/backup/etcd-snapshot-latest.db' ; mv /default.etcd/member/ /var/lib/etcd/"

# Restore kubeadm-config
sudo mkdir /etc/kubeadm
sudo cp backup/kubeadm-config.yaml /etc/kubeadm/

# Initialize the master with backup
sudo kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcd \
    --config /etc/kubeadm/kubeadm-config.yaml

This is pretty much a reversal of the previous steps. Certificates and kubeadm configuration file are restored from the backup location simply by copying files and folders back to where they were. For etcd we restore the snapshot and then move the data to /var/lib/etcd, since that is where kubeadm will tell etcd to store its data.

Note that we have to add an extra flag to the kubeadm init command (--ignore-preflight-errors=DirAvailable--var-lib-etcd) to acknowledge that we want to use the pre-existing data.

Automate etcd backups

Doing a single backup manually may be a good first step but you really need to make regular backups for them to be useful. In other words, let’s automate the procedure! The easiest way to do this is probably to take the commands from the example above, create a small script and a cron job that runs the script every now and then. But since we are running Kubernetes anyway, why not use a Kubernetes CronJob? This would allow you to keep track of the backup jobs inside Kubernetes just like you monitor your workloads!

For more details on how to set up the CronJob check this post from consol labs.

About application data and resources

State is the tricky part here, as is often the case. If your workload is completely stateless, congratulations! You just have to store your YAML manifests somewhere safe and kubectl apply them where ever you want. Unfortunately, everything becomes harder if you have to deal with state. What and how to backup depends on how you are running Kubernetes. Velero may be a good alternative if you are using one of the supported storage providers. Otherwise, you have to investigate how to make snapshots on your chosen provider manually. Databases all have their own backup and restore procedures.