Building and testing base images for Kubernetes cluster nodes with Packer, qemu and Chef InSpec

There are a couple of implementation details to notice. First, the builder does not start from a released ISO image of Ubuntu, but instead it uses a cloud image as a base. Release ISO distributions would need a customized boot command and would make use of pre-seed files to orchestrate the installation. This would require more implementation work while the build would last significantly longer (for example the builder will have to replace the preset boot command at a speed of one character per second). Cloud-ready images are base file systems optimized to be used for provisioning cloud instances and have less packages deployed to ensure smaller image sizes. Ubuntu cloud images such as the server release or the minimal release are shipped with the pre-installed packages required by most cloud platforms. These include cloud-init (used to dynamically push configurations to an instance at first boot) and cloud-guest-utils (containing growpart tool needed for extending partitions). If you want to find out more about building from ISO images, there are detailed steps presented in the official Packer documentation

Secondly, the builder is specifying a couple of arguments for the qemu binary. These include options that specify resource allocation (CPU cores and RAM) and a -smbios option that sets custom data into the SMBIOS table by using its serial field: "-smbios", "type=1,serial=ds=nocloud-net;instance-id=packer;seedfrom=http://{{ .HTTPIP }}:{{ .HTTPPort }}/". Our starting image is running cloud-init, thus cloud-init data sources can be used at first boot to configure the credentials needed by packer’s ssh communicator (username, ssh public key). SMBIOS is used to configure NoCloud data source of cloud-init and define the endpoint where the cloud-init configuration will be accessible. When qemu boots the image, cloud-init will attempt to fetch cloud configuration data from an endpoint provided by a server created by Packer during the build process. This http server will serve the path defined by http_directory, which is set to a local folder containing cloud-init files, over a local address and port that are accessible in the builder configuration as template variables: {{ .HTTPIP}} and {{ .HTTPPort }}.

Next step is to actually create the local folder containing the cloud-init configurations, as configured in the http_directory property of the qemu builder. We will create a nested folder structure ./cloud-init/baseos at the same level as our Packer template file. The NoCloud cloud-init data source expects to find two files meta-data and user-data. The meta-data file should be used to represent any information commonly found in an Amazon EC2 metadata service, while the user-data file is simply a user-data YAML file containing cloud-config syntax.

There is no need to push meta-data to the virtual machine, so create an empty meta-data file. If you do not have an SSH key pair, it is time to generate one. Next add the following cloud configuration to the user-data file and paste your public key under ssh-authorized-keys:

This simple cloud configuration is adding password-less sudo privileges to the default user ubuntu, it injects a public key to be used for SSH authentication, freezes the sources for package updates and disables the package updates in order to ensure a minimal reproducibility of the image build process.

Additionally we clean-up the image and reset the state of cloud-init in order to ensure that future virtual machine provisioned from our image will execute cloud configurations correctly and will satisfy the requirements set by Kubernetes for unique machine-ids and MAC addresses. There are four provisioners to be executed in the context of provisioner baseos:

"provisioners": [
{
"type": "shell",
"script": "./scripts/base-setup.sh",
"expect_disconnect": true,
"environment_vars": [
"DEBIAN_FRONTEND=noninteractive"
],
"execute_command": "sudo -S bash -c '{{ .Vars }} {{ .Path }}'",
"only": ["baseos"] },
{
"type": "shell",
"script": "./scripts/install-docker.sh",
"environment_vars": [
"DEBIAN_FRONTEND=noninteractive"
],
"execute_command": "sudo -S bash -c '{{ .Vars }} {{ .Path }}'",
"only": ["baseos"] },
{
"type": "shell",
"script": "./scripts/install-k8s-tools.sh",
"environment_vars": [
"DEBIAN_FRONTEND=noninteractive"
],
"expect_disconnect": true,
"execute_command": "sudo -S bash -c '{{ .Vars }} {{ .Path }}'",
"only": ["baseos"] ,
{
"type": "shell",
"script": "./scripts/clean-up.sh",
"environment_vars": [
"DEBIAN_FRONTEND=noninteractive"
],
"expect_disconnect": true,
"execute_command": "sudo -S bash -c '{{ .Vars }} {{ .Path }}'",
"only": ["baseos"] },
]