Golden Images and Proxmox Templates Using cloud-init

Intro #

In this tutorial, we’ll create a Proxmox template based on an Ubuntu cloud image and configured using cloud-init; with the goal of creating a cloud-init configuration file that works across Linux distributions and seamlessly integrates with Proxmox’s cloud-init tooling. Notably, this configuration file is easily modified without rebuilding the template. At the end, we will create two shell scripts that streamline the templating process by keeping cloud images up-to-date and creating a template with a simple CLI command. Thus, enabling the effortless generation of multiple templates and facilitating rapid deployment of virtual machines through the PVE GUI or your preferred provisioning tool.

Cloud-init #

Cloud-init is a configuration management tool used to automate the initialization of servers and cloud instances. It features a modular architecture, allowing the definition of various server configurations through modules. These modules handle tasks such as network configuration, user management, package installation, and file system initialization.

Configuration occurs during the final three stages of cloud-init’s boot process, with modules running once-per-instance or on every boot, always, depending on their predefined frequency. Modules are configured in user, network and vendor data files using YAML syntax, resulting in human-readable and reusable configuration files. Overall, cloud-init streamlines server provisioning and ensures consistent configuration across deployments. For a complete overview, see the official documentation (link).

Creating a Universal Cloud-Init Configuration File #

Proxmox cloud-init documentation demonstrates how to add a custom data file to cloud-init’s datasource, that is qm set 9000 --cicustom "user=local:snippets/userconfig.yaml". However, attaching one automatically replaces the corresponding Proxmox generated file for meta, network, or user data. Using this approach typically leads people to creating a user data file and statically setting values for hostname, user and ssh_authorized_keys. Then generating multiple cloud-config files for each VM/template - this is unnecessary overhead!

We are going to capitalize on the fact that Proxmox does not generate a vendor data file, but allows you to attach one. We’ll place the majority of our own user data into a vendor data file, and Proxmox will merge it’s configuration with our own. This has the major benefit of preserving the dynamically generated values from Proxmox, e.g. VM name for the hostname, or other provisioning tools that use the Proxmox API.

Note that Proxmox’s user data takes precedence over values placed into the vendor data file, as cloud-init merges user data over vendor data, so we’ll avoid duplicating configurations and discuss limitations of this approach in a later section.

First, let’s look at the typical VM’s cloud-init settings that are configured using the GUI:

This is a mixture of user and network data. Let’s look specifically at the user data generated by Proxmox using the CLI:

root@pve:~# qm cloudinit dump 100 user
#cloud-config
hostname: vm-example
manage_etc_hosts: true
fqdn: vm-example.example.com
user: jdoe
password: <PASSWORD_HASH>
ssh_authorized_keys:
  - ssh-ed25519 <PUBLIC_KEY> <USER_EMAIL>
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

Here Proxmox is setting values for the hostname, etc hosts, users, passwords, ssh and packages modules. These settings will trigger cloud-init to do the following:

• Set hostname to vm-example.example.com.
• Add 127.0.1.1 vm-example.example.com to etc/hosts.
• Overwrite the default user with jdoe and add the hashed password in /etc/cloud/cloud.cfg. Allowing you to login via the PVE console with the updated username and password.
• Append the SSH key(s) to /home/<DEFAULT_USER>/.ssh/authorized_keys.
• Upgrade all packages using the OS package manager.

So now that we have idea of the user data values Proxmox generates, let’s create a vendor data file that supplements these values.

A Minimal Vendor Data File #

At the bare minimum, Proxmox needs qemu-guest-agent installed on the image to execute commands and safely shutdown the VM. To install and update packages during the initialization process, we will use the packages module. Then, in order for Proxmox to register the guest agent, we’ll use the power state change module to reboot the VM after cloud-init has completely finished.

We’ll store our custom cloud-init files in the default Proxmox snippets directory, see storage for details.

To enable the snippets directory using the GUI, navigate to Datacenter > Storage > 'local' > Edit and highlight Snippets under the Content dropdown menu and click OK.

Alternatively, append snippets to your Proxmox configuration file /etc/pve/storage.cfg:

dir: local
  path /var/lib/vz
  content iso,backup,vztmpl,snippets
  ...

Start by creating a basic vendor-data.yaml:

# SSH into your Proxmox host
user@desktop:~$ ssh root@pve
# create a new vendor-data.yaml file
root@pve:~# vi /var/lib/vz/snippets/vendor-data.yaml

Add the following to vendor-data.yaml:

#cloud-config
packages:
  - qemu-guest-agent
package_update: true
power_state:
  mode: reboot
  timeout: 30

These settings do the following:

• Install qemu-guest-agent using the default OS package manager.
• Update the package database & upgrade all installed packages (this is set in the Proxmox config).
• Reboot the VM 30 seconds after all cloud-init modules have finished.

Next let’s create a simple VM template and attach this vendor file.

Proxmox Template #

We’ll extend on Proxmox’s cloud-init docs example with details from the qm(1) manual page to create a Ubuntu VM template. Yet, we’ll focus on abstracting several of the CLI arguments to create a reusable shell script later.

Download a Cloud Image #

Start by downloading a Ubuntu 20.04 cloud image on your proxmox host.

# SSH into Proxmox
user@desktop:~$ ssh root@pve

# change into the ISO directory
root@pve:~# cd /var/lib/vz/template/iso

# download the image
wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img

# compare shasum
curl -sfL https://cloud-images.ubuntu.com/releases/focal/release/SHA256SUMS | shasum -c --ignore-missing

Create a Template #

Next, set several environment variables to streamline the templating process. Note: Set the VM image storage value for your specific PVE host, for example local-lvm or local-zfs.

## On the PVE Host ##
# set VM variables
export VM_ID=9000
export VM_NAME="ubuntu20"
export VM_IMAGE=/var/lib/vz/template/iso/ubuntu-20.04-server-cloudimg-amd64.img
export VM_STORAGE="local-lvm"

For the template we will:

1. Enable QEMU guest agent support using --agent enabled=1.
2. Attach the Ubuntu cloud image using qm disk import and set it as boot.
3. Attach our custom vendor data file with --cicustom.

Paste the following commands into the terminal:

# create a new VM
qm create "${VM_ID}" --name "${VM_NAME}" \
  --description "created on $(date)" \
  --ostype l26 \
  --bios seabios --machine q35 \
  --scsihw virtio-scsi-pci \
  --cpu cputype=host --memory 1024 \
  --net0 virtio,bridge=vmbr0,tag=1

# enable QEMU guest agent
qm set "${VM_ID}" --agent enabled=1

# import the cloud image
qm disk import "${VM_ID}" "${VM_IMAGE}" "${VM_STORAGE}"

# attach the disk to the VM and set it as boot
qm set "${VM_ID}" --boot order=scsi0 \
  --scsi0 "${VM_STORAGE}":vm-"${VM_ID}"-disk-0,cache=writeback,discard=on,ssd=1

# increase the disk image size
qm resize "${VM_ID}" scsi0 +1G

# add cloud-init drive
qm set "${VM_ID}" --ide2 "${VM_STORAGE}":cloudinit --ipconfig0 ip=dhcp \
  --citype nocloud

# add cloud-init vendor config file
qm set "${VM_ID}" --cicustom "vendor=local:snippets/vendor-data.yaml"

# convert the VM into a template
qm template "${VM_ID}"

Test the Template #

Feel free to test out the new template by creating a clone. Calling qm agent now works due to installing and enabling QEMU guest agent.

## On the PVE Host ##
# create a clone and start it
qm clone $VM_ID 100 --full --name vm-example
qm start 100
# wait for cloud-init to finish and reboot the VM
...
# view network information
qm agent 100 network-get-interfaces

Creating Shell Scripts #

Finally, let’s create two simple scripts to 1) download and/or check that a cloud image is up to date; and 2) create a template. Note: These examples are the foundational idea behind the full scripts that have more options, built-in error checking, and compatible with systemd timers. Grab the full scripts from the GitHub repo (link).

Automatically Update Cloud Images #

Let’s create a simple script that will compare the shasum of our Ubuntu image, and download a new one if the file is out of date.

Create a new file basic-image-updater in /usr/local/bin/:

# use your favorite editor
root@pve:~# vi /usr/local/bin/basic-image-updater

Add the following to it:

#!/bin/bash

FILE_NAME="ubuntu-20.04-server-cloudimg-amd64.img"
REMOTE_URL="https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img"
REMOTE_SHASUM_URL="https://cloud-images.ubuntu.com/releases/focal/release/SHA256SUMS"
STORAGE_PATH=/var/lib/vz/template/iso
tmpfile=$(mktemp /tmp/image-shasum.XXXXXX)

# change into PVE iso directory
cd "${STORAGE_PATH}" || exit

# get latest shasums
curl -L "${REMOTE_SHASUM_URL}" -o "${tmpfile}"

latest_shasum=$(grep "${FILE_NAME}" "${tmpfile}" | awk '{print $1}')
current_shasum=$(shasum -a 256 "${FILE_NAME}" | awk '{print $1}')

rm "$tmpfile"

if [[ $latest_shasum == "${current_shasum}" ]]; then
  echo "SHASUM match, image is up-to-date."
else
  echo "No SHASUM match, downloading new image..."
  curl -sL "${REMOTE_URL}" -o "${FILE_NAME}"
fi

exit

You can manually run this script by calling it from the CLI to download the missing image. Note: the majority of cloud images are only updated every 1 to 2 months.

root@pve:~# basic-image-updater
shasum: ubuntu-20.04-server-cloudimg-amd64.img: No such file or directory
No SHASUM match, downloading new image...

Finally, let’s create a cron job to run this script monthly:

# create/edit crontab
crontab -e

# add a cron job
0 2 10 * * /usr/local/bin/basic-image-updater

Create Proxmox Template #

Let’s create another simple script to streamline the templating process. We’ll create this file in /usr/local/bin/, so PVE users can call the script from any directory. Start by creating a file basic-template-builder:

# use your favorite editor
root@pve:~# vi /usr/local/bin/basic-template-builder

Let’s extend the prior CLI commands into a basic script. Add the following to it and insert the PVE template ‘qm’ commands from above:

#!/bin/bash

# use environment vars or default values
VM_ID=${VM_ID:-"9000"}
VM_NAME=${VM_NAME:-"template"}
VM_IMAGE=${VM_IMAGE:-""}
VM_STORAGE=${VM_STORAGE:-"local-lvm"}

# send error messages to STDERR
function err() {
  echo "Error: $*" >&2
  exit 2
}

# simple error and help message
function usage() {
  printf "Usage: %s --id <VM_ID> --name <VM_NAME> --img <VM_IMAGE> --storage <PVE_STORAGE> \n" "${0##*/}" >&2
  exit 2
}

# parse CLI args
for _ in "$@"; do
  [[ ${2} == -* ]] && {
    err "Missing argument for ${1}"
  }
  case "${1}" in
  --help | -h)
    usage
    ;;
  --id)
    VM_ID=${2}
    shift 2
    ;;
  --name)
    VM_NAME=${2}
    shift 2
    ;;
  --img)
    VM_IMAGE=${2}
    shift 2
    ;;
  --storage)
    VM_STORAGE=${2}
    shift 2
    ;;
  -*)
    echo "Unknown Argument: $1"
    usage
    ;;
  esac
done

# check required variables
if [ -z "${VM_ID}" ]; then err "Missing ID"; fi
if [ -z "${VM_NAME}" ]; then err "Missing Name"; fi
if [ ! -e "${VM_IMAGE}" ]; then err "Image file not found! ${VM_IMAGE}"; fi

### add PVE template 'qm' commands from above ###

exit

Test it out.

# using CLI args
basic-template-builder --id 9000 --name ubuntu20 --img /var/lib/vz/template/iso/ubuntu-20.04-server-cloudimg-amd64.img

# using environment vars
export VM_ID=9000 VM_NAME="ubuntu20" VM_STORAGE="local-lvm"
export VM_IMAGE=/var/lib/vz/template/iso/ubuntu-20.04-server-cloudimg-amd64.img

basic-template-builder

Recap #

Overall, we created a cloud-init configuration file to supplement the dynamically generated values from Proxmox and a template that utilizes Proxmox’s user data alongside our own user data, disguised as a vendor data file. We also created two shell scripts to keep the Ubuntu cloud image up-to-date and facilitate template creation using a simple CLI command, basic-template-builder --id <VM_ID> --name <VM_NAME> --img <VM_IMAGE>.

I personally prefer this approach over others that I have written about, as it keeps your templates modular and allows updating the vendor data file after template creation. Additionally, these templates work with external provisioning tools, e.g. Terraform, by utilizing the automatically generated VM names and VM IDs. Checkout the blog post: Provisioning Proxmox VMs with Terraform (link) for details on quickly deploying these templates with Terraform.

At this point, feel free to explore templating with cloud-init on your own. Yet, there are a few other points worth discussing.

Limitations of Using Vendor Data #

• You cannot add additional users or SSH keys using the users module, as user data generated by PVE takes precedence over vendor data.

• Modifying the custom vendor data file after creating a template will apply the updated configuration to all future clones. To apply the changes to running VMs, run qm cloudinit update <vmid> or click the Regenerate Image button in the PVE GUI to manually regenerate the cloud-init config drive - as Proxmox does not monitor changes to snippets directory and will not automatically regenerate the drive.

  • To view the config drive, read the contents of /dev/sr0/ on a running VM.

• Modifying the custom vendor data file on a running VM does not change the instance-id for the VM, so cloud-init will not re-run once-per-instance modules. Conversely, modules that always run will apply changes at boot.

• If you rename the vendor file, you must update the template configuration by running the CLI command below. Additionally, all pre-existing clones must be updated as well, otherwise Proxmox blocks the VM from starting and throws a TASK ERROR: volume 'local:snippets/vendor-data.yaml' does not exist.

  # update template to use renamed file
  qm set $VM_ID --cicustom "vendor=local:snippets/<NEW_FILENAME>.yaml"
  

Regenerating the Config Drive vs Re-running Cloud-init #

There are two important and distinct steps in Proxmox’s implementation of cloud-init, one is creating/updating the instance-id for cloud-init, and another is generating a cloud-init config drive.

Anytime a new VM is created or cloned, Proxmox generates cloud-init meta data with a instance-id that is unique to each VM. Cloud-init tracks this instance-id and determines whether to run once-per-instance modules.

Proxmox also generates a cloud-init config drive that is unique for each VM and attaches it as a cd-rom drive mounted at /dev/sr0. You can manually regenerate the drive by running qm cloudinit update <vmid> or using the Regenerate Image button in the PVE GUI. However, regenerating the drive does not change the instance-id, thus, cloud-init will not re-run once-per-instance modules.

One method for changing the instance-id is to modify the cloud-init data within the Proxmox GUI.

Updating Cloud-init Data in the PVE GUI #

Proxmox actively monitors for changes to user and network data. Updating cloud-init settings in the GUI triggers Proxmox to update the cloud-init meta data with a new instance-id and regenerate the cloud-init config drive. On the following boot, cloud-init regenerates SSH keys and creates new folder /var/lib/cloud/instances/<instance-id>, symlinking /var/lib/cloud/instances/ to it.

# view the updated instance-id
root@pve:~# qm cloudinit dump 100 meta
instance-id: ab42f82195a1a9315a0b90295b8ed5ee24d251f1

# ssh into the VM
root@pve:~# ssh root@vm-example

# view the cloud-init configuration
root@vm-example:~# tree /var/lib/cloud/
/var/lib/cloud/
├── data
│   ├── instance-id
│   ├── previous-datasource
│   ├── previous-hostname
│   ├── previous-instance-id
│   ├── python-version
│   ├── result.json
│   ├── set-hostname
│   └── status.json
├── handlers
├── instance -> /var/lib/cloud/instances/ab42f82195a1a9315a0b90295b8ed5ee24d251f1
├── instances
│   ├── ab42f82195a1a9315a0b90295b8ed5ee24d251f1
│   │   ├── boot-finished
│   │   ├── cloud-config.txt
...
│   └── f36c5e1ee9827152d057b81eb8ec65e8769f3510
│       ├── cloud-config.txt
...

Additionally, cloud-init creates the file /var/lib/cloud/instances/boot-finished with the updated date, cloud-init version (21.4) and distribution release (Ubuntu 20.04.1).

root@vm-example:~# cat /var/lib/cloud/instance/boot-finished
10.63 - Sat, 12 Mar 2022 17:53:33 +0000 - v. 21.4-0ubuntu1~20.04.1

# view changes to running VM
root@pve:~# qm cloudinit pending 100
cur cicustom: vendor=local:snippets/vendor-data.yaml
cur citype: nocloud
new ciuser: jdoe

## reboot the VM ##

# ssh into the VM
root@pve:~# ssh root@vm-example

# list users
root@vm-example:~# tail -n 2 /etc/passwd
ubuntu:x :1000:1000:Ubuntu:/home/ubuntu:/bin/bash
jdoe:x :1001:1001:Ubuntu:/home/jdoe:/bin/bash

Working with qcow2 and raw Images #

In this post, we used one of the easier images to work with as Ubuntu provides cloud images in the raw format with the *.img extension. Other Linux distributions provide cloud images with *.qcow2 and *.raw extensions. You can use these images to create Proxmox templates and store them in the default folder /var/lib/vz/templates/iso, however, Proxmox’s GUI will mask these files, as it only displays *.img and *.iso files. A easy workaround is to change the file extension to *.img, as Proxmox qm disk import will convert image formats supported by qemu-img. With this workaround, you can use qemu-img info to view the actual format and size of the image.

# change the extension
mv debian-11-generic-amd64.{qcow2,img}

# inspect the image
root@pve:~# qemu-img info debian-11-generic-amd64.img
image: debian-11-generic-amd64.img
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 340 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
Child node '/file':
    filename: debian-11-generic-amd64.img
    protocol type: file
    file length: 340 MiB (356127744 bytes)
    disk size: 340 MiB

If you would like to manually convert *.qcow2 to *.img, you can use the following commands on a linux desktop or PVE host:

# install the qemu package (pre-installed on Proxmox)
sudo apt install qemu

# convert the image
qemu-img convert -f qcow2 -O raw image.qcow2 image.img

For *.raw to *.img just change the extension:

# download the raw or tar file
wget <FILE_URL>
# extract the tar
tar -xvf <FILE>
# change the file extension
mv disk.{raw,img}

Here are a few tips when working with these formats:

• For *.qcow2 images, the SHASUM of a converted unmodified image will match the values of the remote *.raw image.
• For *.raw images, if you wish to save on bandwidth download the *.tar archive. Archives typically extract to a file named disk.raw.

Common Bugs #

CentOS: Probing EDD #

In CentOS, the boot will appear to hang and you’ll see the following message in the console:

Probing EDD (edd=off to disable)... ok

There are two causes for this, 1) the bootloader sets console to serial, console=ttyS0,115200n8, and the console output is masked in PVE GUI; 2) Enhanced Disk Drive Services (EDD) is enabled. Typically, you can ignore this message, and cloud-init will run then reboot the machine.

If you wish to modify the bootloader on the image, checkout the post on using libguestfs.

CentOS: UEFI and Secure Boot #

Use seabios with CentOS, as it requires additional configuration to get secure boot to work with UEFI. I plan to write-up the fix at a later time.

Acknowledgements #

Thanks to all the users in this forum post and the user(s) who pushed the patch to permit attaching vendor data to VMs! And to you for reading this far!

References #

Cloud-init:

• cloud-init
  • user data
  • network data
  • vendor data
• cloud-init docs: Config Examples
• cloud-init docs: Module Reference
• Ubuntu Docs: How to use cloud-init
• Ubuntu Docs: Automated Server Installation

Cloud-init Modules: apt, bootcmd, etc hosts, hostname, passwords, packages, power state change, ssh, users

Proxmox:

• Proxmox Wiki
  • qemu-guest-agent
  • Cloud-Init Support
  • Cloud-Init FAQ
• Proxmox Forum: Combining custom cloud init with auto-generated

Proxmox Storage:

• Proxmox Docs: Performance Tweaks
• Proxmox Forum: VirtIO vs SCSI

Linux Cloud Images:

• OpenStack: Cloud Images, collection of image links.
• CentOS Cloud Images
  • Default User: centos or cloud-user
  • Use CentOS-Stream-GenericCloud-X-latest.x86_64.qcow2
• Debian Cloud Images
  • Default User: debian
  • Use debian-1x-generic-amd64.qcow2, .raw or .tar.zx
  • Avoid:
    • genericcloud images fail to run cloud-init
    • nocloud images do not have cloud-init installed and defaults to a password-less root user.
• Fedora Cloud Images
  • Default User: fedora
  • Use Fedora-Cloud-Base-Generic.x86_64-XX-XX.qcow2
• Ubuntu Cloud Images
  • Default User: ubuntu
  • Use ubuntu-2x.04-server-cloudimg-amd64.img