Golden Images and Proxmox Templates Using libguestfs

Table of Contents

Intro #

In the last post, we created a custom Proxmox template that was configured using cloud-init. While that approach is one of the best ways to configure a new virtual machine (VM), there are several use cases where you may need to modify an image prior to booting a VM. This is easily achieved using libguestfs, which allows you to directly modify cloud images using shell commands.

Libguestfs is a C library and collection of tools that lets you:

Install packages
Transfer files in and out
Directly edit files with your favorite editor
Run shell scripts
And more

Many of the commands mimic familiar Linux commands, e.g. virt-ls lists files within a disk image, making the transition seamless. Additionally, the package, libguestfs-tools, is easily installed via the APT and YUM repositories.

In this tutorial, we’ll use libguestfs to modify a CentOS 9 Stream image and explore its capabilities by:

Installing qemu-guest-agent for seamless VM management
Fortifying the SSH server configuration for enhanced security
Fixing the elusive Probing EDD ... boot screen with a simple tweak

We’ll test out our modifications by creating a simple Proxmox template and deploying a VM.

I recommend modifying images on a local Linux desktop or VM rather than directly on the Proxmox host.

Setup: Download Image #

To start, create a new working directory on your desktop and download the image:

mkdir -p ~/centos9_stream_mod && cd ~/centos9_stream_mod

wget https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2

And validate the checksum of the image:

curl -sfL https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2.SHA256SUM | shasum -c --ignore-missing

Optional, create a folder and copy of the original unmodified image:

mkdir original_img && \
cp CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 original_img/

Setup: Install libguestfs #

Next, install libguestfs using your OS package manager. For example on Debian/Ubuntu:

# install the libguestfs
sudo apt install libguestfs-tools

Modifying Images Using libguestfs #

Installing Packages #

Virt-customize allows you to customize disk images in place, including: installing and uninstalling packages using the distribution’s package manager; editing user passwords; running shell commands; and more. Given, qemu-guest-agent is required for Proxmox to execute commands and safely shut down a VM, let’s install it using virt-customize with the --install option:

# install qemu-guest-agent
sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  --install qemu-guest-agent

Warning: Running virt-customize and installing packages will set the machine-id. Therefore, all VM clones from distros that use systemd-networkd will have the same DHCP unique identifier (DUID) and receive the same IP address. Be sure to run the following command to clear the ID!

# delete the machine id
sudo virt-edit -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  /etc/machine-id -e 's/.*//'

You can confirm that the machine-id is deleted by running virt-cat:

sudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  /etc/machine-id

Retrieving and Adding Files to the Image #

You can copy files in and out of the image using libguestfs’ virt-copy-in and virt-copy-out. This is great when you have pre-existing configurations or wish to edit a file using an IDE, yet we’ll directly edit the SSH configuration in the next section. Here’s an example of how to use these commands:

# pull the SSH host config
# syntax: virt-copy-out -a <IMAGE> <FILE_PATH> <OUTPUT_PATH>
sudo virt-copy-out -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  /etc/ssh/sshd_config .

# edit the file using your favorite editor

# push the SSH server config back into the image
# syntax: virt-copy-in -a <IMAGE> <FILE> <FILE_PATH>
sudo virt-copy-in -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  sshd_config /etc/ssh/

Directly Editing Files in the Image #

You can also edit files directly with virt-edit. We’ll use this approach to edit the sshd_config, enter the following command:

sudo virt-edit -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  /etc/ssh/sshd_config

The default editor is vi/vim. To use a different editor, append EDITOR=<editor> to the command, i.e. sudo EDITOR=nano virt-edit ..., or permanently set it via a systemd-wide environment variable:

echo 'EDITOR="nano"' | sudo tee -a /etc/environment

Then let’s enhance the security of the SSH server, change the settings in sshd_config to the following values:

# disable root login
PermitRootLogin no

# limit the number of failed login attempts before locking out a user
MaxAuthTries 3

# disable password authentication for all users
PasswordAuthentication no
PermitEmptyPasswords no

# disable X11 forwarding
X11Forwarding no

Feel free to set the values according to your security needs. When you’re done, save and exit the editor. You can confirm the changes by running virt-cat:

sudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  /etc/ssh/sshd_config

Running Shell Commands #

Finally, we are going to edit the bootloader configuration to fix the empty boot screen you may see in the Proxmox VM console that only reads: Probing EDD (edd=off to disable)... ok. While this not a hard requirement to get CentOS booted and configured using cloud-init, it provides a good example of how to use virt-customize to run shell commands.

The empty boot screen is caused by two factors: i) the bootloader sets the console to serial, console=ttyS0,115200n8, and the output is masked in PVE GUI; ii) Enhanced Disk Drive (EDD) is enabled. There are several ways to handle this, including:

Ignore it, hope the boot process completes without error and wait for a login screen.
Add a serial terminal to the VM and monitor the boot process from the PVE server console using qm terminal <VMID>.
Add a virtual terminal to the bootloader configuration. Optionally, disable Enhanced Disk Drive.

We’ll go with the last approach as many find it easier use Proxmox’s GUI console to monitor the boot process. To change the console output and disable EDD (optional), we’ll start by using grubby to edit the default grub configuration:

# update the kernel args - warning this sets the machine-id!
sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  --run-command 'grubby  --args="console=tty0"  --update-kernel=ALL'

Grubby will either add a new argument, or replace an existing one. In this case we want to replace console=ttyS0,115200n8 with a new argument that is more appropriate for PVE GUI: console=tty0. The --args argument is used to pass arguments to the bootloader, and --update-kernel=ALL will update all kernels.

If you would like to disable Enhanced Disk Drive (EDD) Services, add edd=off:

# change console and disable EDD
grubby --args="console=tty0 edd=off" ...

For more details checkout man grubby or grubby.

Next, let’s go ahead and update the grub config using grub2-mkconfig:

# update grub config - warning this sets the machine-id!
sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  --run-command 'grub2-mkconfig -o /boot/grub2/grub.cfg'

Now we can confirm that the changes were successful by checking the default grub config, /etc/default/grub:

# confirm the edit
sudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  /etc/default/grub

The new arguments are appended to the end of GRUB_CMDLINE_LINUX:

# /etc/default/grub example
GRUB_TIMEOUT=1
...
GRUB_CMDLINE_LINUX="no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M console=tty0"
...

And we can also check the bootloader entry, /boot/loader/entries/<MACHINE_ID>-<KERNEL_VERSION>.conf:

# get the name of the bootloader file
BOOTLOADER_FILE=$(sudo virt-ls -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 /boot/loader/entries)

# confirm the edit
sudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  /boot/loader/entries/${BOOTLOADER_FILE}

The new args are appended to the options line:

# /boot/loader/entries/<MACHINE_ID>-<KERNEL_VERSION>.conf example
title CentOS Stream (<KERNEL_VERSION>.el9.x86_64) 9
...
options root=UUID=<UUID_VALUE> ro no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M console=tty0
...

Create a Proxmox Template #

Now that we have a modified image, let’s create a Proxmox template from it.

Change the File Extension #

In order to view the image within the Proxmox GUI, the file extension needs to be .img or .iso. While you can manually convert the image (see box below), you can trick Proxmox by changing the file extension and it will automatically convert the image as it is imported into a VM. This approach also saves space on your server. To change the file extension, run:

# change the file extension
mv CentOS-Stream-GenericCloud-9-latest.x86_64.{qcow2,img}

You can use qemu-img to convert the image format from qcow2 to img, however, the converted file size is typically larger. To view the size of the image before converting, run qemu-image info and view the virtual size value.

# install QEMU (pre-installed on Proxmox)
sudo apt install qemu

# view image information
user@desktop:~$ qemu-img info <IMAGE>
image: CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes) # converted size
disk size: 1.13 GiB                      # original size
...

To convert the image run:

# convert the image
qemu-img convert -f qcow2 -O raw <INPUT>.qcow2 <OUTPUT>.img

Upload the Image #

Now transfer the image to your Proxmox server using your preferred method, e.g. web GUI or scp.

To transfer the file using the web GUI, navigate to your storage pool for ISO images, i.e. Datacenter > NODE_NAME > 'local' > ISO Images > Upload, and select the image and click Upload.

Proxmox GUI showing the server storage panel for uploading ISO images

Alternatively, you can use scp to copy the file into your ISO storage, e.g. /var/lib/vz/template/iso:

# copy image to proxmox server
scp CentOS-Stream-GenericCloud-9-latest.x86_64.img \
  root@<PROXMOX_ADDRESS>:/var/lib/vz/template/iso

Create a Template #

Next, let’s create the template using the image we just uploaded. We’ll set several environment variables to make the process easier. Start by logging into your Proxmox server terminal and running the following commands as a privileged user:

Set VM_STORAGE to your specific VM image storage, e.g. local-lvm or local-zfs.

## On the PVE Server ##
# set VM variables
export VM_ID=9009
export VM_NAME="centos9"
export VM_IMAGE=/var/lib/vz/template/iso/CentOS-Stream-GenericCloud-9-latest.x86_64.img
export VM_STORAGE="local-lvm"

Run the following code to create a simple VM with the modified disk image attached:

# create a new VM
qm create "${VM_ID}" --name "${VM_NAME}" \
  --description "created on $(date)" \
  --ostype l26 \
  --bios seabios --machine q35 \
  --scsihw virtio-scsi-pci \
  --cpu cputype=host --memory 1024 \
  --net0 virtio,bridge=vmbr0,tag=1

# enable QEMU guest agent
qm set "${VM_ID}" --agent enabled=1

# import the cloud image
qm disk import "${VM_ID}" "${VM_IMAGE}" "${VM_STORAGE}"

# attach the disk to the VM and set it as boot
qm set "${VM_ID}" --boot order=scsi0 \
  --scsi0 "${VM_STORAGE}":vm-"${VM_ID}"-disk-0,cache=writeback,discard=on,ssd=1

# add cloud-init drive
qm set "${VM_ID}" --ide2 "${VM_STORAGE}":cloudinit --ipconfig0 ip=dhcp \
  --citype nocloud

Now, convert the VM to a template:

# convert the VM into a template
qm template "${VM_ID}"

For additional configuration options see man qm or qm(1). For a bash script to simplify the above steps, checkout the build-template script on github or companion blog post.

Test the Template #

Create a new VM from your template:

## On the PVE Server ##
# create a clone and start it
qm clone ${VM_ID} 100 --full --name ${VM_NAME}-test
qm start 100

Because qemu-guest-agent is installed on the VM, you can use qm agent to get information on the running VM:

# view network information
qm agent 100 network-get-interfaces

If you would like to SSH into the VM, add a SSH key to the VM cloud-init configuration using the CLI or web GUI, then reboot the VM.

To add a key using the CLI:

## On the PVE Server ##
root@pve:~# qm set 100 --sshkeys ~/.ssh/id_ed25519.pub
root@pve:~# qm reboot 100

To add a key via the web GUI, navigate to the VM’s “Cloud-init” tab then click the “SSH public key” line and the “Edit” button. Then, reboot the VM.

Proxmox GUI showing the virtual machine cloud-init settings panel for SSH keys

Now try SSHing into the VM using the default account (centos or cloud-user):

ssh cloud-user@<VM_IP_ADDRESS>

Feel free to explore the VM, otherwise cleanup your environment by running:

# stop and remove the VM
root@pve:~# qm stop 100
root@pve:~# qm destroy 100

Recap #

Overall, utilizing libguestfs offers a powerful and efficient approach for pre-boot customization of cloud images. While this approach is more manual than relying on cloud-init, it offers more flexibility in terms of customizing the OS bootloader. Just be aware anytime a libguestfs command outputs Setting the machine ID in /etc/machine-id, you must clear the ID or replace the value with uninitialized. For tips on building clean images, checkout Systemd: Building Images Safely and my other posts on Proxmox templates. Thanks for reading!

Additional Libguestfs Commands #

Editing User Passwords #

The SELECTOR field takes one of the password inputs (link).

# change root password
# syntax: virt-customize -a <IMAGE> --root-password <SELECTOR>
sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \
  --root-password password:SecretRootPassword

Troubleshooting #

Libguestfs: Error Status 1 #

libguestfs: error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

This often occurs when you run libguestfs as a non-privileged user, try adding sudo to the command. Otherwise, add the LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 environment variables, run the command again and check the log output. Example output:

# error when running as non privileged user
libguestfs: trace: disk_create "/tmp/libguestfstmQ4KK/overlay1.qcow2" "qcow2" -1 "backingfile:/home/user/libguestfs-demo/ubuntu-20.04-server-cloudimg-amd64.img"
...
cp: cannot open '/boot/vmlinuz-5.11.0-7620-generic' for reading: Permission denied
supermin: cp -p '/boot/vmlinuz-5.11.0-7620-generic' '/var/tmp/.guestfs-1000/appliance.d.8d88y3ny/kernel': command failed, see earlier errors
libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above
libguestfs: trace: launch = -1 (error)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x60fa326dd050 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfstmQ4KK

References #

Cloud Images:

OpenStack: Cloud Images, collection image links.
CentOS Cloud Images
- Default User: centos or cloud-user
- Use CentOS-Stream-generic-x-latest.x86_64.qcow2
Debian Cloud Images
- Default User: debian
- Use debian-1x-generic-amd64.qcow2
Ubuntu Cloud Images
- Default User: ubuntu
- Use ubuntu-2x.04-server-cloudimg-amd64.img

Cloud-init:

Cloud-init Documentation
- Cloud-init Docs: Config Examples
- Cloud-init Docs: Module Reference

Proxmox:

Tools:

Other: