↓Skip to main content

Using Terraform to Create Proxmox Templates

·12 mins
Cloud-Init Proxmox Proxmox-Template Terraform
Table of Contents

Intro #

Last post we used the BPG Proxmox provider to provision VMs using a pre-existing template. This post we’ll explore using the provider to create a Proxmox template, from downloading the image to building and customizing the template. You’ll need to have Terraform installed on your local machine and SSH access to a Proxmox VE (PVE) server with a privileged user account.

BPG module is available on GitHub (link)

Grant Terraform Access to Proxmox #

In order for Terraform to be able to create templates in Proxmox, it will need to have SSH access to the PVE server. Unlike prior post, we’ll create a new linux user, terraform, on the PVE host server and add that user within the Proxmox UI under the PAM realm.

If you already created a user under the PVE realm, terraform@pve, I suggest removing that user and permissions to avoid confusion:

# delete user and token
pveum user delete terraform@pve

For additional information, see pveum help user or Proxmox Docs: User Authentication.

Create User on PVE Server #

First, start by logging into the PVE server as a privileged user. Then create a new linux user and password:

# SSH into the PVE server
ssh root@<PVE_SERVER_ADDRESS>

# create user 'terraform'
adduser --home /home/terraform --shell /bin/bash terraform

# add user to sudoers
usermod -aG sudo terraform

Next, give the terraform user permission to use PVE commands without a password. We’ll use the commands from the provider documentation on ‘SSH User’:

# create a sudoers file for terraform user
visudo -f /etc/sudoers.d/terraform

Add the following line at the end of the file and save it:

terraform ALL=(root) NOPASSWD: /sbin/pvesm
terraform ALL=(root) NOPASSWD: /sbin/qm
terraform ALL=(root) NOPASSWD: /usr/bin/tee /var/lib/vz/*

SSH Access #

Now let’s create a dedicated SSH key and add it to the /home/terraform/.ssh/authorized_keys file on the PVE server. Start by creating a new key on your local desktop using ssh-keygen:

# create a new ssh key pair
ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/terraform_id_ed25519 -C "USER_EMAIL"

# example output
user@desktop:~$ ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/terraform_id_ed25519 -C "USER_EMAIL"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/terraform_id_ed25519
Your public key has been saved in /home/user/.ssh/terraform_id_ed25519.pub
The key fingerprint is:
SHA256:uimKIsvhgDs8J5B155+e4S1HSKl9XewoqsgE1SEVLNA USER_EMAIL
The key's randomart image is:
+--[ED25519 256]--+
|  .o.o+.         |
|    Eo..         |
|    ...  .   .   |
|  ... . o     o  |
| o.. o +S. . +   |
|+  .  o.o + o .  |
|=.  . ...= .     |
|B*.+.. +=+.      |
|*==.o.+o+o.      |
+----[SHA256]-----+

# view the public key
user@desktop:~$ cat ~/.ssh/terraform_id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPQYTV18SN+39z9W99SzaJoc8VncoLyjhLulVH+pkkZ2 USER_EMAIL

Now, add the key to the terraform user’s authorized keys file on the Proxmox server. You can do this by running ssh-copy-id from your local desktop:

ssh-copy-id -i ~/.ssh/terraform_id_ed25519.pub terraform@<PVE_SERVER_ADDRESS>

Alternatively, you can manually append the public key value into /home/terraform/.ssh/authorized_keys:

# Get the public key from your local machine
user@desktop:~$ cat ~/.ssh/terraform_id_ed25519.pub
# SSH into Proxmox
user@desktop:~$ ssh root@<PVE_SERVER_ADDRESS>

# Use your favorite editor to edit the file
root@pve:~# vim /home/terraform/.ssh/authorized_keys

# Paste the public key value into the file and save it

Confirm the key was added by SSHing into the server:

ssh -i ~/.ssh/terraform_id_ed25519 terraform@<PVE_SERVER_ADDRESS>

API Access #

Next, let’s create a Terraform user, group and token to access the Proxmox API. You’ll need to permit access to the root path, /, of the Proxmox server, as the BPG file download resource will need to gather file metadata information. On your Proxmox server, run the following as a privileged user:

# create role in PVE 8
pveum role add TerraformUser -privs "Datastore.Allocate \
  Datastore.AllocateSpace Datastore.AllocateTemplate \
  Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify \
  SDN.Use VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \
  VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType \
  VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate \
  VM.Monitor VM.PowerMgmt User.Modify"

# create group
pveum group add terraform-users

# add permissions
pveum acl modify / -group terraform-users -role TerraformUser

# create user 'terraform'
pveum useradd terraform@pam -groups terraform-users

# generate a token
pveum user token add terraform@pam token -privsep 0

The last command will output a token value similar to the following:

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ terraform@pam!token                  │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ 782a7700-4010-4802-8f4d-820f1b226850 │
└──────────────┴──────────────────────────────────────┘

Terraform Configuration #

For this post, we will create three files: main.tf, variables.tf, and proxmox.tfvars. The first two are used to define the template while the third is used for variables that are specific to your Proxmox server. The code examples below will follow a similar pattern to prior post, in which we’ll hard-code several of the attributes and use variables to add flexibility to the configuration and for sensitive values we do not want exposed in logs.

main.tf #

Now that we have a working SSH key pair and API token, we can configure the BPG provider to use them. Notably, the provider block contains an ssh block that defines the SSH user, username, and key file path, private_key. Additionally, we’ll concatenate the API token name and value in the api_token attribute.

There are three resource blocks in the main file that will: download a image; create a cloud-init vendor data file; and create a VM template. The first two blocks are pretty self-explanatory, however, they both contain a lifecycle block that will prevent the resources from being destroyed when calling terraform destroy. The motivation for this is two-fold: 1) To keep the image around for future templates and avoid excessive downloads; and 2) The vendor file is used by all VMs cloned from the template, removing the file will prevent the VMs from booting.

The third vm_template resource block is a bit more complex. First, it establishes a dependency on the first image block, in which the cloud image must be downloaded before creating the template, using the depends_on meta-argument. Next, given the Proxmox workflow of creating a VM then converting the VM to a template, we need to ensure that the VM is not booted to avoid: running cloud-init; creating machine IDs; and generating host SSH keys. Thus, setting started and template are critical to building a clean template. We’ll use the same dynamic block trick from the previous post, to create an EFI disk whenever the bios is set to ovmf. Additionally, the downloaded image is mounted to the VM’s root disk using the file_id attribute within the disk block - feel free to change the additional boot device settings to match your PVE storage configuration. Lastly, it will create a cloud-init configuration drive using the initialization block and pull in vendor data file configuration with vendor_data_file_id.

# main.tf
terraform {
  required_version = ">=1.5.0"
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = ">=0.53.1"
    }
  }
}

provider "proxmox" {
  endpoint  = var.pve_api_url
  api_token = "${var.pve_token_id}=${var.pve_token_secret}"
  insecure  = false
  ssh {
    agent       = true
    username    = var.pve_user
    private_key = var.pve_ssh_key_private
  }
}

# Download a cloud image using BPG provider
resource "proxmox_virtual_environment_download_file" "image" {
  node_name          = var.node
  content_type       = "iso"
  datastore_id       = "local"
  file_name          = var.image_filename
  url                = var.image_url
  checksum           = var.image_checksum
  checksum_algorithm = var.image_checksum_algorithm
  overwrite          = false

  lifecycle {
    prevent_destroy = true
  }
}

# Create a custom cloud-init config using BPG provider
resource "proxmox_virtual_environment_file" "vendor_data" {
  node_name    = var.node
  datastore_id = "local"
  content_type = "snippets"

  source_raw {
    file_name = "vendor-data.yaml"
    data      = <<-EOF
      #cloud-config
      packages:
        - qemu-guest-agent
      package_update: true
      power_state:
        mode: reboot
        timeout: 30
      EOF
  }

  lifecycle {
    prevent_destroy = true
  }
}

# Create a VM template
resource "proxmox_virtual_environment_vm" "vm_template" {
  depends_on = [proxmox_virtual_environment_download_file.image]

  node_name = var.node
  vm_id     = var.vm_id
  name      = var.vm_name
  bios      = var.bios
  machine   = "q35"
  started   = false # Don't boot the VM
  template  = true  # Turn the VM into a template

  agent {
    enabled = true
  }

  cpu {
    cores = 1
    type  = "host"
  }

  memory {
    dedicated = 1024
    floating  = 1024
  }

  # create an EFI disk when the bios is set to ovmf
  dynamic "efi_disk" {
    for_each = (var.bios == "ovmf" ? [1] : [])
    content {
      datastore_id      = "local-lvm"
      file_format       = "raw"
      type              = "4m"
      pre_enrolled_keys = true
    }
  }

  disk {
    file_id      = proxmox_virtual_environment_download_file.image.id
    datastore_id = "local-lvm"
    interface    = "scsi0"
    size         = 8
    file_format  = "raw"
    cache        = "writeback"
    iothread     = false
    ssd          = true
    discard      = "on"
  }

  network_device {
    bridge  = "vmbr0"
    vlan_id = "1"
  }

  # cloud-init config
  initialization {
    interface           = "ide2"
    type                = "nocloud"
    vendor_data_file_id = "local:snippets/vendor-data.yaml"

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}

variables.tf #

Now, create a variables.tf file and define the variables that will be used throughout the configuration. The one notable variable is setting the default value for image_name to null. This default setting will extract the image name from the image_url, but allow the user to override this value to convert image extension (next section).

# variables.tf
## Provider Login Variables
variable "pve_token_id" {
  description = "Proxmox API Token Name."
  sensitive   = true
}

variable "pve_token_secret" {
  description = "Proxmox API Token Value."
  sensitive   = true
}

variable "pve_api_url" {
  description = "Proxmox API Endpoint, e.g. 'https://pve.example.com/api2/json'"
  type        = string
  sensitive   = true
  validation {
    condition     = can(regex("(?i)^http[s]?://.*/api2/json$", var.pve_api_url))
    error_message = "Proxmox API Endpoint Invalid. Check URL - Scheme and Path required."
  }
}

## Proxmox SSH Variables
variable "pve_user" {
  description = "Proxmox username"
  type        = string
  sensitive   = true
}

variable "pve_ssh_key_private" {
  description = "File path to private SSH key for PVE - overrides 'pve_password'"
  type        = string
  sensitive   = true
  default     = null
}

## Common Variables
variable "node" {
  description = "Name of Proxmox node to download image on, e.g. `pve`."
  type        = string
}

## Image Variables
variable "image_filename" {
  description = "Filename, default `null` will extract name from URL."
  type        = string
  default     = null
}

variable "image_url" {
  description = "Image URL."
  type        = string
}

variable "image_checksum" {
  description = "Image checksum value."
  type        = string
}

variable "image_checksum_algorithm" {
  description = "Image checksum algorithm."
  type        = string
  default     = "sha256"
}

## VM Variables
variable "vm_id" {
  description = "ID number for new VM."
  type        = number
}

variable "vm_name" {
  description = "Name, must be alphanumeric (may contain dash: `-`). Defaults to PVE naming, `VM <VM_ID>`."
  type        = string
  default     = null
}

variable "bios" {
  description = "VM bios, setting to `ovmf` will automatically create a EFI disk."
  type        = string
  default     = "seabios"
  validation {
    condition     = contains(["seabios", "ovmf"], var.bios)
    error_message = "Invalid bios setting: ${var.bios}. Valid options: 'seabios' or 'ovmf'."
  }
}

proxmox.tfvars #

Finally, to make use of the above variables and reduce the number of variables to pass via the command line, create a proxmox.tfvars file. Set the pve_api_url to your PVE server IP address or FQDN and pve_ssh_key_private to the file path of your terraform SSH key.

Several linux distributions provide cloud images with the *.qcow2 file extensions, however, the Proxmox GUI will mask all files not ending in *.img. To avoid this, you must convert the *.qcow2 file to *.img, this is as simple as defining the image_filename variable. In the example below, we are downloading Debian 12 and creating a file named debian-12-generic-amd64.img. If you were to examine the downloaded file using qemu-img info, you will see that the file format is still qcow2, however, Proxmox will automatically convert the file when creating a template - for more information check out this prior post.

For the SHASUM_VALUE, pull the latest debian-12-generic-amd64.qcow2 hash from the Debian 12 release page.

# proxmox.tfvars
## Node Variables
node                = "pve"
pve_api_url         = "https://pve.example.com/api2/json"
pve_user            = "terraform"
pve_ssh_key_private = "~/.ssh/terraform_id_ed25519"

## Image Variables
image_filename           = "debian-12-generic-amd64.img"
image_url                = "https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2"
image_checksum           = "<SHASUM_VALUE>"
image_checksum_algorithm = "sha512"

## VM Variables
vm_id   = 9012
vm_name = "debian12"
bios    = "seabios"

Build the Template #

Now, we are ready to build the template. Set your PVE token values using environment variables and run terraform apply:

# use environment variables to pass the token id and secret
export TF_VAR_pve_token_id="terraform@pam!token"
export TF_VAR_pve_token_secret="782a7700-4010-4802-8f4d-820f1b226850"

# initialize the provider
terraform init

# create a terraform plan & apply it
terraform plan -var-file proxmox.tfvars -out tfplan
terraform apply tfplan

Because we added the lifecycle block and prevented the image and vendor file from being destroyed, we’ll need to pass the -target flag to remove the template:

# destroy the template
terraform destroy -var-file proxmox.tfvars -target='proxmox_virtual_environment_vm.vm_template'

Recap #

Overall, we’ve created a Terraform configuration that downloads a cloud image from a remote URL onto the Proxmox server; creates a custom cloud-init vendor data file within the Proxmox snippets storage; and generates a template that uses the cloud-init vendor data to configure all virtual machine clones. You can easily use this configuration to create additional Fedora and Ubuntu templates. With Ubuntu, you can skip setting image_filename as all cloud images are provided as img files.

If you want to skip creating your own module, checkout the companion repository on Github (link).

Troubleshooting #

Error Downloading Image #

│ Error: Error initiating file download
│
│   with proxmox_virtual_environment_download_file.image,
│   on main.tf line 24, in resource "proxmox_virtual_environment_download_file" "image":
│   24: resource "proxmox_virtual_environment_download_file" "image" {
│
│ Could not get file metadata, unexpected error: error fetching metadata from download url,
│ unexpected error in GetQueryURLMetadata: error retrieving URL metadata for 0xc000136c38:
│ received an HTTP 403 response - Reason: Permission check failed

This is due to a permission issue. To fix this, you’ll need to permit the Terraform group, terraform-users, access to the root path, /.

Error Creating Custom Disk #

│ Error: creating custom disk: unable to authenticate user "" over SSH to "192.168.x.x:22". Please verify that
ssh-agent is correctly loaded with an authorized key via 'ssh-add -L' (NOTE: configurations in ~/.ssh/config are not
considered by the provider): failed to dial 192.168.x.x:22: ssh: handshake failed: ssh: unable to authenticate,
attempted methods [none password], no supported methods remain

This is due to failed SSH access. To fix this, you’ll need to ensure that you are using the terraform user and the SSH key is added to the terraform user’s authorized keys.

References #

Cloud Images:

Cloud-init and Proxmox:

Terraform: