Provisioning Proxmox VMs with Terraform

Intro #

In this tutorial, we’ll explore using Terraform to provision virtual machines on Proxmox. We’ll cover granting Terraform limited access to the Proxmox API, then create a reusable module for the Telmate Proxmox provider plugin and use it to create a three node K3s cluster. If you are familiar with Terraform, feel free to skip the background section and jump straight into creating the module.

Terraform #

Terraform is an open-source infrastructure as code (IaC) tool that allows you to define, provision, and manage infrastructure resources across various cloud platforms and on-premises environments. It uses a declarative configuration language, HashiCorp Configuration Language (HCL), to define the desired state of your infrastructure. One of the key differences to other IaC tools is that Terraform stores infrastructure state in a single file, terraform.tfstate, allowing you to use version control systems to track changes, revert to previous states, and maintain synchronization of environments between end-users.

Terraform consist of two parts: core and plugins. Core is the main program, written in Go, that is responsible for reading configuration files and modules; creating resource graphs and plans; managing state; and communicating with plugins via remote procedure calls. While, plugins are Go binaries that extend the functionality of the core tool by adding new providers and provisioners. You can find plugins for all the major cloud-providers and hypervisors in the Terraform Registry. There are several plugins for Proxmox, however, we are going to focus on using Telmate/Terraform-Provider-Proxmox for this tutorial.

State Storage #

Terraform state is a crucial component of infrastructure management, as Terraform uses it to determine the changes required to reach the desired state. The state file should be stored using a secure remote backend, as it may contain sensitive data, such as database credentials. Additionally, consider using one that supports state locking, to ensure that only one user can modify the state at any given time.

For this post, we’ll use the default local backend, keeping the state file on the local filesystem and focus on provisioning. However, this is not recommended for production environments as it uses JSON format and stores the majority of information in plain text. If you interested in storing state securely and locally, see the post: Storing Terraform State in MinIO and PostgreSQL.

Terraform Modules #

A module is simply a collection of Terraform configuration files in a single directory. Now taking this idea further, it allows you to organize your configuration files into logical groupings to: 1) increase code-reuse and reduce duplication across your codebase; 2) encapsulate complex infrastructure configurations; and 3) share configurations across team members and the broader community.

For this post, we’ll create a complex folder structure that mimics one you might see in a production environment, but rather than creating folders for different environments (e.g., dev, staging, prod), we’ll just use a single homelab/ folder to store our K3s cluster configuration. Moreover, we’ll use the standard structure for creating local modules and include a examples/ folder along with the modules/ folder.

terraform-telmate-proxmox/
├── examples
│   ├── lxc
│   └── vm
│       ├── main.tf
│       └── variables.tf
├── homelab
│   └── k3s-cluster
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── modules
│   ├── lxc
│   └── vm
│       ├── main.tf
│       ├── outputs.tf
│       ├── README.md
│       └── variables.tf

Now before we get into writing the module, it’s important to cover what a module should accomplish. General advice is that modules should increase the abstraction of your infrastructure configuration, rather than just a simple wrapper for a single resource. Typically example of this might be a module that creates a virtual network consisting of multiple resources, for example: firewall, load balancer, and several subnets.

In our case we’ll create a module that slightly goes against this convention, as the majority of Proxmox workflows consist of creating a VM or LXC container and the provider only exposes these two resources. Yet, we’ll still encapsulate the complexity of creating a virtual machine and redefine several of the default values from the provider to suit our infrastructure requirements. Additionally, it will handle the conditional addition of several attributes with complex values depending on the value of select variables.

Grant Terraform Access to Proxmox #

Telmate plugin works by making API request to the Proxmox Virtual Environment (PVE) endpoint. For this tutorial we’ll use the Proxmox CLI to create a new role, group with permissions, and user; then generate an access token for the terraform user. We’ll also only allow permissions for specific paths in Proxmox, /storage and /vms. For additional configuration options see Proxmox Wiki: User Management or pveum help.

From the PVE CLI enter the following commands:

# create role
pveum role add TerraformUser -privs "Datastore.AllocateSpace \
  Datastore.Audit VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \
  VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk \
  VM.Config.HWType VM.Config.Memory VM.Config.Network \
  VM.Config.Options VM.Migrate VM.Monitor VM.PowerMgmt"

# create group
pveum group add terraform-users

# add permissions
pveum acl modify /storage -group terraform-users -role TerraformUser

pveum acl modify /vms -group terraform-users -role TerraformUser

# create user 'terraform'
pveum useradd terraform@pve -groups terraform-users

# generate a token
pveum user token add terraform@pve token -privsep 0

The last command will output a token value similar to the following, which we will use to validate Terraform with Proxmox during the final provisioning step.

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ terraform@pve!token                  │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ 782a7700-4010-4802-8f4d-820f1b226850 │
└──────────────┴──────────────────────────────────────┘

Creating a Terraform Module #

In this tutorial, we will create a single module, vm. The following sections discuss the logic behind each file and Terraform code using snippets with complete code examples available at the GitHub repo.

Main File #

We’ll start by creating the main.tf file, which is the entry point for our module. Here we will define the required provider plugins and minimal Terraform version in the terraform block. We’ll also create a single resource block that uses the resource type proxmox_vm_qemu from the Telmate provider to create a new virtual machine.

Within this resource we’ll set several attributes including three required settings: target_node, clone, and vmid. The target_node and clone attributes take string values that reference the Proxmox node and template name to clone from, respectively. The vmid attribute takes a number value which is the ID of the virtual machine we want to create. We’ll set all attributes using variables that we will define in the next section.

Each VM will have at minimum a single disk, but we’ll also want to provide the flexibility to add additional disk. The provider allows us to do this by defining multiple disk block within our resource block. To avoid manually duplicating a block of code for each disk, we’ll use a dynamic block to create multiple disk blocks based on the number of disks requested by the user. Within the dynamic block, it iterates through a disks variable that we’ll define later, however, this is a list of objects that each contain values to populate the disk attributes within the content block.

Lastly, we’ll use several ternary conditional expressions to set cloud-init attributes depending upon whether the user provides an SSH key or sets a static IP address. The syntax for this type of expression is if CONDITION ? TRUE_VALUE : FALSE_VALUE. For example with sshkeys, if the variable ci_ssh_key is set, then use the built-in function file() to pass the contents as a string, else set the value to null.

# modules/vm/main.tf
terraform {
  required_version = ">=1.3.0"
  required_providers {
    proxmox = {
      source  = "Telmate/proxmox"
      version = "~> 2.9.0"
    }
  }
}

resource "proxmox_vm_qemu" "vm" {
  target_node = var.node
  vmid        = var.vm_id
  ...
  clone       = var.template_name
  ...

  dynamic "disk" {
    for_each = var.disks
    content {
      type     = disk.value.disk_interface
      slot     = disk.value.disk_slot
      storage  = disk.value.disk_storage
      size     = disk.value.disk_size
      ...
    }
  }

  ...

  # cloud-init config
  ciuser       = var.ci_user
  sshkeys      = (var.ci_ssh_key != null ? file("${var.ci_ssh_key}") : null)
  ...
  ipconfig0    = (var.ci_ipv4_cidr != null ? "ip=${var.ci_ipv4_cidr},gw=${var.ci_ipv4_gateway}" : "ip=dhcp")
  ...
}

For a list of all available attributes, see Telmate documentation.

Variables File #

HCL variables are used to define inputs that will be passed into the module and that you would like to: 1) set default values for; 2) mask sensitive values from being displayed in logs or output; 3) enforce type constraints on; and/or 4) validate values. These inputs can be either required or optional, depending on if a default value is defined in the variable block. Terraform will also enforce that all required variables are provided when running terraform <plan/apply/destroy>, and met the type constraints requirements and pass any custom validation rules. In the example below, node and vm_id are required variables, while bios is optional. Additionally, bios inputs are limited to either seabios or ovmf, otherwise an error will be raised.

Terraform allows you to set their values in a number of ways, including using the Terraform CLI, environmental variables, and configuration files (*.tfvars). We’ll discuss these methods in more detail later.

# module/vm/variables.tf
variable "node" {
  description = "Name of Proxmox node to provision VM on, e.g. 'pve'."
  type        = string
}

variable "vm_id" {
  description = "ID number for new VM."
  type        = number
}
...

variable "bios" {
  description = "VM bios, setting to 'ovmf' will automatically create a EFI disk."
  type        = string
  default     = "seabios"
  validation {
    condition     = contains(["seabios", "ovmf"], var.bios)
    error_message = "Invalid bios setting. Valid options: 'seabios' or 'ovmf'."
  }
}
...

Above in the main.tf file, we created a dynamic block for the disk block. Now we need to create a corresponding disks variable that will be a list of objects, list(object()). Terraform objects are simply a map of key-value pairs, in which you can specify type constraints for each key attribute including: type, default value, and optionality. We can also define a default object for the variable that will be used if no disk are specified by the user. In the example below, the default disk object is one that is 8GB in size and in the slot, scsi0, which matches the boot disk drive in the Proxmox template that we’ll create later.

# module/vm/variables.tf
...

variable "disks" {
  description = "Terraform object with disk configurations."
  type = list(object({
    disk_interface = optional(string, "scsi")
    disk_slot      = optional(number, 0)
    disk_storage   = optional(string, "local-lvm")
    disk_size      = optional(string, "8G")
    disk_format    = optional(string, "raw")
    disk_cache     = optional(string, "writeback")
    disk_backup    = optional(number, 0)
    disk_iothread  = optional(number, 0)
    disk_ssd       = optional(number, 1)
    disk_discard   = optional(string, "on")
    }
  ))
  default = [{
    disk_interface = "scsi"
    disk_slot      = 0
    disk_storage   = "local-lvm"
    disk_size      = "8G"
    disk_format    = "raw"
    disk_cache     = "writeback"
    disk_backup    = 0
    disk_iothread  = 0
    disk_ssd       = 1
    disk_discard   = "on"
  }]
}
...

Output File #

In the last module file, we define two output variables that will be used to retrieve the instance ID and IPv4 address of the virtual machine. The value attribute references the provider resource, proxmox_vm_qemu; the simple but unique identifier for the resource, vm; and an expression that is exposed by the Telmate provider, e.g. id.

# module/vm/outputs.tf
output "id" {
  description = "Instance VM ID"
  value       = proxmox_vm_qemu.vm.id
}

output "public_ipv4" {
  description = "Instance Public IPv4 Address"
  value       = proxmox_vm_qemu.vm.default_ipv4_address
}

Using the Module: Building a K3s Cluster #

Now that we have a module that creates a virtual machine, let’s use it to create three VMs and use Terraform to create a K3s cluster.

Configuring the Provider #

We’ll start by adding the provider configuration to homelab/k3s-cluster/main.tf. We’ll use the same Terraform block configuration as we did in the vm module, but this time we’ll add a provider block that defines access to the Proxmox API.

# homelab/k3s-cluster/main.tf
terraform {
  required_providers {
    proxmox = {
      source  = "Telmate/proxmox"
      version = "~> 2.9.0"
    }
  }
}

provider "proxmox" {
  pm_api_url          = var.pve_api_url
  pm_api_token_id     = var.pve_token_id
  pm_api_token_secret = var.pve_token_secret
}

...

Let’s create a few variables to mask sensitive API token values from Terraform logs, the homelab/k3s-cluster/variables file contains the following:

# homelab/k3s-cluster/variables.tf
variable "pve_token_id" {
  sensitive = true
}

variable "pve_token_secret" {
  sensitive = true
}

variable "pve_api_url" {
  description = "Proxmox API Endpoint, e.g. 'https://pve.example.com/api2/json'"
  type        = string
  sensitive   = true
  validation {
    condition     = can(regex("(?i)^http[s]?://.*/api2/json$", var.pve_api_url))
    error_message = "Proxmox API Endpoint Invalid. Check URL - Scheme and Path required."
  }
}

Using the Module #

In order to use the vm module, we’ll create a module block within the homelab/k3s-cluster/main.tf file and set the source to the modules/vm/ folder. To create multiple VM instances, we’ll use the for_each argument and pass a map of the VM instances we want to the child vm module. Specifically, we’ll only set values that are unique for a given instance, and set all common values in the root k3s_cluster module.

# homelab/k3s-cluster/main.tf
...
module "k3s_cluster" {
  source = "../../modules/vm"

  for_each = tomap({
    "k3s-controller" = {
      id           = 210
      ipv4_cidr    = "192.168.1.210/24"
      ipv4_gateway = "192.168.1.1"
    },
    "k3s-node1" = {
      id           = 211
      ipv4_cidr    = null # Use DHCP
      ipv4_gateway = null # Use DHCP
    },
    "k3s-node2" = {
      id           = 212
      ipv4_cidr    = null # Use DHCP
      ipv4_gateway = null # Use DHCP
    },
  })

  node            = "pve"                                    # required
  vm_id           = each.value.id                            # required
  vm_name         = each.key                                 # optional
  template_name   = "ubuntu20"                               # required
  vcpu            = 2                                        # optional
  memory          = 4096                                     # optional
  ci_custom_data  = "vendor=local:snippets/vendor-data.yaml" # optional
  ci_ssh_key      = "~/.ssh/id_ed25519.pub"                  # optional, add SSH key to 'default' user
  ci_ipv4_cidr    = each.value.ipv4_cidr                     # optional
  ci_ipv4_gateway = each.value.ipv4_gateway                  # optional
}

...

Using Terraform to Run a Shell Command #

Lastly, we can use Terraform to generate a random string that we’ll use as the K3s join token and to run the K3s install script. To pass the IP address of our K3s control node to the install script, we’ll create a local variable, controller_ip, that captures the module output public_ipv4 for the control node.

# homelab/k3s-cluster/main.tf
...
locals {
  controller_ip = module.k3s_cluster["k3s-controller"].public_ipv4
  agent_ips     = { for k, v in module.k3s_cluster : k => v.public_ipv4 if k != "k3s-controller" }
}

...

resource "random_string" "k3s_token" {
  length  = 64
  special = false
}

...

resource "null_resource" "agents" {
  for_each = local.agent_ips

  connection {
    host = each.value
    user = "ubuntu"
  }

  provisioner "remote-exec" {
    inline = [
      "curl -sfL https://get.k3s.io | K3S_URL=https://${local.controller_ip}:6443 K3S_TOKEN=${random_string.k3s_token.result} sh -"
    ]
  }
}

Now, let’s bring it all together and deploy our K3s cluster using Terraform.

Provisioning using Terraform #

Proxmox Template: Ubuntu 20.04 #

Let’s create a simple Ubuntu 20.04 template in Proxmox to use as the base OS for our cluster nodes. If you already have a template available, change the clone value to the name of your template; otherwise, SSH into your Proxmox node and run the following commands as a privileged user:

# change into the ISO directory
cd /var/lib/vz/template/iso

# download the image
wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img

# create a virtual machine
qm create 9000 --name ubuntu20 --machine q35 --scsihw virtio-scsi-pci
# import the ubuntu image as the root disk
qm disk import 9000 ubuntu-20.04-server-cloudimg-amd64.img local-lvm
# attach and set the root disk as boot drive
qm set 9000 --boot order=scsi0 --scsi0 "local-lvm:vm-9000-disk-0"
# add cloud-init config drive
qm set 9000 --ide2 local-lvm:cloudinit --citype nocloud
# convert the vm to a template
qm template 9000

For more details on creating Proxmox templates, see Proxmox: Cloud-Init Support or checkout the post Golden Images and Proxmox Templates Using cloud-init.

Provision #

Finally, we will use Terraform to provision the three virtual machines and run the K3s scripts. We will define the API variables in-line using the -var flag. If you created a token using the directions above, the pve_api_token_id value is terraform@pve!token and the pve_api_token_secret value is similar to the output from above, e.g. 782a7700-4010-4802-8f4d-820f1b226850. Your pve_api_url value may be something similar to https://pve.example.com/api2/json if your node is behind a proxy, otherwise you can use the IP address, e.g. http://192.168.1.100/api2/json.

# create a terraform plan
terraform plan \
  -var='pve_api_url=https://pve.example.com/api2/json' \
  -var='pve_api_token_id=TOKEN' \
  -var='pve_api_token_secret=SECRET' \
  -out tfplan

# build the cluster
terraform apply tfplan

Test #

Once provisioning is complete connect to the controller node and view the cluster using kubectl.

# connect to the controller node
user@desktop:~$ ssh ubuntu@$(terraform output -raw controller_ip)

# list the active nodes
ubuntu@k3s-controller:~$ sudo kubectl get nodes
NAME            STATUS   ROLES                  AGE   VERSION
k3s-controller   Ready    control-plane,master   51s   v1.23.3+k3s1
k3s-node1        Ready    <none>                 44s   v1.23.3+k3s1
k3s-node2        Ready    <none>                 41s   v1.23.3+k3s1

Remove #

Once you are finished with the cluster you can tear it down using terraform destroy.

# destroy the cluster
terraform destroy \
  -var='pve_api_url=https://pve.example.com/api2/json' \
  -var='pve_api_token_id=TOKEN' \
  -var='pve_api_token_secret=SECRET'

Recap #

Terraform is a powerful IaC tool that simplifies infrastructure management, promotes collaboration, and enhances efficiency. Here, we created a module to simplify provisioning virtual machines on Proxmox VE and used it to create three nodes for our K3s cluster. We also used Terraform’s built-in resources to run the shell commands to create the cluster and generate a K3s join token. Furthermore, because we used Terraform, the k3s_token is stored in the terraform.tfstate file, which allows you to add additional nodes to the cluster without destroying the cluster and rebuilding it - test this out by adding an additional VM instance and re-running terraform apply. Additionally, because Terraform captures the IP addresses of the VMs we created, setting a static IP address for the control node is not necessary and works with DHCP.

Hopefully, you’ve found this tutorial useful, feel free to use it and the companion repo as a reference for your own infrastructure. Thanks for reading!

References #

• Github: trfore/terraform-telmate-proxmox

Terraform:

• Terraform
• Terraform Documentation
• HCL language
  • HCL types
• Github: Hashicorp/Terraform
  • Github: Example Terraform .gitignore
• HashiCorp Learn: Modules
• Hashicorp Docs: Locals
• HashiCorp Learn: Modules - Overview

Terraform Providers / Plugins:

• Telmate/Terraform-Provider-Proxmox
• Telmate Documentation

Other Resources:

• Terraform: Up & Running