[{"content":"","date":null,"permalink":"https://www.trfore.com/tags/cloud-init/","section":"Tags","summary":"","title":"Cloud-Init"},{"content":"","date":null,"permalink":"https://www.trfore.com/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/proxmox/","section":"Tags","summary":"","title":"Proxmox"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/proxmox-template/","section":"Tags","summary":"","title":"Proxmox-Template"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/terraform/","section":"Tags","summary":"","title":"Terraform"},{"content":"","date":null,"permalink":"https://www.trfore.com/","section":"trfore","summary":"","title":"trfore"},{"content":"Intro #Last post we used the BPG Proxmox provider to provision VMs using a pre-existing template. This post we\u0026rsquo;ll explore using the provider to create a Proxmox template, from downloading the image to building and customizing the template. You’ll need to have Terraform installed on your local machine and SSH access to a Proxmox VE (PVE) server with a privileged user account.\nBPG module is available on GitHub (link) Grant Terraform Access to Proxmox #In order for Terraform to be able to create templates in Proxmox, it will need to have SSH access to the PVE server. Unlike prior post, we\u0026rsquo;ll create a new linux user, terraform, on the PVE host server and add that user within the Proxmox UI under the PAM realm.\nIf you already created a user under the PVE realm, terraform@pve, I suggest removing that user and permissions to avoid confusion:\n# delete user and token pveum user delete terraform@pve For additional information, see pveum help user or Proxmox Docs: User Authentication.\nCreate User on PVE Server #First, start by logging into the PVE server as a privileged user. Then create a new linux user and password:\n# SSH into the PVE server ssh root@\u0026lt;PVE_SERVER_ADDRESS\u0026gt; # create user \u0026#39;terraform\u0026#39; adduser --home /home/terraform --shell /bin/bash terraform # add user to sudoers usermod -aG sudo terraform Next, give the terraform user permission to use PVE commands without a password. We\u0026rsquo;ll use the commands from the provider documentation on \u0026lsquo;SSH User\u0026rsquo;:\n# create a sudoers file for terraform user visudo -f /etc/sudoers.d/terraform Add the following line at the end of the file and save it:\nterraform ALL=(root) NOPASSWD: /sbin/pvesm terraform ALL=(root) NOPASSWD: /sbin/qm terraform ALL=(root) NOPASSWD: /usr/bin/tee /var/lib/vz/* SSH Access #Now let\u0026rsquo;s create a dedicated SSH key and add it to the /home/terraform/.ssh/authorized_keys file on the PVE server. Start by creating a new key on your local desktop using ssh-keygen:\n# create a new ssh key pair ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/terraform_id_ed25519 -C \u0026#34;USER_EMAIL\u0026#34; # example output user@desktop:~$ ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/terraform_id_ed25519 -C \u0026#34;USER_EMAIL\u0026#34; Generating public/private ed25519 key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/user/.ssh/terraform_id_ed25519 Your public key has been saved in /home/user/.ssh/terraform_id_ed25519.pub The key fingerprint is: SHA256:uimKIsvhgDs8J5B155+e4S1HSKl9XewoqsgE1SEVLNA USER_EMAIL The key\u0026#39;s randomart image is: +--[ED25519 256]--+ | .o.o+. | | Eo.. | | ... . . | | ... . o o | | o.. o +S. . + | |+ . o.o + o . | |=. . ...= . | |B*.+.. +=+. | |*==.o.+o+o. | +----[SHA256]-----+ # view the public key user@desktop:~$ cat ~/.ssh/terraform_id_ed25519.pub ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPQYTV18SN+39z9W99SzaJoc8VncoLyjhLulVH+pkkZ2 USER_EMAIL Now, add the key to the terraform user\u0026rsquo;s authorized keys file on the Proxmox server. You can do this by running ssh-copy-id from your local desktop:\nssh-copy-id -i ~/.ssh/terraform_id_ed25519.pub terraform@\u0026lt;PVE_SERVER_ADDRESS\u0026gt; Alternatively, you can manually append the public key value into /home/terraform/.ssh/authorized_keys:\n# Get the public key from your local machine user@desktop:~$ cat ~/.ssh/terraform_id_ed25519.pub # SSH into Proxmox user@desktop:~$ ssh root@\u0026lt;PVE_SERVER_ADDRESS\u0026gt; # Use your favorite editor to edit the file root@pve:~# vim /home/terraform/.ssh/authorized_keys # Paste the public key value into the file and save it Confirm the key was added by SSHing into the server:\nssh -i ~/.ssh/terraform_id_ed25519 terraform@\u0026lt;PVE_SERVER_ADDRESS\u0026gt; API Access #Next, let\u0026rsquo;s create a Terraform user, group and token to access the Proxmox API. You\u0026rsquo;ll need to permit access to the root path, /, of the Proxmox server, as the BPG file download resource will need to gather file metadata information. On your Proxmox server, run the following as a privileged user:\n# create role in PVE 8 pveum role add TerraformUser -privs \u0026#34;Datastore.Allocate \\ Datastore.AllocateSpace Datastore.AllocateTemplate \\ Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify \\ SDN.Use VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \\ VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType \\ VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate \\ VM.Monitor VM.PowerMgmt User.Modify\u0026#34; # create group pveum group add terraform-users # add permissions pveum acl modify / -group terraform-users -role TerraformUser # create user \u0026#39;terraform\u0026#39; pveum useradd terraform@pam -groups terraform-users # generate a token pveum user token add terraform@pam token -privsep 0 The last command will output a token value similar to the following:\n┌──────────────┬──────────────────────────────────────┐ │ key │ value │ ╞══════════════╪══════════════════════════════════════╡ │ full-tokenid │ terraform@pam!token │ ├──────────────┼──────────────────────────────────────┤ │ info │ {\u0026#34;privsep\u0026#34;:\u0026#34;0\u0026#34;} │ ├──────────────┼──────────────────────────────────────┤ │ value │ 782a7700-4010-4802-8f4d-820f1b226850 │ └──────────────┴──────────────────────────────────────┘ Terraform Configuration #For this post, we will create three files: main.tf, variables.tf, and proxmox.tfvars. The first two are used to define the template while the third is used for variables that are specific to your Proxmox server. The code examples below will follow a similar pattern to prior post, in which we\u0026rsquo;ll hard-code several of the attributes and use variables to add flexibility to the configuration and for sensitive values we do not want exposed in logs.\nmain.tf #Now that we have a working SSH key pair and API token, we can configure the BPG provider to use them. Notably, the provider block contains an ssh block that defines the SSH user, username, and key file path, private_key. Additionally, we\u0026rsquo;ll concatenate the API token name and value in the api_token attribute.\nThere are three resource blocks in the main file that will: download a image; create a cloud-init vendor data file; and create a VM template. The first two blocks are pretty self-explanatory, however, they both contain a lifecycle block that will prevent the resources from being destroyed when calling terraform destroy. The motivation for this is two-fold: 1) To keep the image around for future templates and avoid excessive downloads; and 2) The vendor file is used by all VMs cloned from the template, removing the file will prevent the VMs from booting.\nThe third vm_template resource block is a bit more complex. First, it establishes a dependency on the first image block, in which the cloud image must be downloaded before creating the template, using the depends_on meta-argument. Next, given the Proxmox workflow of creating a VM then converting the VM to a template, we need to ensure that the VM is not booted to avoid: running cloud-init; creating machine IDs; and generating host SSH keys. Thus, setting started and template are critical to building a clean template. We\u0026rsquo;ll use the same dynamic block trick from the previous post, to create an EFI disk whenever the bios is set to ovmf. Additionally, the downloaded image is mounted to the VM\u0026rsquo;s root disk using the file_id attribute within the disk block - feel free to change the additional boot device settings to match your PVE storage configuration. Lastly, it will create a cloud-init configuration drive using the initialization block and pull in vendor data file configuration with vendor_data_file_id.\n# main.tf terraform { required_version = \u0026#34;\u0026gt;=1.5.0\u0026#34; required_providers { proxmox = { source = \u0026#34;bpg/proxmox\u0026#34; version = \u0026#34;\u0026gt;=0.53.1\u0026#34; } } } provider \u0026#34;proxmox\u0026#34; { endpoint = var.pve_api_url api_token = \u0026#34;${var.pve_token_id}=${var.pve_token_secret}\u0026#34; insecure = false ssh { agent = true username = var.pve_user private_key = var.pve_ssh_key_private } } # Download a cloud image using BPG provider resource \u0026#34;proxmox_virtual_environment_download_file\u0026#34; \u0026#34;image\u0026#34; { node_name = var.node content_type = \u0026#34;iso\u0026#34; datastore_id = \u0026#34;local\u0026#34; file_name = var.image_filename url = var.image_url checksum = var.image_checksum checksum_algorithm = var.image_checksum_algorithm overwrite = false lifecycle { prevent_destroy = true } } # Create a custom cloud-init config using BPG provider resource \u0026#34;proxmox_virtual_environment_file\u0026#34; \u0026#34;vendor_data\u0026#34; { node_name = var.node datastore_id = \u0026#34;local\u0026#34; content_type = \u0026#34;snippets\u0026#34; source_raw { file_name = \u0026#34;vendor-data.yaml\u0026#34; data = \u0026lt;\u0026lt;-EOF #cloud-config packages: - qemu-guest-agent package_update: true power_state: mode: reboot timeout: 30 EOF } lifecycle { prevent_destroy = true } } # Create a VM template resource \u0026#34;proxmox_virtual_environment_vm\u0026#34; \u0026#34;vm_template\u0026#34; { depends_on = [proxmox_virtual_environment_download_file.image] node_name = var.node vm_id = var.vm_id name = var.vm_name bios = var.bios machine = \u0026#34;q35\u0026#34; started = false # Don\u0026#39;t boot the VM template = true # Turn the VM into a template agent { enabled = true } cpu { cores = 1 type = \u0026#34;host\u0026#34; } memory { dedicated = 1024 floating = 1024 } # create an EFI disk when the bios is set to ovmf dynamic \u0026#34;efi_disk\u0026#34; { for_each = (var.bios == \u0026#34;ovmf\u0026#34; ? [1] : []) content { datastore_id = \u0026#34;local-lvm\u0026#34; file_format = \u0026#34;raw\u0026#34; type = \u0026#34;4m\u0026#34; pre_enrolled_keys = true } } disk { file_id = proxmox_virtual_environment_download_file.image.id datastore_id = \u0026#34;local-lvm\u0026#34; interface = \u0026#34;scsi0\u0026#34; size = 8 file_format = \u0026#34;raw\u0026#34; cache = \u0026#34;writeback\u0026#34; iothread = false ssd = true discard = \u0026#34;on\u0026#34; } network_device { bridge = \u0026#34;vmbr0\u0026#34; vlan_id = \u0026#34;1\u0026#34; } # cloud-init config initialization { interface = \u0026#34;ide2\u0026#34; type = \u0026#34;nocloud\u0026#34; vendor_data_file_id = \u0026#34;local:snippets/vendor-data.yaml\u0026#34; ip_config { ipv4 { address = \u0026#34;dhcp\u0026#34; } } } } variables.tf #Now, create a variables.tf file and define the variables that will be used throughout the configuration. The one notable variable is setting the default value for image_name to null. This default setting will extract the image name from the image_url, but allow the user to override this value to convert image extension (next section).\n# variables.tf ## Provider Login Variables variable \u0026#34;pve_token_id\u0026#34; { description = \u0026#34;Proxmox API Token Name.\u0026#34; sensitive = true } variable \u0026#34;pve_token_secret\u0026#34; { description = \u0026#34;Proxmox API Token Value.\u0026#34; sensitive = true } variable \u0026#34;pve_api_url\u0026#34; { description = \u0026#34;Proxmox API Endpoint, e.g. \u0026#39;https://pve.example.com/api2/json\u0026#39;\u0026#34; type = string sensitive = true validation { condition = can(regex(\u0026#34;(?i)^http[s]?://.*/api2/json$\u0026#34;, var.pve_api_url)) error_message = \u0026#34;Proxmox API Endpoint Invalid. Check URL - Scheme and Path required.\u0026#34; } } ## Proxmox SSH Variables variable \u0026#34;pve_user\u0026#34; { description = \u0026#34;Proxmox username\u0026#34; type = string sensitive = true } variable \u0026#34;pve_ssh_key_private\u0026#34; { description = \u0026#34;File path to private SSH key for PVE - overrides \u0026#39;pve_password\u0026#39;\u0026#34; type = string sensitive = true default = null } ## Common Variables variable \u0026#34;node\u0026#34; { description = \u0026#34;Name of Proxmox node to download image on, e.g. `pve`.\u0026#34; type = string } ## Image Variables variable \u0026#34;image_filename\u0026#34; { description = \u0026#34;Filename, default `null` will extract name from URL.\u0026#34; type = string default = null } variable \u0026#34;image_url\u0026#34; { description = \u0026#34;Image URL.\u0026#34; type = string } variable \u0026#34;image_checksum\u0026#34; { description = \u0026#34;Image checksum value.\u0026#34; type = string } variable \u0026#34;image_checksum_algorithm\u0026#34; { description = \u0026#34;Image checksum algorithm.\u0026#34; type = string default = \u0026#34;sha256\u0026#34; } ## VM Variables variable \u0026#34;vm_id\u0026#34; { description = \u0026#34;ID number for new VM.\u0026#34; type = number } variable \u0026#34;vm_name\u0026#34; { description = \u0026#34;Name, must be alphanumeric (may contain dash: `-`). Defaults to PVE naming, `VM \u0026lt;VM_ID\u0026gt;`.\u0026#34; type = string default = null } variable \u0026#34;bios\u0026#34; { description = \u0026#34;VM bios, setting to `ovmf` will automatically create a EFI disk.\u0026#34; type = string default = \u0026#34;seabios\u0026#34; validation { condition = contains([\u0026#34;seabios\u0026#34;, \u0026#34;ovmf\u0026#34;], var.bios) error_message = \u0026#34;Invalid bios setting: ${var.bios}. Valid options: \u0026#39;seabios\u0026#39; or \u0026#39;ovmf\u0026#39;.\u0026#34; } } proxmox.tfvars #Finally, to make use of the above variables and reduce the number of variables to pass via the command line, create a proxmox.tfvars file. Set the pve_api_url to your PVE server IP address or FQDN and pve_ssh_key_private to the file path of your terraform SSH key.\nSeveral linux distributions provide cloud images with the *.qcow2 file extensions, however, the Proxmox GUI will mask all files not ending in *.img. To avoid this, you must convert the *.qcow2 file to *.img, this is as simple as defining the image_filename variable. In the example below, we are downloading Debian 12 and creating a file named debian-12-generic-amd64.img. If you were to examine the downloaded file using qemu-img info, you will see that the file format is still qcow2, however, Proxmox will automatically convert the file when creating a template - for more information check out this prior post.\nFor the SHASUM_VALUE, pull the latest debian-12-generic-amd64.qcow2 hash from the Debian 12 release page.\n# proxmox.tfvars ## Node Variables node = \u0026#34;pve\u0026#34; pve_api_url = \u0026#34;https://pve.example.com/api2/json\u0026#34; pve_user = \u0026#34;terraform\u0026#34; pve_ssh_key_private = \u0026#34;~/.ssh/terraform_id_ed25519\u0026#34; ## Image Variables image_filename = \u0026#34;debian-12-generic-amd64.img\u0026#34; image_url = \u0026#34;https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2\u0026#34; image_checksum = \u0026#34;\u0026lt;SHASUM_VALUE\u0026gt;\u0026#34; image_checksum_algorithm = \u0026#34;sha512\u0026#34; ## VM Variables vm_id = 9012 vm_name = \u0026#34;debian12\u0026#34; bios = \u0026#34;seabios\u0026#34; Build the Template #Now, we are ready to build the template. Set your PVE token values using environment variables and run terraform apply:\n# use environment variables to pass the token id and secret export TF_VAR_pve_token_id=\u0026#34;terraform@pam!token\u0026#34; export TF_VAR_pve_token_secret=\u0026#34;782a7700-4010-4802-8f4d-820f1b226850\u0026#34; # initialize the provider terraform init # create a terraform plan \u0026amp; apply it terraform plan -var-file proxmox.tfvars -out tfplan terraform apply tfplan Because we added the lifecycle block and prevented the image and vendor file from being destroyed, we\u0026rsquo;ll need to pass the -target flag to remove the template:\n# destroy the template terraform destroy -var-file proxmox.tfvars -target=\u0026#39;proxmox_virtual_environment_vm.vm_template\u0026#39; Recap #Overall, we\u0026rsquo;ve created a Terraform configuration that downloads a cloud image from a remote URL onto the Proxmox server; creates a custom cloud-init vendor data file within the Proxmox snippets storage; and generates a template that uses the cloud-init vendor data to configure all virtual machine clones. You can easily use this configuration to create additional Fedora and Ubuntu templates. With Ubuntu, you can skip setting image_filename as all cloud images are provided as img files.\nIf you want to skip creating your own module, checkout the companion repository on Github (link).\nTroubleshooting #Error Downloading Image #│ Error: Error initiating file download │ │ with proxmox_virtual_environment_download_file.image, │ on main.tf line 24, in resource \u0026#34;proxmox_virtual_environment_download_file\u0026#34; \u0026#34;image\u0026#34;: │ 24: resource \u0026#34;proxmox_virtual_environment_download_file\u0026#34; \u0026#34;image\u0026#34; { │ │ Could not get file metadata, unexpected error: error fetching metadata from download url, │ unexpected error in GetQueryURLMetadata: error retrieving URL metadata for 0xc000136c38: │ received an HTTP 403 response - Reason: Permission check failed This is due to a permission issue. To fix this, you\u0026rsquo;ll need to permit the Terraform group, terraform-users, access to the root path, /.\nError Creating Custom Disk #│ Error: creating custom disk: unable to authenticate user \u0026#34;\u0026#34; over SSH to \u0026#34;192.168.x.x:22\u0026#34;. Please verify that ssh-agent is correctly loaded with an authorized key via \u0026#39;ssh-add -L\u0026#39; (NOTE: configurations in ~/.ssh/config are not considered by the provider): failed to dial 192.168.x.x:22: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain This is due to failed SSH access. To fix this, you\u0026rsquo;ll need to ensure that you are using the terraform user and the SSH key is added to the terraform user\u0026rsquo;s authorized keys.\nReferences #Cloud Images:\nDebian Fedora Ubuntu Cloud-init and Proxmox:\nCloud-init Documentation Vendor Data Proxmox Proxmox Documentation Proxmox Docs: User Authentication Terraform:\nTerraform Terraform Documentation Sensitive Variables BPG Proxmox Provider BPG Authentication Requirements ","date":"May 13, 2024","permalink":"https://www.trfore.com/posts/using-terraform-to-create-proxmox-templates/","section":"Posts","summary":"Leverage Terraform to effortlessly download images and create custom Proxmox templates","title":"Using Terraform to Create Proxmox Templates"},{"content":"Intro #Since the post Provisioning Proxmox VMs with Terraform, I switched to using the BPG Proxmox provider. The schema is different but easily adaptable to the prior code base. This provider is feature pack and provisioning is faster than other providers. Two features I am excited for is the ability to:\nDownload LXC and VM cloud images directly onto the Proxmox server.\nBuild VM templates from cloud images using Terraform.\nI will be covering this provider across several post, but this one covers the basics using pre-existing templates.\nIn a prior post we created a Proxmox template that was configured using cloud-init vendor data file. This included several shell scripts for downloading cloud images and building templates. In this post, we\u0026rsquo;ll use those scripts to download and create an Ubuntu 24.04 template, then test out the BPG provider. You\u0026rsquo;ll need to have Terraform installed on your local machine and access to a Proxmox VE (PVE) server with the shell scripts installed.\nBPG module is available on GitHub (link) Create a Template #To start, create a minimal cloud-init vendor data file for the template on your Proxmox server. Note: For multi-node PVE clusters, the vendor data file and template must be available on each node.\n# SSH into Proxmox user@desktop:~$ ssh root@pve Create the vendor-data.yaml file in /var/lib/vz/snippets/ with the following content:\ncat \u0026lt;\u0026lt; EOF \u0026gt; /var/lib/vz/snippets/vendor-data.yaml #cloud-config packages: - qemu-guest-agent package_update: true power_state: mode: reboot timeout: 30 EOF Next, download the latest Ubuntu 24.04 release and create template using the shell scripts:\n# download/update Ubuntu 24.04 image-update -d ubuntu -r 24 # create a template build-template --id 9024 --name ubuntu24 --img /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img --bios ovmf This template will already have QEMU Guest Agent enabled, along with an EFI disk, cloud-init drive, and boot disk based on the Ubuntu 24.04 cloud image. Additionally, it will use the cloud-init vendor file we created earlier to install qemu-guest-agent, update all packages, then reboot the VM to ensure that Proxmox registers the guest agent. Because the cloud-init configuration file is imported as vendor data, you can still configure cloud-init user data using the Proxmox GUI - for details on why this works, see this blog post.\nGrant Terraform Access to Proxmox #Next, let\u0026rsquo;s create a Terraform user, group and token to access the Proxmox API. We\u0026rsquo;ll limit the group\u0026rsquo;s access to only the Proxmox resources it requires. On your Proxmox server, run the following as a privileged user:\n# create role in PVE 8 pveum role add TerraformUser -privs \u0026#34;Datastore.Allocate \\ Datastore.AllocateSpace Datastore.AllocateTemplate \\ Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify \\ SDN.Use VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \\ VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType \\ VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate \\ VM.Monitor VM.PowerMgmt User.Modify\u0026#34; # create group pveum group add terraform-users # add permissions pveum acl modify /storage -group terraform-users -role TerraformUser pveum acl modify /vms -group terraform-users -role TerraformUser pveum acl modify /sdn/zones -group terraform-users -role TerraformUser # create user \u0026#39;terraform\u0026#39; pveum useradd terraform@pve -groups terraform-users # generate a token pveum user token add terraform@pve token -privsep 0 The last command will output a token value similar to the following, save this information as we\u0026rsquo;ll pass it via environment variables to Terraform at the end.\n┌──────────────┬──────────────────────────────────────┐ │ key │ value │ ╞══════════════╪══════════════════════════════════════╡ │ full-tokenid │ terraform@pve!token │ ├──────────────┼──────────────────────────────────────┤ │ info │ {\u0026#34;privsep\u0026#34;:\u0026#34;0\u0026#34;} │ ├──────────────┼──────────────────────────────────────┤ │ value │ 782a7700-4010-4802-8f4d-820f1b226850 │ └──────────────┴──────────────────────────────────────┘ The Provider notes that root SSH access is required for several functions. However, for this post token access is sufficient. In the next post we\u0026rsquo;ll upload images and config files that will require root SSH access. BGP Provider #Next, let\u0026rsquo;s create the several Terraform files that will utilize the BGP provider. For simplicity, we\u0026rsquo;ll hard-code the majority of attributes, however, let\u0026rsquo;s create a few variables to mask sensitive information and provide more flexibility to the end-user. Start by creating a new folder called bgp-example in your working directory and add the following files:\nmain.tf #Within the main.tf file, create a terraform block and provider block that uses the BPG Proxmox provider. The provider block attributes for the PVE host address, endpoint, and the API token, api_token, are variables that we\u0026rsquo;ll define later in variable.tf. Additionally, the api_token is a concatenation of the PVE token ID and value, \u0026lt;TOKEN_ID\u0026gt;=\u0026lt;TOKEN_VALUE\u0026gt;, that we\u0026rsquo;ll want to avoid exposing in logs.\nNotably, this provider uses custom blocks, e.g. efi_disk, to configure many of the VM settings. To add flexibility to the Terraform configuration, we\u0026rsquo;ll conditionally use a dynamic block to create an EFI disk when the bios is set to ovmf; otherwise, a null list is passed to the for_each statement and a EFI disk is not created.\nThe cloud-init disk is configured using the initialization block, within this block we\u0026rsquo;ll keep the cloud-init vendor data file attached and set the datasource, type, to NoCloud. This will ensure that cloud-init pulls the configuration from the attached config drive. Lastly, in order to login into the VM using the default user, ubuntu, we\u0026rsquo;ll add a public ssh key to the cloud-init config, keys, and use DHCP to configure the network interface.\n# bgp-example/main.tf terraform { required_version = \u0026#34;\u0026gt;=1.5.0\u0026#34; required_providers { proxmox = { source = \u0026#34;bpg/proxmox\u0026#34; version = \u0026#34;\u0026gt;=0.53.1\u0026#34; } } } provider \u0026#34;proxmox\u0026#34; { endpoint = var.pve_api_url api_token = \u0026#34;${var.pve_token_id}=${var.pve_token_secret}\u0026#34; insecure = true } resource \u0026#34;proxmox_virtual_environment_vm\u0026#34; \u0026#34;vm\u0026#34; { node_name = \u0026#34;pve\u0026#34; vm_id = 100 name = \u0026#34;vm-example\u0026#34; description = \u0026#34;Managed by Terraform\u0026#34; tags = [\u0026#34;terraform\u0026#34;, \u0026#34;ubuntu\u0026#34;] bios = var.bios # clone from the Ubuntu 24.04 template we created earlier clone { vm_id = 9024 full = true } # keep the first disk as boot disk disk { datastore_id = \u0026#34;local-lvm\u0026#34; interface = \u0026#34;scsi0\u0026#34; size = 8 file_format = \u0026#34;raw\u0026#34; cache = \u0026#34;writeback\u0026#34; iothread = false ssd = true discard = \u0026#34;on\u0026#34; } # create an EFI disk when the bios is set to ovmf dynamic \u0026#34;efi_disk\u0026#34; { for_each = (var.bios == \u0026#34;ovmf\u0026#34; ? [1] : []) content { datastore_id = \u0026#34;local-lvm\u0026#34; file_format = \u0026#34;raw\u0026#34; type = \u0026#34;4m\u0026#34; pre_enrolled_keys = true } } network_device { bridge = \u0026#34;vmbr0\u0026#34; vlan_id = \u0026#34;1\u0026#34; } # cloud-init config initialization { interface = \u0026#34;ide2\u0026#34; type = \u0026#34;nocloud\u0026#34; vendor_data_file_id = \u0026#34;local:snippets/vendor-data.yaml\u0026#34; upgrade = true # add SSH key to cloud-init default user user_account { keys = [file(\u0026#34;${var.ci_ssh_key}\u0026#34;)] } ip_config { ipv4 { address = \u0026#34;dhcp\u0026#34; } } } # cloud-init SSH keys will cause a forced replacement, this is expected # behavior see https://github.com/bpg/terraform-provider-proxmox/issues/373 lifecycle { ignore_changes = [initialization[\u0026#34;user_account\u0026#34;], ] } } For a full list of available resources and their attributes, please refer to the BPG provider documentation.\noutput.tf #Given the network interface is using DHCP, let\u0026rsquo;s create an output.tf file with a single variable that will return the IPv4 address:\n# bgp-example/outputs.tf output \u0026#34;public_ipv4\u0026#34; { description = \u0026#34;Instance Public IPv4 Address\u0026#34; value = flatten(proxmox_virtual_environment_vm.vm.ipv4_addresses[1]) } variables.tf #Now, create a variables.tf file and define the variables that will be used throughout the configuration:\n# bgp-example/variables.tf ## Provider Login Variables variable \u0026#34;pve_token_id\u0026#34; { description = \u0026#34;Proxmox API Token Name.\u0026#34; sensitive = true } variable \u0026#34;pve_token_secret\u0026#34; { description = \u0026#34;Proxmox API Token Value.\u0026#34; sensitive = true } variable \u0026#34;pve_api_url\u0026#34; { description = \u0026#34;Proxmox API Endpoint, e.g. \u0026#39;https://pve.example.com/api2/json\u0026#39;\u0026#34; type = string sensitive = true validation { condition = can(regex(\u0026#34;(?i)^http[s]?://.*/api2/json$\u0026#34;, var.pve_api_url)) error_message = \u0026#34;Proxmox API Endpoint Invalid. Check URL - Scheme and Path required.\u0026#34; } } ## VM Variables variable \u0026#34;bios\u0026#34; { description = \u0026#34;VM bios, setting to `ovmf` will automatically create a EFI disk.\u0026#34; type = string default = \u0026#34;seabios\u0026#34; validation { condition = contains([\u0026#34;seabios\u0026#34;, \u0026#34;ovmf\u0026#34;], var.bios) error_message = \u0026#34;Invalid bios setting: ${var.bios}. Valid options: \u0026#39;seabios\u0026#39; or \u0026#39;ovmf\u0026#39;.\u0026#34; } } variable \u0026#34;ci_ssh_key\u0026#34; { description = \u0026#34;File path to SSH key for \u0026#39;default\u0026#39; user, e.g. `~/.ssh/id_ed25519.pub`.\u0026#34; type = string default = null } proxmox.tfvars #Finally, to make use of the above variables and reduce the number of variables to pass via the command line, create a proxmox.tfvars file and set the pve_api_url and ci_ssh_key to your PVE server IP address or FQDN and SSH file path:\n# bgp-example/proxmox.tfvars pve_api_url = \u0026#34;https://pve.example.com/api2/json\u0026#34; bios = \u0026#34;ovmf\u0026#34; ci_ssh_key = \u0026#34;~/.ssh/id_ed25519.pub\u0026#34; Run Terraform #Now let\u0026rsquo;s test out the configuration, Terraform allows you to set variables using: 1) environment variables, prefixing the variable name with TF_VAR_; 2) CLI args, using the -var= flag; and/or 3) variable files, having the extension *.tfvars and using the -var-file flag. The following commands will initialize and apply the terraform configuration:\n# use environment variables to pass the token id and secret export TF_VAR_pve_token_id=\u0026#34;terraform@pve!token\u0026#34; export TF_VAR_pve_token_secret=\u0026#34;782a7700-4010-4802-8f4d-820f1b226850\u0026#34; # initialize the provider terraform init # create a terraform plan \u0026amp; apply it terraform plan -var-file proxmox.tfvars -out tfplan terraform apply tfplan Once the plan is applied successfully, you should see output similar to this:\n... Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Outputs: public_ipv4 = [ \u0026#34;192.168.1.16\u0026#34;, ] Test out the VM by SSHing into it and running uname -a:\nuser@desktop:~$ ssh ubuntu@192.168.1.16 ubuntu@vm-example:~$ uname -a To cleanup the environment, run the following command:\n# remove the vm terraform destroy -var-file proxmox.tfvars Recap #So far we created a simple root module to deploy a VM from a template. This is a great starting point for creating more complex modules, especially given the BPG Proxmox provider consist of multiple resources that combine together to encapsulate complex Proxmox deployments. If you are interested I have a BPG module (link) that can: download VM and LXC images; create VM templates; and deploy LXC containers and VMs from templates. Feel free to use it and thanks for reading!\nReferences #Proxmox:\nProxmox Proxmox Documentation Terraform:\nTerraform Terraform Documentation provider block dynamic block Terraform Modules Github: Hashicorp/Terraform Github: Example Terraform .gitignore HashiCorp Learn: Modules HashiCorp Learn: Modules - Overview Terraform Providers / Plugins:\nBPG/Terraform-Provider-Proxmox https://registry.terraform.io/providers/bpg/proxmox/latest/docs/resources/virtual_environment_vm Other:\nCloud-init Documentation ","date":"May 12, 2024","permalink":"https://www.trfore.com/posts/provisioning-proxmox-8-vms-with-terraform-and-bpg/","section":"Posts","summary":"Quickly provision VMs and LXCs with the fast BPG Proxmox Terraform provider","title":"Provisioning Proxmox 8 VMs with Terraform and BPG"},{"content":" This post is work in progress and update to the prior Terraform post. The Telmate Proxmox provider is undergoing significant changes in v3, requiring additional permissions and featuring an updated disk schema. As for now, v2.9 is no longer compatible with Proxmox 8 and v3 is still in the testing phase. To use updated disk schema, you will need to build the plugin. Updated Permissions for PVE 8 #With the pending release of Telmate v3, you\u0026rsquo;ll need to grant additional privileges: Pool.Allocate, SDN.Use, Sys.Audit Sys.Console and Sys.Modify. Furthermore, you\u0026rsquo;ll need to grant access to the root path, /.\nWe\u0026rsquo;ll still use the Proxmox CLI to create a new role, group with permissions, and user; then generate an access token for the terraform user, as follows:\n# create role pveum role add TerraformUser -privs \u0026#34;Datastore.AllocateSpace \\ Datastore.Audit Pool.Allocate SDN.Use Sys.Audit Sys.Console \\ Sys.Modify VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \\ VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk \\ VM.Config.HWType VM.Config.Memory VM.Config.Network \\ VM.Config.Options VM.Migrate VM.Monitor VM.PowerMgmt\u0026#34; # create group pveum group add terraform-users # add root permissions pveum acl modify / -group terraform-users -role TerraformUser # create user \u0026#39;terraform\u0026#39; pveum useradd terraform@pve -groups terraform-users # generate a token pveum user token add terraform@pve token -privsep 0 The last command will output a token value similar to the following:\n┌──────────────┬──────────────────────────────────────┐ │ key │ value │ ╞══════════════╪══════════════════════════════════════╡ │ full-tokenid │ terraform@pve!token │ ├──────────────┼──────────────────────────────────────┤ │ info │ {\u0026#34;privsep\u0026#34;:\u0026#34;0\u0026#34;} │ ├──────────────┼──────────────────────────────────────┤ │ value │ 782a7700-4010-4802-8f4d-820f1b226850 │ └──────────────┴──────────────────────────────────────┘ Terraform Provider: Telmate Proxmox #Building the Plugin #Currently, the revised disk block is only available in the \u0026rsquo;new-disk\u0026rsquo; branch. So we\u0026rsquo;ll need to compile it from source. To build the plugin, you will need to have Go and GNU make installed on your machine. After that, you can run the following commands to build and install the plugin:\n# clone the repo git clone https://github.com/Telmate/terraform-provider-proxmox cd terraform-provider-proxmox # change the branch git checkout new-disk # build the binary make This will generate a binary file under bin/terraform-provider-proxmox. To use the binary, we\u0026rsquo;ll need to move it into the plugins directory, ~/.terraform.d/plugins (Unix) or %APPDATA%\\terraform.d\\plugins (Windows). For practicality\u0026rsquo;s sake, we\u0026rsquo;ll name it 3.0.1, however, note that this is not an official release. The following directions are for linux:\n# move the binary into the terraform plugins directory VERSION=\u0026#39;3.0.1\u0026#39; mkdir -p ~/.terraform.d/plugins/registry.terraform.io/telmate/proxmox/\u0026#34;${VERSION}\u0026#34;/linux_amd64/ cp bin/terraform-provider-proxmox ~/.terraform.d/plugins/registry.terraform.io/telmate/proxmox/\u0026#34;${VERSION}\u0026#34;/linux_amd64/terraform-provider-proxmox_v\u0026#34;${VERSION}\u0026#34; Update your terraform files to use the new binary:\nterraform { required_providers { proxmox = { source = \u0026#34;Telmate/proxmox\u0026#34; version = \u0026#34;3.0.1\u0026#34; } } } This will create a symlink in ~/.terraform.d/plugin-cache/registry.terraform.io/telmate/proxmox\n➜ tree ~/.terraform.d/plugin-cache/registry.terraform.io/telmate/proxmox/ /home/$USER/.terraform.d/plugin-cache/registry.terraform.io/telmate/proxmox/ ... ├── 3.0.1 │ └── linux_amd64 -\u0026gt; /home/$USER/.terraform.d/plugins/registry.terraform.io/telmate/proxmox/3.0.1/linux_amd64 ... Provisioning Disk in V3 #With the new version, you can provision disks using either the new disks block or revised disk block. We\u0026rsquo;ll use the later, as it shares a similar schema to v2.9 and requires minimal modification to the module created in the prior post. Here are the changes to the disk attributes:\n# hard-coded example disk { type = \u0026#34;disk\u0026#34; # change - value, \u0026#39;disk\u0026#39; or \u0026#39;cdrom\u0026#39; slot = \u0026#34;scsi0\u0026#34; # change - merge of \u0026#39;type\u0026#39; and \u0026#39;slot\u0026#39; from v2.9 storage = \u0026#34;local-lvm\u0026#34; size = \u0026#34;8G\u0026#34; format = \u0026#34;raw\u0026#34; cache = \u0026#34;writeback\u0026#34; backup = false # change (v2.9.13) - type, boolean iothread = false # change - type, boolean emulatessd = true # change - attribute name, formerly \u0026#39;ssd\u0026#39; discard = true # change - type, boolean } Updated Module #The updated module is available here (GitHub link). This is a work in progress and will be updated as new release candidates become available. You can use the module by setting the source to github.com/trfore/terraform-telmate-proxmox//modules/vm?ref=v3 in the module block, as follows:\n# main.tf terraform { required_providers { proxmox = { source = \u0026#34;Telmate/proxmox\u0026#34; version = \u0026#34;~\u0026gt; 3.0.0\u0026#34; } } } provider \u0026#34;proxmox\u0026#34; { pm_api_url = var.pve_api_url pm_api_token_id = var.pve_token_id pm_api_token_secret = var.pve_token_secret } module \u0026#34;vm_minimal_config\u0026#34; { source = \u0026#34;github.com/trfore/terraform-telmate-proxmox//modules/vm?ref=v3\u0026#34; node = \u0026#34;pve\u0026#34; vm_id = 100 vm_name = \u0026#34;vm-example-minimal\u0026#34; template_name = \u0026#34;ubuntu20\u0026#34; ci_ssh_key = \u0026#34;~/.ssh/id_ed25519.pub\u0026#34; } The LXC module works as expected.\nmodule \u0026#34;lxc_minimal_config\u0026#34; { source = \u0026#34;github.com/trfore/terraform-telmate-proxmox//modules/lxc?ref=v3\u0026#34; node = \u0026#34;pve\u0026#34; lxc_id = 100 os_template = \u0026#34;local:vztmpl/ubuntu-20.04-standard_20.04-1_amd64.tar.gz\u0026#34; os_type = \u0026#34;ubuntu\u0026#34; user_ssh_key_public = \u0026#34;~/.ssh/id_ed25519.pub\u0026#34; } Current Limitations # The required permissions are quite broad compared to v2.9.11, this is currently an open issue (#784). Customizing the cloud-init drive is not supported yet, but is planned (issue #986). Currently, setting cloudinit_cdrom_storage will create a config drive attached to ide3. Adding pre-enrolled keys for EFI disk is not supported yet (issue #1021). References #Companion Repo: Github: trfore/terraform-telmate-proxmox\nTerraform:\nTerraform Terraform Documentation dynamic blocks Github: Hashicorp/Terraform Github: Example Terraform .gitignore HashiCorp Learn: Modules Hashicorp Docs: Locals HashiCorp Learn: Modules - Overview Terraform Providers / Plugins:\nTelmate/Terraform-Provider-Proxmox Other Resources:\nTerraform: Up \u0026amp; Running ","date":"April 28, 2024","permalink":"https://www.trfore.com/posts/provisioning-proxmox-8-vms-with-terraform-and-telmate/","section":"Posts","summary":"","title":"Provisioning Proxmox 8 VMs with Terraform and Telmate"},{"content":"Intro #Hashicorp Packer is an open-source tool that lets you build consistent and reproducible virtual machines and containers from a configuration file. It supports various platforms and provides a wide range of features for building custom images. It is highly extensible, with broad community and official support for major cloud providers, local hypervisors and virtualization software via builder plugins.\nA configuration file consist of several top-level code blocks - source, variable and build, with each block containing sets of key-value pairs. The source block configures a builder plugin, typically defining machine attributes, e.g. vcpus, and platform access, e.g. API tokens. You can set these values using variables that are defined in variable blocks with default values and HCL types. Lastly, the build block is used to combine builders and provisioners to create a final image.\nFor this tutorial, we\u0026rsquo;ll use the Proxmox plugin and it\u0026rsquo;s proxmox-iso builder to create a Proxmox template based on Ubuntu server 22.04. We\u0026rsquo;ll craft the template so that meets the following criteria:\nAll packages are up-to-date and cloud-init, openssh-server and qemu-guest-agent are pre-installed. There are no pre-existing users or credentials on the image. The image is reset to a clean state, so that it can be fully configured using cloud-init. To achieve this, we\u0026rsquo;ll use several tricks in the autoinstall configuration file and build the image using only the root user.\nFor a universal Packer configuration to create Debian, CentOS, Fedora and Ubuntu templates, see the companion repo: Packer Proxmox Templates. Grant Packer Access to Proxmox #Packer configures the VM template by making API request to a Proxmox Virtual Environment (PVE) endpoint. To grant Packer access, let\u0026rsquo;s use the Proxmox CLI to create a new role, group with permissions, and user; then generate an access token for the packer user. For additional configuration options see Proxmox Wiki: User Management or pveum help.\nFrom the PVE CLI enter the following commands:\n# create role pveum role add PackerUser --privs \u0026#34;Datastore.AllocateSpace \\ Datastore.AllocateTemplate Datastore.Audit Sys.Audit Sys.Modify \\ VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk \\ VM.Config.HWType VM.Config.Memory VM.Config.Network \\ VM.Config.Options VM.Console VM.Monitor VM.PowerMgmt\u0026#34; # additional permissions for PVE 8 pveum role modify PackerUser --privs \u0026#34;SDN.Use\u0026#34; --append # create group pveum group add packer-users # add permissions pveum acl modify / -group packer-users -role PackerUser # create user \u0026#39;packer\u0026#39; pveum useradd packer@pve -groups packer-users # generate a token pveum user token add packer@pve token -privsep 0 The last command will output a token value similar to the following, which we will use to validate Packer with Proxmox.\n┌──────────────┬──────────────────────────────────────┐ │ key │ value │ ╞══════════════╪══════════════════════════════════════╡ │ full-tokenid │ packer@pve!token │ ├──────────────┼──────────────────────────────────────┤ │ info │ {\u0026#34;privsep\u0026#34;:\u0026#34;0\u0026#34;} │ ├──────────────┼──────────────────────────────────────┤ │ value │ 782a7700-4010-4802-8f4d-820f1b226850 │ └──────────────┴──────────────────────────────────────┘ Packer: Configure the Build #In this tutorial, we will create several files: pve-image.pkr.hcl, pve-vars.pkr.hcl, meta-data and user-data. The following sections discuss the logic behind each file and Packer code using snippets with complete code examples provided at the end. Code examples primarily use hard-coded values for simplicity, however, consider replacing these values with variables, using pve-vars.pkr.hcl as a reference.\nSource Block: Writing a Reusable Top-Level Source Block #Start by creating a top-level source block that uses the proxmox-iso builder with a simple but unique identifier, image. This block will be used as a common source configuration for all of the builds, with the goal of defining stable common values such as: 1) Proxmox login credentials; 2) SSH login credentials for Packer; and 3) PVE template parameters. For enhanced readability, group similar values together and use comment headers to demarcate each group. Later, we will create a build-level source block that uses distribution specific configuration values such as the ISO image and template name.\nAll of the templates will use cloud-init to configure the machine image, so we\u0026rsquo;ll attach a cloud-init drive and add a single boot disk to install the base ISO onto. Additionally, we will install qemu-guest-agent using cloud-init to enable PVE management of VMs cloned from a template, so enable qemu_agent in the Packer configuration. Furthermore, we will add a dynamic description that includes the creation date using the built-in timestamp function.\nPacker variable blocks do not support using functions in the default value, so it\u0026rsquo;s best to hard code template_description. If we used this value in multiple locations, using a local that supports functions would be more appropriate. To mask sensitive values from Packer\u0026rsquo;s output and pass via environment variables, we\u0026rsquo;ll use variables for PVE and SSH login credentials, e.g. var.pve_token. In the next section , we\u0026rsquo;ll create the corresponding variable blocks.\n# pve-image.pkr.hcl source \u0026#34;proxmox-iso\u0026#34; \u0026#34;image\u0026#34; { // PVE login proxmox_url = var.pve_api_url username = var.pve_username token = var.pve_token node = var.pve_node insecure_skip_tls_verify = true // SSH ssh_username = var.ssh_username ssh_keypair_name = var.ssh_keypair_name ssh_private_key_file = var.ssh_private_key_file ssh_clear_authorized_keys = true ssh_timeout = \u0026#34;20m\u0026#34; // ISO ... template_description = \u0026#34;Packer generated template image on ${timestamp()}\u0026#34; // System ... qemu_agent = true // Disks disks { type = \u0026#34;scsi\u0026#34; storage_pool = \u0026#34;local-lvm\u0026#34; disk_size = \u0026#34;10G\u0026#34; cache_mode = \u0026#34;writeback\u0026#34; format = \u0026#34;raw\u0026#34; } // Cloud-init cloud_init = true cloud_init_storage_pool = \u0026#34;local-lvm\u0026#34; // CPU \u0026amp; Memory ... // Network ... } Variable Blocks: Masking Sensitive Data \u0026amp; Validating Inputs #Next, let\u0026rsquo;s create several variables using variable blocks to validate inputs, set reasonable defaults and mask sensitive data. This will make it easier to pass in sensitive and non-sensitive values via environment variables and command line arguments, respectively.\nPacker expects all variables passed via the CLI to be strings, but you can specify the HCL type for each variable by setting type, ensuring that type constraints and transformations are properly applied. You can also validate a input further using validation rules within a validation block. In the example below, we\u0026rsquo;ll validate that the pve_api_url input begins with http(s):// and ends with /api2/json using the built-in regex function. The can function converts the regex output into a boolean for the validation rule to use as a conditional.\nSetting a default value for a variable ensures that Packer will always pass a value when building an image, turning it into an optional parameter for the end-user. Conversely, excluding the default key will cause Packer to fail if a value is not passed via a variable file or CLI. Setting default = null is a special use-case, in which Packer skips checking for an input and assumes the plugin builder key using the variable is an optional parameter and handles null values.\nTo validate a variable configuration file against the schema, pass dummy values for undefined variables to packer validate:\npacker validate \\ -var \u0026#39;pve_token=TOKEN\u0026#39; \\ -var \u0026#39;pve_username=packer@pve!token\u0026#39; \\ -var \u0026#39;pve_api_url=http://pve.example.com/api2/json\u0026#39; \\ pve-vars.pkr.hcl Next, let\u0026rsquo;s set all PVE login and SSH variables to sensitive = true to mask these values from Packer\u0026rsquo;s logs.\n# pve-vars.pkr.hcl // PVE Variables ... variable \u0026#34;pve_api_url\u0026#34; { description = \u0026#34;Proxmox API Endpoint, e.g. \u0026#39;https://pve.example.com/api2/json\u0026#39;\u0026#34; type = string sensitive = true validation { condition = can(regex(\u0026#34;(?i)^http[s]?://.*/api2/json$\u0026#34;, var.pve_api_url)) error_message = \u0026#34;Proxmox API Endpoint Invalid. Check URL - Scheme and Path required.\u0026#34; } } // SSH Variables variable \u0026#34;ssh_private_key_file\u0026#34; { description = \u0026#34;Private SSH Key for VM\u0026#34; default = \u0026#34;~/.ssh/packer_id_ed25519\u0026#34; type = string sensitive = true } ... Build Block: Extending Top-Level Source Block Configurations #In this last part, we\u0026rsquo;ll create a build block and extend the top-level source block configuration from above with distribution specific boot commands and cloud-init configuration files. Additionally, we\u0026rsquo;ll use a built-in provisioner to run several shell commands to prepare the image for provisioning.\nTo extend a top-level source block, we\u0026rsquo;ll create a build-level source block that references the builder and identifier, proxmox-iso.image, of the top-level block and give it a unique name, i.e. ubuntu22. This will allow us to reference this specific build configuration in the CLI using the only or except flags, e.g. packer build -only=proxmox-iso.ubuntu22. This name is not to be confused with the template_name, which is the template name in Proxmox.\nHere we\u0026rsquo;ll also define variables that are specific to a given distribution, iso_url and iso_checksum, and set the template ID to 9000. I\u0026rsquo;ll cover the additional parts and provider block over the next few sections.\n# pve-image.pkr.hcl ... build { source \u0026#34;proxmox-iso.image\u0026#34; { name = \u0026#34;ubuntu22\u0026#34; # Packer Reference Name template_name = \u0026#34;ubuntu22\u0026#34; # Proxmox Template Name vm_id = 9000 iso_url = \u0026#34;https://releases.ubuntu.com/22.04/ubuntu-22.04.2-live-server-amd64.iso\u0026#34; iso_checksum = \u0026#34;file:https://releases.ubuntu.com/22.04/SHA256SUMS\u0026#34; boot_wait = \u0026#34;5s\u0026#34; boot_command = [ \u0026#34;c\u0026#34;, \u0026#34;linux /casper/vmlinuz --- autoinstall \u0026#39;ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/\u0026#39; \u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;initrd /casper/initrd\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;boot\u0026lt;enter\u0026gt;\u0026#34; ] http_content = { \u0026#34;/meta-data\u0026#34; = file(\u0026#34;configs/meta-data\u0026#34;) \u0026#34;/user-data\u0026#34; = templatefile(\u0026#34;configs/user-data\u0026#34;, { var = var, ssh_public_key = chomp(file(var.ssh_public_key_file)) }) } } provisioner \u0026#34;shell\u0026#34; { inline = [ \u0026#34;while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo \u0026#39;Waiting for cloud-init...\u0026#39;; sleep 1; done\u0026#34;, // clean image identifiers \u0026#34;cloud-init clean --machine-id --seed\u0026#34;, \u0026#34;rm /etc/hostname /etc/ssh/ssh_host_* /var/lib/systemd/random-seed\u0026#34;, \u0026#34;truncate -s 0 /root/.ssh/authorized_keys\u0026#34;, // disable SSH password authentication and root access \u0026#34;sed -i \u0026#39;s/^#PasswordAuthentication\\\\ yes/PasswordAuthentication\\\\ no/\u0026#39; /etc/ssh/sshd_config\u0026#34;, \u0026#34;sed -i \u0026#39;s/^#PermitRootLogin\\\\ prohibit-password/PermitRootLogin\\\\ no/\u0026#39; /etc/ssh/sshd_config\u0026#34; ] } } Distribution Specific Boot Commands #The proxmox-iso builder boot command provides a set of special keys to interact with an image\u0026rsquo;s boot loader. Also, it can generate an HTTP server and serve files defined in http_content, which is useful for providing configuration files. Here, we will escape GRUB\u0026rsquo;s automatic boot sequence, c, and instruct the kernel to use autoinstall to skip the guided menu prompts and set the datasource as ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/, which will instruct cloud-init to fetch the autoinstall configuration from Packer\u0026rsquo;s HTTP server.\n# pve-image.pkr.hcl build{ source \u0026#34;proxmox-iso.image\u0026#34; { boot_wait = \u0026#34;5s\u0026#34; boot_command = [ \u0026#34;c\u0026#34;, \u0026#34;linux /casper/vmlinuz --- autoinstall \u0026#39;ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/\u0026#39; \u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;initrd /casper/initrd\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;boot\u0026lt;enter\u0026gt;\u0026#34; ] ... } } Autoinstall Configuration File #For the subiquity autoinstall, we\u0026rsquo;ll create two files: meta-data and user-data. The meta-data file is a single line file that contains the word uninitialized. While the user-data file is a mixture of autoinstall directives and cloud-init directives. For the user-data file, we\u0026rsquo;ll create a Packer template that uses variables to define several key values. The ssh_public_key_file variable is transformed using built-in functions and assigned to a new key prior to being passed into the template; while all other variables are passed by including: var = var.\n# pve-image.pkr.hcl build{ source \u0026#34;proxmox-iso.image\u0026#34; { ... http_content = { \u0026#34;/meta-data\u0026#34; = file(\u0026#34;configs/meta-data\u0026#34;) \u0026#34;/user-data\u0026#34; = templatefile(\u0026#34;configs/user-data\u0026#34;, { var = var, ssh_public_key = chomp(file(var.ssh_public_key_file)) }) } } } For the user-data file, we\u0026rsquo;ll avoid creating a default user and instead use the root user during the build process. Autoinstall requires that either the identity or user-data section be present in the configuration. Using identity is a non-starter as it will create a new user with a password that is required for all calls to sudo, which complicates using commands in the shell provisioner. Instead we\u0026rsquo;ll use the user-data directive and skip creating a user by supplying an empty list, users: []; and enable root SSH access by setting disable_root: false. We\u0026rsquo;ll also disable password authentication for all users with ssh.allow-pw: false and add a SSH key for Packer to use during the build. Both the root SSH key and access will be removed at the end of the build with the shell provisioner (next section).\nBy using the root user to build the image, we avoid creating the standard default user, ubuntu, or another one that will remain on the image after the build - even if you set a different name for the default user in cloud-init. This approach does require that a user is created during provisioning with a password and/or SSH key, but this is easily configured using the PVE GUI or a provisioning tool like Terraform (discussed later).\nOne of the advantages of using a Packer template is the ability to conditionally apply key-values. In the example below, if values are supplied for apt_proxy_http and/or apt_proxy_https, then an APT proxy configuration file is created at /etc/apt/apt.conf.d/90curtin-aptproxy; otherwise, these keys are never generated in the configuration file.\n#cloud-config autoinstall: version: 1 user-data: disable_root: false # allow SSH for root users: [] # skip creating default users locale: en_US.UTF-8 packages: - qemu-guest-agent refresh-installer: update: true shutdown: reboot ssh: install-server: true # enable SSH server allow-pw: false # disable password authentication authorized-keys: - ${ssh_public_key} # add SSH key to root user timezone: UTC updates: all # update all packages apt: %{ if var.apt_proxy_http != \u0026#34;\u0026#34; } http_proxy: ${var.apt_proxy_http} %{ endif } %{ if var.apt_proxy_https != \u0026#34;\u0026#34; } https_proxy: ${var.apt_proxy_https} %{ endif } Additional top-level keys can be found on the autoinstall configuration site.\nPreparing the Image for Provisioning #Finally, to prepare the image for provisioning we\u0026rsquo;ll use the shell provisioner to reset cloud-init; remove the hostname; and re-configure SSH access. The while ... line is a common practice to ensure that cloud-init finishes prior to shutting down the instance or running additional commands.\nTo remove the /var/lib/cloud/ directory and clear /etc/machine-id, add cloud-init clean --machine-id --seed. Additionally, we\u0026rsquo;ll clear /var/lib/systemd/random-seed to avoid clones from starting with the same random pool. To secure SSH access, we\u0026rsquo;ll clear the SSH keys generated during the build and used by Packer for the root user; and pre-configure the OpenSSH server to disable password authentication and root access.\n# pve-image.pkr.hcl build{ ... provisioner \u0026#34;shell\u0026#34; { inline = [ \u0026#34;while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo \u0026#39;Waiting for cloud-init...\u0026#39;; sleep 1; done\u0026#34;, // clean image identifiers \u0026#34;cloud-init clean --machine-id --seed\u0026#34;, \u0026#34;rm /etc/hostname /etc/ssh/ssh_host_* /var/lib/systemd/random-seed\u0026#34;, \u0026#34;truncate -s 0 /root/.ssh/authorized_keys\u0026#34;, // disable SSH password authentication and root access \u0026#34;sed -i \u0026#39;s/^#PasswordAuthentication\\\\ yes/PasswordAuthentication\\\\ no/\u0026#39; /etc/ssh/sshd_config\u0026#34;, \u0026#34;sed -i \u0026#39;s/^#PermitRootLogin\\\\ prohibit-password/PermitRootLogin\\\\ no/\u0026#39; /etc/ssh/sshd_config\u0026#34; ] } } For additional information on cleaning images prior to provisioning, see systemd: Building Images.\nPacker: Build the Image #With the completed configuration files, we can now build our image and PVE template. Packer allows you to set variables using: 1) environment variables, prefixing the variable name with PKR_VAR_; 2) CLI args, using the -var= flag; and/or 3) variable files, having the extension *.pkrvars.hcl. In the following example we\u0026rsquo;ll set two environment variables for our Proxmox API token, then pass the Proxmox URL via the CLI.\nexport PKR_VAR_pve_username=\u0026#39;packer@pve!token\u0026#39; export PKR_VAR_pve_token=\u0026#39;782a7700-4010-4802-8f4d-820f1b226850\u0026#39; # build packer image packer build \\ -var=\u0026#39;pve_api_url=https://pve.example.com/api2/json\u0026#39; \\ . After the image is built, you should see a new template in your Proxmox GUI. Test it out by cloning the template, then adding a default user in the cloud-init section. You must add a password for console access and/or a SSH key for remote access. Leaving the user field blank will default to creating a user named ubuntu. Also, you must configure the \u0026lsquo;IP Config\u0026rsquo; section.\nRecap #Packer provides a streamlined approach for building Proxmox VM templates. This post explored the steps involved in creating a custom Proxmox template using Packer, covering builder plugin configurations, variable definitions, and image sanitation. By leveraging Packer, you can automate the process of building and managing your Proxmox infrastructure, ensuring consistency and efficiency in your VM deployments.\nBy using a common top-level source block, you can easily extend this the build block to create multiple templates from different linux distributions. Furthermore, by building your images using only the root user, you can ensure that your images are free of any user level configurations and ready for cloud-init.\nFor ideas on how to extend this template for other operating systems, please refer to the companion repo: Packer Proxmox Templates. Limitations # Avoid adding a cloud-init network-config file to the template, as you will want to configure network settings using Proxmox\u0026rsquo;s cloud-init integration. Also the network configuration is cleared at the end of the template creation via cloud-init clean. The Proxmox plugin does not provide a way to configure cloud-init settings that are available in the Proxmox cloud-init GUI, Proxmox plugin - issue #7. This only leaves the option of manually configuring the values via Proxmox or third-party provisioner, e.g. Terraform. Note: With plugin version 1.1.3, you can use the \u0026ldquo;clone\u0026rdquo; builder to add cloud-init networking information, however, the information is removed during template finalization. Also the \u0026ldquo;clone\u0026rdquo; builder is only for creating template clones - not to be confused with \u0026ldquo;clones\u0026rdquo; in Proxmox, which is a way to provision VMs from Proxmox templates. You are unable to use *.qcow2 cloud images as a base OS, see Proxmox plugin - issue #29. Building the Image With a Non-Root User #If you decide create a new user using the identity or user-data in the autoinstall configuration, you will need to provide a password for sudo in the shell provisioner. This can be done by piping the password into sudo -S from stdin as follows:\n# pve-image.pkr.hcl provisioner \u0026#34;shell\u0026#34; { env = { USER_PASSWORD = var.ssh_password } inline = [ ... // clean image identifiers \u0026#34;echo $USER_PASSWORD | sudo -S cloud-init clean --machine-id --seed\u0026#34;, ... ] } Change and add the following variables:\n# pve-vars.pkr.hcl variable \u0026#34;ssh_username\u0026#34; { description = \u0026#34;SSH Username\u0026#34; type = string default = \u0026#34;packer\u0026#34; } variable \u0026#34;ssh_password\u0026#34; { description = \u0026#34;SSH User Password\u0026#34; type = string default = \u0026#34;password\u0026#34; sensitive = true } Update the autoinstall configuration:\n#cloud-config autoinstall: version: 1 user-data: users: - name: ${var.ssh_username} groups: users, admin sudo: ALL=(ALL) NOPASSWD:ALL shell: /bin/bash ssh_authorized_keys: - ${ssh_public_key} ... ssh: install-server: true timezone: UTC See cloud-init module \u0026lsquo;users\u0026rsquo;, for more configuration options.\nComplete Code Examples #pve-image.pkr.hcl #packer { required_plugins { proxmox = { version = \u0026#34;\u0026gt;=1.1.2\u0026#34; source = \u0026#34;github.com/hashicorp/proxmox\u0026#34; } } } source \u0026#34;proxmox-iso\u0026#34; \u0026#34;image\u0026#34; { // PVE login proxmox_url = var.pve_api_url username = var.pve_username token = var.pve_token node = var.pve_node insecure_skip_tls_verify = true // SSH ssh_username = var.ssh_username ssh_keypair_name = var.ssh_keypair_name ssh_private_key_file = var.ssh_private_key_file ssh_clear_authorized_keys = true ssh_timeout = \u0026#34;20m\u0026#34; // ISO iso_download_pve = true # added in v1.1.2 iso_storage_pool = \u0026#34;local\u0026#34; unmount_iso = true os = \u0026#34;l26\u0026#34; template_description = \u0026#34;Packer generated template image on ${timestamp()}\u0026#34; // System machine = \u0026#34;q35\u0026#34; bios = \u0026#34;seabios\u0026#34; qemu_agent = true // Disks scsi_controller = \u0026#34;virtio-scsi-pci\u0026#34; disks { type = \u0026#34;scsi\u0026#34; storage_pool = \u0026#34;local-lvm\u0026#34; // storage_pool_type = \u0026#34;lvm\u0026#34; # depreciated in v1.1.2 disk_size = \u0026#34;10G\u0026#34; cache_mode = \u0026#34;writeback\u0026#34; format = \u0026#34;raw\u0026#34; io_thread = false } // Cloud-init cloud_init = true cloud_init_storage_pool = \u0026#34;local-lvm\u0026#34; // CPU \u0026amp; Memory sockets = 1 cores = 2 cpu_type = \u0026#34;host\u0026#34; memory = 2048 // Network network_adapters { bridge = \u0026#34;vmbr0\u0026#34; model = \u0026#34;virtio\u0026#34; vlan_tag = \u0026#34;1\u0026#34; firewall = false } } build { source \u0026#34;proxmox-iso.image\u0026#34; { name = \u0026#34;ubuntu22\u0026#34; template_name = \u0026#34;ubuntu22\u0026#34; vm_id = 9000 iso_url = \u0026#34;https://releases.ubuntu.com/22.04/ubuntu-22.04.2-live-server-amd64.iso\u0026#34; iso_checksum = \u0026#34;file:https://releases.ubuntu.com/22.04/SHA256SUMS\u0026#34; boot_wait = \u0026#34;5s\u0026#34; boot_command = [ \u0026#34;c\u0026#34;, \u0026#34;linux /casper/vmlinuz --- autoinstall \u0026#39;ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/\u0026#39; \u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;initrd /casper/initrd\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;boot\u0026lt;enter\u0026gt;\u0026#34; ] http_content = { \u0026#34;/meta-data\u0026#34; = file(\u0026#34;configs/meta-data\u0026#34;) \u0026#34;/user-data\u0026#34; = templatefile(\u0026#34;configs/user-data\u0026#34;, { var = var, ssh_public_key = chomp(file(var.ssh_public_key_file)) }) } } provisioner \u0026#34;shell\u0026#34; { inline = [ \u0026#34;while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo \u0026#39;Waiting for cloud-init...\u0026#39;; sleep 1; done\u0026#34;, // clean image identifiers \u0026#34;cloud-init clean --machine-id --seed\u0026#34;, \u0026#34;rm /etc/hostname /etc/ssh/ssh_host_* /var/lib/systemd/random-seed\u0026#34;, \u0026#34;truncate -s 0 /root/.ssh/authorized_keys\u0026#34;, // disable SSH password authentication and root access \u0026#34;sed -i \u0026#39;s/^#PasswordAuthentication\\\\ yes/PasswordAuthentication\\\\ no/\u0026#39; /etc/ssh/sshd_config\u0026#34;, \u0026#34;sed -i \u0026#39;s/^#PermitRootLogin\\\\ prohibit-password/PermitRootLogin\\\\ no/\u0026#39; /etc/ssh/sshd_config\u0026#34; ] } } pve-vars.pkr.hcl #// PVE Variables // // Pass Sensitive Variables CLI Args or Env Vars variable \u0026#34;pve_token\u0026#34; { description = \u0026#34;Proxmox API Token, e.g. \u0026#39;782a7700-4010-4802-8f4d-820f1b226850\u0026#39;.\u0026#34; type = string sensitive = true } variable \u0026#34;pve_username\u0026#34; { description = \u0026#34;Username when authenticating to Proxmox, e.g. \u0026#39;packer@pve!token\u0026#39;.\u0026#34; type = string sensitive = true } variable \u0026#34;pve_api_url\u0026#34; { description = \u0026#34;Proxmox API Endpoint, e.g. \u0026#39;https://pve.example.com/api2/json\u0026#39;.\u0026#34; type = string sensitive = true validation { condition = can(regex(\u0026#34;(?i)^http[s]?://.*/api2/json$\u0026#34;, var.pve_api_url)) error_message = \u0026#34;Proxmox API Endpoint Invalid. Check URL - Scheme and Path required.\u0026#34; } } variable \u0026#34;pve_node\u0026#34; { type = string default = \u0026#34;pve\u0026#34; } // SSH Variables // variable \u0026#34;ssh_username\u0026#34; { description = \u0026#34;Image SSH Username\u0026#34; type = string default = \u0026#34;root\u0026#34; } variable \u0026#34;ssh_keypair_name\u0026#34; { default = \u0026#34;packer_id_ed25519\u0026#34; type = string } variable \u0026#34;ssh_private_key_file\u0026#34; { description = \u0026#34;Private SSH Key for VM\u0026#34; default = \u0026#34;~/.ssh/packer_id_ed25519\u0026#34; type = string sensitive = true } variable \u0026#34;ssh_public_key_file\u0026#34; { description = \u0026#34;Public SSH Key for VM\u0026#34; default = \u0026#34;~/.ssh/packer_id_ed25519.pub\u0026#34; type = string sensitive = true } // Config Template Variables variable \u0026#34;apt_proxy_http\u0026#34; { description = \u0026lt;\u0026lt;EOT APT proxy URL for Ubuntu, format: \u0026#39;http://[[user][:pass]@]host[:port]/\u0026#39;. Default \u0026#39;null\u0026#39; skips setting proxy. EOT type = string default = \u0026#34;\u0026#34; } variable \u0026#34;apt_proxy_https\u0026#34; { description = \u0026lt;\u0026lt;EOT APT proxy URL for Ubuntu, format: \u0026#39;https://[[user][:pass]@]host[:port]/\u0026#39;. Default \u0026#39;null\u0026#39; skips setting proxy. EOT type = string default = \u0026#34;\u0026#34; } meta-data #uninitialized user-data ##cloud-config autoinstall: version: 1 user-data: disable_root: false # allow SSH for root users: [] # skip creating default users locale: en_US.UTF-8 packages: - qemu-guest-agent refresh-installer: update: true shutdown: reboot ssh: install-server: true # enable SSH server allow-pw: false # disable password authentication authorized-keys: - ${ssh_public_key} # add SSH key to root user timezone: UTC updates: all # update all packages apt: %{ if var.apt_proxy_http != \u0026#34;\u0026#34; } http_proxy: ${var.apt_proxy_http} %{ endif } %{ if var.apt_proxy_https != \u0026#34;\u0026#34; } https_proxy: ${var.apt_proxy_https} %{ endif } References # Source code: Packer Proxmox Templates cloud-init systemd: Building Images Ubuntu Docs: Automated Server Installation Packer:\nHashiCorp Packer: Install Packer Build block Source block Variable blocks Provisioner Shell Provisioner Packer Integrations Packer Integrations: Proxmox SSH communicator Github: Packer Plugin Proxmox Proxmox:\nProxmox Proxmox template Proxmox Wiki: User Management qemu-guest-agent ","date":"February 26, 2023","permalink":"https://www.trfore.com/posts/golden-images-and-proxmox-templates-with-packer/","section":"Posts","summary":"Streamline Proxmox VE templating with Packer and Cloud-init","title":"Golden Images and Proxmox Templates with Packer"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/packer/","section":"Tags","summary":"","title":"Packer"},{"content":"Intro #In the last post, we created a custom Proxmox template that was configured using cloud-init. While that approach is one of the best ways to configure a new virtual machine (VM), there are several use cases where you may need to modify an image prior to booting a VM. This is easily achieved using libguestfs, which allows you to directly modify cloud images using shell commands.\nLibguestfs is a C library and collection of tools that lets you:\nInstall packages Transfer files in and out Directly edit files with your favorite editor Run shell scripts And more Many of the commands mimic familiar Linux commands, e.g. virt-ls lists files within a disk image, making the transition seamless. Additionally, the package, libguestfs-tools, is easily installed via the APT and YUM repositories.\nIn this tutorial, we\u0026rsquo;ll use libguestfs to modify a CentOS 9 Stream image and explore its capabilities by:\nInstalling qemu-guest-agent for seamless VM management Fortifying the SSH server configuration for enhanced security Fixing the elusive Probing EDD ... boot screen with a simple tweak We\u0026rsquo;ll test out our modifications by creating a simple Proxmox template and deploying a VM.\nI recommend modifying images on a local Linux desktop or VM rather than directly on the Proxmox host. Setup: Download Image #To start, create a new working directory on your desktop and download the image:\nmkdir -p ~/centos9_stream_mod \u0026amp;\u0026amp; cd ~/centos9_stream_mod wget https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 And validate the checksum of the image:\ncurl -sfL https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2.SHA256SUM | shasum -c --ignore-missing Optional, create a folder and copy of the original unmodified image:\nmkdir original_img \u0026amp;\u0026amp; \\ cp CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 original_img/ Setup: Install libguestfs #Next, install libguestfs using your OS package manager. For example on Debian/Ubuntu:\n# install the libguestfs sudo apt install libguestfs-tools Modifying Images Using libguestfs #Installing Packages #Virt-customize allows you to customize disk images in place, including: installing and uninstalling packages using the distribution’s package manager; editing user passwords; running shell commands; and more. Given, qemu-guest-agent is required for Proxmox to execute commands and safely shut down a VM, let’s install it using virt-customize with the --install option:\n# install qemu-guest-agent sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ --install qemu-guest-agent Warning: Running virt-customize and installing packages will set the machine-id. Therefore, all VM clones from distros that use systemd-networkd will have the same DHCP unique identifier (DUID) and receive the same IP address. Be sure to run the following command to clear the ID! # delete the machine id sudo virt-edit -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ /etc/machine-id -e \u0026#39;s/.*//\u0026#39; You can confirm that the machine-id is deleted by running virt-cat:\nsudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ /etc/machine-id Retrieving and Adding Files to the Image #You can copy files in and out of the image using libguestfs\u0026rsquo; virt-copy-in and virt-copy-out. This is great when you have pre-existing configurations or wish to edit a file using an IDE, yet we\u0026rsquo;ll directly edit the SSH configuration in the next section. Here\u0026rsquo;s an example of how to use these commands:\n# pull the SSH host config # syntax: virt-copy-out -a \u0026lt;IMAGE\u0026gt; \u0026lt;FILE_PATH\u0026gt; \u0026lt;OUTPUT_PATH\u0026gt; sudo virt-copy-out -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ /etc/ssh/sshd_config . # edit the file using your favorite editor # push the SSH server config back into the image # syntax: virt-copy-in -a \u0026lt;IMAGE\u0026gt; \u0026lt;FILE\u0026gt; \u0026lt;FILE_PATH\u0026gt; sudo virt-copy-in -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ sshd_config /etc/ssh/ Directly Editing Files in the Image #You can also edit files directly with virt-edit. We\u0026rsquo;ll use this approach to edit the sshd_config, enter the following command:\nsudo virt-edit -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ /etc/ssh/sshd_config The default editor is vi/vim. To use a different editor, append EDITOR=\u0026lt;editor\u0026gt; to the command, i.e. sudo EDITOR=nano virt-edit ..., or permanently set it via a systemd-wide environment variable:\necho \u0026#39;EDITOR=\u0026#34;nano\u0026#34;\u0026#39; | sudo tee -a /etc/environment Then let\u0026rsquo;s enhance the security of the SSH server, change the settings in sshd_config to the following values:\n# disable root login PermitRootLogin no # limit the number of failed login attempts before locking out a user MaxAuthTries 3 # disable password authentication for all users PasswordAuthentication no PermitEmptyPasswords no # disable X11 forwarding X11Forwarding no Feel free to set the values according to your security needs. When you\u0026rsquo;re done, save and exit the editor. You can confirm the changes by running virt-cat:\nsudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ /etc/ssh/sshd_config Running Shell Commands #Finally, we are going to edit the bootloader configuration to fix the empty boot screen you may see in the Proxmox VM console that only reads: Probing EDD (edd=off to disable)... ok. While this not a hard requirement to get CentOS booted and configured using cloud-init, it provides a good example of how to use virt-customize to run shell commands.\nThe empty boot screen is caused by two factors: i) the bootloader sets the console to serial, console=ttyS0,115200n8, and the output is masked in PVE GUI; ii) Enhanced Disk Drive (EDD) is enabled. There are several ways to handle this, including:\nIgnore it, hope the boot process completes without error and wait for a login screen. Add a serial terminal to the VM and monitor the boot process from the PVE server console using qm terminal \u0026lt;VMID\u0026gt;. Add a virtual terminal to the bootloader configuration. Optionally, disable Enhanced Disk Drive. We\u0026rsquo;ll go with the last approach as many find it easier use Proxmox\u0026rsquo;s GUI console to monitor the boot process. To change the console output and disable EDD (optional), we\u0026rsquo;ll start by using grubby to edit the default grub configuration:\n# update the kernel args - warning this sets the machine-id! sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ --run-command \u0026#39;grubby --args=\u0026#34;console=tty0\u0026#34; --update-kernel=ALL\u0026#39; Grubby will either add a new argument, or replace an existing one. In this case we want to replace console=ttyS0,115200n8 with a new argument that is more appropriate for PVE GUI: console=tty0. The --args argument is used to pass arguments to the bootloader, and --update-kernel=ALL will update all kernels.\nIf you would like to disable Enhanced Disk Drive (EDD) Services, add edd=off:\n# change console and disable EDD grubby --args=\u0026#34;console=tty0 edd=off\u0026#34; ... For more details checkout man grubby or grubby.\nNext, let\u0026rsquo;s go ahead and update the grub config using grub2-mkconfig:\n# update grub config - warning this sets the machine-id! sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ --run-command \u0026#39;grub2-mkconfig -o /boot/grub2/grub.cfg\u0026#39; Now we can confirm that the changes were successful by checking the default grub config, /etc/default/grub:\n# confirm the edit sudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ /etc/default/grub The new arguments are appended to the end of GRUB_CMDLINE_LINUX:\n# /etc/default/grub example GRUB_TIMEOUT=1 ... GRUB_CMDLINE_LINUX=\u0026#34;no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M console=tty0\u0026#34; ... And we can also check the bootloader entry, /boot/loader/entries/\u0026lt;MACHINE_ID\u0026gt;-\u0026lt;KERNEL_VERSION\u0026gt;.conf:\n# get the name of the bootloader file BOOTLOADER_FILE=$(sudo virt-ls -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 /boot/loader/entries) # confirm the edit sudo virt-cat -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ /boot/loader/entries/${BOOTLOADER_FILE} The new args are appended to the options line:\n# /boot/loader/entries/\u0026lt;MACHINE_ID\u0026gt;-\u0026lt;KERNEL_VERSION\u0026gt;.conf example title CentOS Stream (\u0026lt;KERNEL_VERSION\u0026gt;.el9.x86_64) 9 ... options root=UUID=\u0026lt;UUID_VALUE\u0026gt; ro no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M console=tty0 ... Create a Proxmox Template #Now that we have a modified image, let\u0026rsquo;s create a Proxmox template from it.\nChange the File Extension #In order to view the image within the Proxmox GUI, the file extension needs to be .img or .iso. While you can manually convert the image (see box below), you can trick Proxmox by changing the file extension and it will automatically convert the image as it is imported into a VM. This approach also saves space on your server. To change the file extension, run:\n# change the file extension mv CentOS-Stream-GenericCloud-9-latest.x86_64.{qcow2,img} You can use qemu-img to convert the image format from qcow2 to img, however, the converted file size is typically larger. To view the size of the image before converting, run qemu-image info and view the virtual size value.\n# install QEMU (pre-installed on Proxmox) sudo apt install qemu # view image information user@desktop:~$ qemu-img info \u0026lt;IMAGE\u0026gt; image: CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 file format: qcow2 virtual size: 10 GiB (10737418240 bytes) # converted size disk size: 1.13 GiB # original size ... To convert the image run:\n# convert the image qemu-img convert -f qcow2 -O raw \u0026lt;INPUT\u0026gt;.qcow2 \u0026lt;OUTPUT\u0026gt;.img Upload the Image #Now transfer the image to your Proxmox server using your preferred method, e.g. web GUI or scp.\nTo transfer the file using the web GUI, navigate to your storage pool for ISO images, i.e. Datacenter \u0026gt; NODE_NAME \u0026gt; 'local' \u0026gt; ISO Images \u0026gt; Upload, and select the image and click Upload.\nAlternatively, you can use scp to copy the file into your ISO storage, e.g. /var/lib/vz/template/iso:\n# copy image to proxmox server scp CentOS-Stream-GenericCloud-9-latest.x86_64.img \\ root@\u0026lt;PROXMOX_ADDRESS\u0026gt;:/var/lib/vz/template/iso Create a Template #Next, let\u0026rsquo;s create the template using the image we just uploaded. We\u0026rsquo;ll set several environment variables to make the process easier. Start by logging into your Proxmox server terminal and running the following commands as a privileged user:\nSet VM_STORAGE to your specific VM image storage, e.g. local-lvm or local-zfs. ## On the PVE Server ## # set VM variables export VM_ID=9009 export VM_NAME=\u0026#34;centos9\u0026#34; export VM_IMAGE=/var/lib/vz/template/iso/CentOS-Stream-GenericCloud-9-latest.x86_64.img export VM_STORAGE=\u0026#34;local-lvm\u0026#34; Run the following code to create a simple VM with the modified disk image attached:\n# create a new VM qm create \u0026#34;${VM_ID}\u0026#34; --name \u0026#34;${VM_NAME}\u0026#34; \\ --description \u0026#34;created on $(date)\u0026#34; \\ --ostype l26 \\ --bios seabios --machine q35 \\ --scsihw virtio-scsi-pci \\ --cpu cputype=host --memory 1024 \\ --net0 virtio,bridge=vmbr0,tag=1 # enable QEMU guest agent qm set \u0026#34;${VM_ID}\u0026#34; --agent enabled=1 # import the cloud image qm disk import \u0026#34;${VM_ID}\u0026#34; \u0026#34;${VM_IMAGE}\u0026#34; \u0026#34;${VM_STORAGE}\u0026#34; # attach the disk to the VM and set it as boot qm set \u0026#34;${VM_ID}\u0026#34; --boot order=scsi0 \\ --scsi0 \u0026#34;${VM_STORAGE}\u0026#34;:vm-\u0026#34;${VM_ID}\u0026#34;-disk-0,cache=writeback,discard=on,ssd=1 # add cloud-init drive qm set \u0026#34;${VM_ID}\u0026#34; --ide2 \u0026#34;${VM_STORAGE}\u0026#34;:cloudinit --ipconfig0 ip=dhcp \\ --citype nocloud Now, convert the VM to a template:\n# convert the VM into a template qm template \u0026#34;${VM_ID}\u0026#34; For additional configuration options see man qm or qm(1). For a bash script to simplify the above steps, checkout the build-template script on github or companion blog post. Test the Template #Create a new VM from your template:\n## On the PVE Server ## # create a clone and start it qm clone ${VM_ID} 100 --full --name ${VM_NAME}-test qm start 100 Because qemu-guest-agent is installed on the VM, you can use qm agent to get information on the running VM:\n# view network information qm agent 100 network-get-interfaces If you would like to SSH into the VM, add a SSH key to the VM cloud-init configuration using the CLI or web GUI, then reboot the VM.\nTo add a key using the CLI:\n## On the PVE Server ## root@pve:~# qm set 100 --sshkeys ~/.ssh/id_ed25519.pub root@pve:~# qm reboot 100 To add a key via the web GUI, navigate to the VM\u0026rsquo;s \u0026ldquo;Cloud-init\u0026rdquo; tab then click the \u0026ldquo;SSH public key\u0026rdquo; line and the \u0026ldquo;Edit\u0026rdquo; button. Then, reboot the VM.\nNow try SSHing into the VM using the default account (centos or cloud-user):\nssh cloud-user@\u0026lt;VM_IP_ADDRESS\u0026gt; Feel free to explore the VM, otherwise cleanup your environment by running:\n# stop and remove the VM root@pve:~# qm stop 100 root@pve:~# qm destroy 100 Recap #Overall, utilizing libguestfs offers a powerful and efficient approach for pre-boot customization of cloud images. While this approach is more manual than relying on cloud-init, it offers more flexibility in terms of customizing the OS bootloader. Just be aware anytime a libguestfs command outputs Setting the machine ID in /etc/machine-id, you must clear the ID or replace the value with uninitialized. For tips on building clean images, checkout Systemd: Building Images Safely and my other posts on Proxmox templates. Thanks for reading!\nAdditional Libguestfs Commands #Editing User Passwords #The SELECTOR field takes one of the password inputs (link).\n# change root password # syntax: virt-customize -a \u0026lt;IMAGE\u0026gt; --root-password \u0026lt;SELECTOR\u0026gt; sudo virt-customize -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 \\ --root-password password:SecretRootPassword Troubleshooting #Libguestfs: Error Status 1 #libguestfs: error: /usr/bin/supermin exited with error status 1. To see full error messages you may need to enable debugging. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run \u0026#39;libguestfs-test-tool\u0026#39; and post the *complete* output into a bug report or message to the libguestfs mailing list. This often occurs when you run libguestfs as a non-privileged user, try adding sudo to the command. Otherwise, add the LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 environment variables, run the command again and check the log output. Example output:\n# error when running as non privileged user libguestfs: trace: disk_create \u0026#34;/tmp/libguestfstmQ4KK/overlay1.qcow2\u0026#34; \u0026#34;qcow2\u0026#34; -1 \u0026#34;backingfile:/home/user/libguestfs-demo/ubuntu-20.04-server-cloudimg-amd64.img\u0026#34; ... cp: cannot open \u0026#39;/boot/vmlinuz-5.11.0-7620-generic\u0026#39; for reading: Permission denied supermin: cp -p \u0026#39;/boot/vmlinuz-5.11.0-7620-generic\u0026#39; \u0026#39;/var/tmp/.guestfs-1000/appliance.d.8d88y3ny/kernel\u0026#39;: command failed, see earlier errors libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above libguestfs: trace: launch = -1 (error) libguestfs: trace: close libguestfs: closing guestfs handle 0x60fa326dd050 (state 0) libguestfs: command: run: rm libguestfs: command: run: \\ -rf /tmp/libguestfstmQ4KK References #Cloud Images:\nOpenStack: Cloud Images, collection image links. CentOS Cloud Images Default User: centos or cloud-user Use CentOS-Stream-generic-x-latest.x86_64.qcow2 Debian Cloud Images Default User: debian Use debian-1x-generic-amd64.qcow2 Ubuntu Cloud Images Default User: ubuntu Use ubuntu-2x.04-server-cloudimg-amd64.img Cloud-init:\nCloud-init Documentation Cloud-init Docs: Config Examples Cloud-init Docs: Module Reference Proxmox:\nProxmox Docs: Cloud-Init FAQ Proxmox Docs: Cloud-Init Support Proxmox Docs: Storage Proxmox Docs: qemu-guest-agent Proxmox Template Tools:\nlibguestfs virt-cat virt-copy-out virt-copy-in virt-customize virt-edit virt-ls libguestfs recipes qemu-img Other:\nSystemd: Building Images Safely Debian Wiki: Machine-id ","date":"March 20, 2022","permalink":"https://www.trfore.com/posts/golden-images-and-proxmox-templates-using-libguestfs/","section":"Posts","summary":"Directly modify cloud image files with libguestfs","title":"Golden Images and Proxmox Templates Using libguestfs"},{"content":"Intro #In this tutorial, we\u0026rsquo;ll create a Proxmox template based on an Ubuntu cloud image and configured using cloud-init; with the goal of creating a cloud-init configuration file that works across Linux distributions and seamlessly integrates with Proxmox\u0026rsquo;s cloud-init tooling. Notably, this configuration file is easily modified without rebuilding the template. At the end, we will create two shell scripts that streamline the templating process by keeping cloud images up-to-date and creating a template with a simple CLI command. Thus, enabling the effortless generation of multiple templates and facilitating rapid deployment of virtual machines through the PVE GUI or your preferred provisioning tool.\nFull shell scripts, cloud-init files and systemd templates available at the GitHub repo (link). Cloud-init #Cloud-init is a configuration management tool used to automate the initialization of servers and cloud instances. It features a modular architecture, allowing the definition of various server configurations through modules. These modules handle tasks such as network configuration, user management, package installation, and file system initialization.\nConfiguration occurs during the final three stages of cloud-init\u0026rsquo;s boot process, with modules running once-per-instance or on every boot, always, depending on their predefined frequency. Modules are configured in user, network and vendor data files using YAML syntax, resulting in human-readable and reusable configuration files. Overall, cloud-init streamlines server provisioning and ensures consistent configuration across deployments. For a complete overview, see the official documentation (link).\nCreating a Universal Cloud-Init Configuration File #Proxmox cloud-init documentation demonstrates how to add a custom data file to cloud-init\u0026rsquo;s datasource, that is qm set 9000 --cicustom \u0026quot;user=local:snippets/userconfig.yaml\u0026quot;. However, attaching one automatically replaces the corresponding Proxmox generated file for meta, network, or user data. Using this approach typically leads people to creating a user data file and statically setting values for hostname, user and ssh_authorized_keys. Then generating multiple cloud-config files for each VM/template - this is unnecessary overhead!\nWe are going to capitalize on the fact that Proxmox does not generate a vendor data file, but allows you to attach one. We\u0026rsquo;ll place the majority of our own user data into a vendor data file, and Proxmox will merge it\u0026rsquo;s configuration with our own. This has the major benefit of preserving the dynamically generated values from Proxmox, e.g. VM name for the hostname, or other provisioning tools that use the Proxmox API.\nNote that Proxmox\u0026rsquo;s user data takes precedence over values placed into the vendor data file, as cloud-init merges user data over vendor data, so we\u0026rsquo;ll avoid duplicating configurations and discuss limitations of this approach in a later section.\nFirst, let\u0026rsquo;s look at the typical VM\u0026rsquo;s cloud-init settings that are configured using the GUI:\nThis is a mixture of user and network data. Let\u0026rsquo;s look specifically at the user data generated by Proxmox using the CLI:\nroot@pve:~# qm cloudinit dump 100 user #cloud-config hostname: vm-example manage_etc_hosts: true fqdn: vm-example.example.com user: jdoe password: \u0026lt;PASSWORD_HASH\u0026gt; ssh_authorized_keys: - ssh-ed25519 \u0026lt;PUBLIC_KEY\u0026gt; \u0026lt;USER_EMAIL\u0026gt; chpasswd: expire: False users: - default package_upgrade: true Here Proxmox is setting values for the hostname, etc hosts, users, passwords, ssh and packages modules. These settings will trigger cloud-init to do the following:\nSet hostname to vm-example.example.com. Add 127.0.1.1 vm-example.example.com to etc/hosts. Overwrite the default user with jdoe and add the hashed password in /etc/cloud/cloud.cfg. Allowing you to login via the PVE console with the updated username and password. Append the SSH key(s) to /home/\u0026lt;DEFAULT_USER\u0026gt;/.ssh/authorized_keys. Upgrade all packages using the OS package manager. So now that we have idea of the user data values Proxmox generates, let\u0026rsquo;s create a vendor data file that supplements these values.\nA Minimal Vendor Data File #At the bare minimum, Proxmox needs qemu-guest-agent installed on the image to execute commands and safely shutdown the VM. To install and update packages during the initialization process, we will use the packages module. Then, in order for Proxmox to register the guest agent, we\u0026rsquo;ll use the power state change module to reboot the VM after cloud-init has completely finished.\nWe\u0026rsquo;ll store our custom cloud-init files in the default Proxmox snippets directory, see storage for details.\nTo enable the snippets directory using the GUI, navigate to Datacenter \u0026gt; Storage \u0026gt; 'local' \u0026gt; Edit and highlight Snippets under the Content dropdown menu and click OK.\nAlternatively, append snippets to your Proxmox configuration file /etc/pve/storage.cfg:\ndir: local path /var/lib/vz content iso,backup,vztmpl,snippets ... Start by creating a basic vendor-data.yaml:\n# SSH into your Proxmox host user@desktop:~$ ssh root@pve # create a new vendor-data.yaml file root@pve:~# vi /var/lib/vz/snippets/vendor-data.yaml Add the following to vendor-data.yaml:\n#cloud-config packages: - qemu-guest-agent package_update: true power_state: mode: reboot timeout: 30 These settings do the following:\nInstall qemu-guest-agent using the default OS package manager. Update the package database \u0026amp; upgrade all installed packages (this is set in the Proxmox config). Reboot the VM 30 seconds after all cloud-init modules have finished. This configuration file can be easily modified without rebuilding the template, see the section limitations of using vendor data. The #cloud-config line is required, as it specifies the user data format. You must reboot the VM after first boot for PVE to recognize the guest agent. Next let\u0026rsquo;s create a simple VM template and attach this vendor file.\nProxmox Template #We\u0026rsquo;ll extend on Proxmox\u0026rsquo;s cloud-init docs example with details from the qm(1) manual page to create a Ubuntu VM template. Yet, we\u0026rsquo;ll focus on abstracting several of the CLI arguments to create a reusable shell script later.\nDownload a Cloud Image #Start by downloading a Ubuntu 20.04 cloud image on your proxmox host.\n# SSH into Proxmox user@desktop:~$ ssh root@pve # change into the ISO directory root@pve:~# cd /var/lib/vz/template/iso # download the image wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img # compare shasum curl -sfL https://cloud-images.ubuntu.com/releases/focal/release/SHA256SUMS | shasum -c --ignore-missing Create a Template #Next, set several environment variables to streamline the templating process. Note: Set the VM image storage value for your specific PVE host, for example local-lvm or local-zfs.\n## On the PVE Host ## # set VM variables export VM_ID=9000 export VM_NAME=\u0026#34;ubuntu20\u0026#34; export VM_IMAGE=/var/lib/vz/template/iso/ubuntu-20.04-server-cloudimg-amd64.img export VM_STORAGE=\u0026#34;local-lvm\u0026#34; For the template we will:\nEnable QEMU guest agent support using --agent enabled=1. Attach the Ubuntu cloud image using qm disk import and set it as boot. Attach our custom vendor data file with --cicustom. Paste the following commands into the terminal:\n# create a new VM qm create \u0026#34;${VM_ID}\u0026#34; --name \u0026#34;${VM_NAME}\u0026#34; \\ --description \u0026#34;created on $(date)\u0026#34; \\ --ostype l26 \\ --bios seabios --machine q35 \\ --scsihw virtio-scsi-pci \\ --cpu cputype=host --memory 1024 \\ --net0 virtio,bridge=vmbr0,tag=1 # enable QEMU guest agent qm set \u0026#34;${VM_ID}\u0026#34; --agent enabled=1 # import the cloud image qm disk import \u0026#34;${VM_ID}\u0026#34; \u0026#34;${VM_IMAGE}\u0026#34; \u0026#34;${VM_STORAGE}\u0026#34; # attach the disk to the VM and set it as boot qm set \u0026#34;${VM_ID}\u0026#34; --boot order=scsi0 \\ --scsi0 \u0026#34;${VM_STORAGE}\u0026#34;:vm-\u0026#34;${VM_ID}\u0026#34;-disk-0,cache=writeback,discard=on,ssd=1 # increase the disk image size qm resize \u0026#34;${VM_ID}\u0026#34; scsi0 +1G # add cloud-init drive qm set \u0026#34;${VM_ID}\u0026#34; --ide2 \u0026#34;${VM_STORAGE}\u0026#34;:cloudinit --ipconfig0 ip=dhcp \\ --citype nocloud # add cloud-init vendor config file qm set \u0026#34;${VM_ID}\u0026#34; --cicustom \u0026#34;vendor=local:snippets/vendor-data.yaml\u0026#34; # convert the VM into a template qm template \u0026#34;${VM_ID}\u0026#34; Test the Template #Feel free to test out the new template by creating a clone. Calling qm agent now works due to installing and enabling QEMU guest agent.\n## On the PVE Host ## # create a clone and start it qm clone $VM_ID 100 --full --name vm-example qm start 100 # wait for cloud-init to finish and reboot the VM ... # view network information qm agent 100 network-get-interfaces Creating Shell Scripts #Finally, let\u0026rsquo;s create two simple scripts to 1) download and/or check that a cloud image is up to date; and 2) create a template. Note: These examples are the foundational idea behind the full scripts that have more options, built-in error checking, and compatible with systemd timers. Grab the full scripts from the GitHub repo (link).\nAutomatically Update Cloud Images #Let\u0026rsquo;s create a simple script that will compare the shasum of our Ubuntu image, and download a new one if the file is out of date.\nCreate a new file basic-image-updater in /usr/local/bin/:\n# use your favorite editor root@pve:~# vi /usr/local/bin/basic-image-updater Add the following to it:\n#!/bin/bash FILE_NAME=\u0026#34;ubuntu-20.04-server-cloudimg-amd64.img\u0026#34; REMOTE_URL=\u0026#34;https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img\u0026#34; REMOTE_SHASUM_URL=\u0026#34;https://cloud-images.ubuntu.com/releases/focal/release/SHA256SUMS\u0026#34; STORAGE_PATH=/var/lib/vz/template/iso tmpfile=$(mktemp /tmp/image-shasum.XXXXXX) # change into PVE iso directory cd \u0026#34;${STORAGE_PATH}\u0026#34; || exit # get latest shasums curl -L \u0026#34;${REMOTE_SHASUM_URL}\u0026#34; -o \u0026#34;${tmpfile}\u0026#34; latest_shasum=$(grep \u0026#34;${FILE_NAME}\u0026#34; \u0026#34;${tmpfile}\u0026#34; | awk \u0026#39;{print $1}\u0026#39;) current_shasum=$(shasum -a 256 \u0026#34;${FILE_NAME}\u0026#34; | awk \u0026#39;{print $1}\u0026#39;) rm \u0026#34;$tmpfile\u0026#34; if [[ $latest_shasum == \u0026#34;${current_shasum}\u0026#34; ]]; then echo \u0026#34;SHASUM match, image is up-to-date.\u0026#34; else echo \u0026#34;No SHASUM match, downloading new image...\u0026#34; curl -sL \u0026#34;${REMOTE_URL}\u0026#34; -o \u0026#34;${FILE_NAME}\u0026#34; fi exit You can manually run this script by calling it from the CLI to download the missing image. Note: the majority of cloud images are only updated every 1 to 2 months.\nroot@pve:~# basic-image-updater shasum: ubuntu-20.04-server-cloudimg-amd64.img: No such file or directory No SHASUM match, downloading new image... Finally, let\u0026rsquo;s create a cron job to run this script monthly:\n# create/edit crontab crontab -e # add a cron job 0 2 10 * * /usr/local/bin/basic-image-updater Create Proxmox Template #Let\u0026rsquo;s create another simple script to streamline the templating process. We\u0026rsquo;ll create this file in /usr/local/bin/, so PVE users can call the script from any directory. Start by creating a file basic-template-builder:\n# use your favorite editor root@pve:~# vi /usr/local/bin/basic-template-builder Let\u0026rsquo;s extend the prior CLI commands into a basic script. Add the following to it and insert the PVE template \u0026lsquo;qm\u0026rsquo; commands from above:\n#!/bin/bash # use environment vars or default values VM_ID=${VM_ID:-\u0026#34;9000\u0026#34;} VM_NAME=${VM_NAME:-\u0026#34;template\u0026#34;} VM_IMAGE=${VM_IMAGE:-\u0026#34;\u0026#34;} VM_STORAGE=${VM_STORAGE:-\u0026#34;local-lvm\u0026#34;} # send error messages to STDERR function err() { echo \u0026#34;Error: $*\u0026#34; \u0026gt;\u0026amp;2 exit 2 } # simple error and help message function usage() { printf \u0026#34;Usage: %s --id \u0026lt;VM_ID\u0026gt; --name \u0026lt;VM_NAME\u0026gt; --img \u0026lt;VM_IMAGE\u0026gt; --storage \u0026lt;PVE_STORAGE\u0026gt; \\n\u0026#34; \u0026#34;${0##*/}\u0026#34; \u0026gt;\u0026amp;2 exit 2 } # parse CLI args for _ in \u0026#34;$@\u0026#34;; do [[ ${2} == -* ]] \u0026amp;\u0026amp; { err \u0026#34;Missing argument for ${1}\u0026#34; } case \u0026#34;${1}\u0026#34; in --help | -h) usage ;; --id) VM_ID=${2} shift 2 ;; --name) VM_NAME=${2} shift 2 ;; --img) VM_IMAGE=${2} shift 2 ;; --storage) VM_STORAGE=${2} shift 2 ;; -*) echo \u0026#34;Unknown Argument: $1\u0026#34; usage ;; esac done # check required variables if [ -z \u0026#34;${VM_ID}\u0026#34; ]; then err \u0026#34;Missing ID\u0026#34;; fi if [ -z \u0026#34;${VM_NAME}\u0026#34; ]; then err \u0026#34;Missing Name\u0026#34;; fi if [ ! -e \u0026#34;${VM_IMAGE}\u0026#34; ]; then err \u0026#34;Image file not found! ${VM_IMAGE}\u0026#34;; fi ### add PVE template \u0026#39;qm\u0026#39; commands from above ### exit Test it out.\n# using CLI args basic-template-builder --id 9000 --name ubuntu20 --img /var/lib/vz/template/iso/ubuntu-20.04-server-cloudimg-amd64.img # using environment vars export VM_ID=9000 VM_NAME=\u0026#34;ubuntu20\u0026#34; VM_STORAGE=\u0026#34;local-lvm\u0026#34; export VM_IMAGE=/var/lib/vz/template/iso/ubuntu-20.04-server-cloudimg-amd64.img basic-template-builder Recap #Overall, we created a cloud-init configuration file to supplement the dynamically generated values from Proxmox and a template that utilizes Proxmox\u0026rsquo;s user data alongside our own user data, disguised as a vendor data file. We also created two shell scripts to keep the Ubuntu cloud image up-to-date and facilitate template creation using a simple CLI command, basic-template-builder --id \u0026lt;VM_ID\u0026gt; --name \u0026lt;VM_NAME\u0026gt; --img \u0026lt;VM_IMAGE\u0026gt;.\nI personally prefer this approach over others that I have written about, as it keeps your templates modular and allows updating the vendor data file after template creation. Additionally, these templates work with external provisioning tools, e.g. Terraform, by utilizing the automatically generated VM names and VM IDs. Checkout the blog post: Provisioning Proxmox VMs with Terraform (link) for details on quickly deploying these templates with Terraform.\nAt this point, feel free to explore templating with cloud-init on your own. Yet, there are a few other points worth discussing.\nLimitations of Using Vendor Data # You cannot add additional users or SSH keys using the users module, as user data generated by PVE takes precedence over vendor data.\nModifying the custom vendor data file after creating a template will apply the updated configuration to all future clones. To apply the changes to running VMs, run qm cloudinit update \u0026lt;vmid\u0026gt; or click the Regenerate Image button in the PVE GUI to manually regenerate the cloud-init config drive - as Proxmox does not monitor changes to snippets directory and will not automatically regenerate the drive.\nTo view the config drive, read the contents of /dev/sr0/ on a running VM. Modifying the custom vendor data file on a running VM does not change the instance-id for the VM, so cloud-init will not re-run once-per-instance modules. Conversely, modules that always run will apply changes at boot.\nIf you rename the vendor file, you must update the template configuration by running the CLI command below. Additionally, all pre-existing clones must be updated as well, otherwise Proxmox blocks the VM from starting and throws a TASK ERROR: volume 'local:snippets/vendor-data.yaml' does not exist.\n# update template to use renamed file qm set $VM_ID --cicustom \u0026#34;vendor=local:snippets/\u0026lt;NEW_FILENAME\u0026gt;.yaml\u0026#34; Regenerating the Config Drive vs Re-running Cloud-init #There are two important and distinct steps in Proxmox\u0026rsquo;s implementation of cloud-init, one is creating/updating the instance-id for cloud-init, and another is generating a cloud-init config drive.\nAnytime a new VM is created or cloned, Proxmox generates cloud-init meta data with a instance-id that is unique to each VM. Cloud-init tracks this instance-id and determines whether to run once-per-instance modules.\nProxmox also generates a cloud-init config drive that is unique for each VM and attaches it as a cd-rom drive mounted at /dev/sr0. You can manually regenerate the drive by running qm cloudinit update \u0026lt;vmid\u0026gt; or using the Regenerate Image button in the PVE GUI. However, regenerating the drive does not change the instance-id, thus, cloud-init will not re-run once-per-instance modules.\nOne method for changing the instance-id is to modify the cloud-init data within the Proxmox GUI.\nUpdating Cloud-init Data in the PVE GUI #Proxmox actively monitors for changes to user and network data. Updating cloud-init settings in the GUI triggers Proxmox to update the cloud-init meta data with a new instance-id and regenerate the cloud-init config drive. On the following boot, cloud-init regenerates SSH keys and creates new folder /var/lib/cloud/instances/\u0026lt;instance-id\u0026gt;, symlinking /var/lib/cloud/instances/ to it.\n# view the updated instance-id root@pve:~# qm cloudinit dump 100 meta instance-id: ab42f82195a1a9315a0b90295b8ed5ee24d251f1 # ssh into the VM root@pve:~# ssh root@vm-example # view the cloud-init configuration root@vm-example:~# tree /var/lib/cloud/ /var/lib/cloud/ ├── data │ ├── instance-id │ ├── previous-datasource │ ├── previous-hostname │ ├── previous-instance-id │ ├── python-version │ ├── result.json │ ├── set-hostname │ └── status.json ├── handlers ├── instance -\u0026gt; /var/lib/cloud/instances/ab42f82195a1a9315a0b90295b8ed5ee24d251f1 ├── instances │ ├── ab42f82195a1a9315a0b90295b8ed5ee24d251f1 │ │ ├── boot-finished │ │ ├── cloud-config.txt ... │ └── f36c5e1ee9827152d057b81eb8ec65e8769f3510 │ ├── cloud-config.txt ... Proxmox calculates a SHA1 hash from user and network data to create a unique instance-id. Cloud-init will not re-run if the instance-id is already present in /var/lib/cloud/instances. This occurs when user or network data is modified, cloud-init re-initializes on the change, then the data is reverted and the VM is rebooted. Cloud-init will update the symlink to the original id folder and keep the previous SSH keys. Additionally, cloud-init creates the file /var/lib/cloud/instances/boot-finished with the updated date, cloud-init version (21.4) and distribution release (Ubuntu 20.04.1).\nroot@vm-example:~# cat /var/lib/cloud/instance/boot-finished 10.63 - Sat, 12 Mar 2022 17:53:33 +0000 - v. 21.4-0ubuntu1~20.04.1 If changes are made to the user and the image has not been initialized then the user value becomes the VM\u0026rsquo;s default user. If the image was already initialized then a new user is created with a new uid and home directory.\n# view changes to running VM root@pve:~# qm cloudinit pending 100 cur cicustom: vendor=local:snippets/vendor-data.yaml cur citype: nocloud new ciuser: jdoe ## reboot the VM ## # ssh into the VM root@pve:~# ssh root@vm-example # list users root@vm-example:~# tail -n 2 /etc/passwd ubuntu:x :1000:1000:Ubuntu:/home/ubuntu:/bin/bash jdoe:x :1001:1001:Ubuntu:/home/jdoe:/bin/bash Working with qcow2 and raw Images #In this post, we used one of the easier images to work with as Ubuntu provides cloud images in the raw format with the *.img extension. Other Linux distributions provide cloud images with *.qcow2 and *.raw extensions. You can use these images to create Proxmox templates and store them in the default folder /var/lib/vz/templates/iso, however, Proxmox\u0026rsquo;s GUI will mask these files, as it only displays *.img and *.iso files. A easy workaround is to change the file extension to *.img, as Proxmox qm disk import will convert image formats supported by qemu-img. With this workaround, you can use qemu-img info to view the actual format and size of the image.\n# change the extension mv debian-11-generic-amd64.{qcow2,img} # inspect the image root@pve:~# qemu-img info debian-11-generic-amd64.img image: debian-11-generic-amd64.img file format: qcow2 virtual size: 2 GiB (2147483648 bytes) disk size: 340 MiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false extended l2: false Child node \u0026#39;/file\u0026#39;: filename: debian-11-generic-amd64.img protocol type: file file length: 340 MiB (356127744 bytes) disk size: 340 MiB This is what happens under the hood in the full scripts (link). If you would like to manually convert *.qcow2 to *.img, you can use the following commands on a linux desktop or PVE host:\n# install the qemu package (pre-installed on Proxmox) sudo apt install qemu # convert the image qemu-img convert -f qcow2 -O raw image.qcow2 image.img For *.raw to *.img just change the extension:\n# download the raw or tar file wget \u0026lt;FILE_URL\u0026gt; # extract the tar tar -xvf \u0026lt;FILE\u0026gt; # change the file extension mv disk.{raw,img} Here are a few tips when working with these formats:\nFor *.qcow2 images, the SHASUM of a converted unmodified image will match the values of the remote *.raw image. For *.raw images, if you wish to save on bandwidth download the *.tar archive. Archives typically extract to a file named disk.raw. Common Bugs #CentOS: Probing EDD #In CentOS, the boot will appear to hang and you\u0026rsquo;ll see the following message in the console:\nProbing EDD (edd=off to disable)... ok There are two causes for this, 1) the bootloader sets console to serial, console=ttyS0,115200n8, and the console output is masked in PVE GUI; 2) Enhanced Disk Drive Services (EDD) is enabled. Typically, you can ignore this message, and cloud-init will run then reboot the machine.\nIf you wish to modify the bootloader on the image, checkout the post on using libguestfs.\nCentOS: UEFI and Secure Boot #Use seabios with CentOS, as it requires additional configuration to get secure boot to work with UEFI. I plan to write-up the fix at a later time.\nAcknowledgements #Thanks to all the users in this forum post and the user(s) who pushed the patch to permit attaching vendor data to VMs! And to you for reading this far!\nReferences #Cloud-init:\ncloud-init user data network data vendor data cloud-init docs: Config Examples cloud-init docs: Module Reference Ubuntu Docs: How to use cloud-init Ubuntu Docs: Automated Server Installation Cloud-init Modules: apt, bootcmd, etc hosts, hostname, passwords, packages, power state change, ssh, users\nProxmox:\nProxmox Wiki qemu-guest-agent Cloud-Init Support Cloud-Init FAQ Proxmox Forum: Combining custom cloud init with auto-generated Proxmox Storage:\nProxmox Docs: Performance Tweaks Proxmox Forum: VirtIO vs SCSI Linux Cloud Images:\nOpenStack: Cloud Images, collection of image links. CentOS Cloud Images Default User: centos or cloud-user Use CentOS-Stream-GenericCloud-X-latest.x86_64.qcow2 Debian Cloud Images Default User: debian Use debian-1x-generic-amd64.qcow2, .raw or .tar.zx Avoid: genericcloud images fail to run cloud-init nocloud images do not have cloud-init installed and defaults to a password-less root user. Fedora Cloud Images Default User: fedora Use Fedora-Cloud-Base-Generic.x86_64-XX-XX.qcow2 Ubuntu Cloud Images Default User: ubuntu Use ubuntu-2x.04-server-cloudimg-amd64.img For qcow2 and raw images see the section: Working with qcow2 and raw Images ","date":"March 13, 2022","permalink":"https://www.trfore.com/posts/golden-images-and-proxmox-templates-using-cloud-init/","section":"Posts","summary":"Discover the advantage of utilizing cloud-init vendor data when creating Proxmox templates","title":"Golden Images and Proxmox Templates Using cloud-init"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/docker/","section":"Tags","summary":"","title":"Docker"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/minio/","section":"Tags","summary":"","title":"Minio"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/postgresql/","section":"Tags","summary":"","title":"Postgresql"},{"content":"Intro #As discussed in the previous post, storing Terraform state in a secure remote backend is important as it typically contains sensitive data, such as database credentials. Additionally, using a backend that supports state locking is important in multi-user environments to ensure that only one user can modify the state at any given time. In this post we will demonstrate using MinIO or PostgreSQL with Terraform to securely store the state within your local environment. To keep things simple we will use docker containers to test each backend, but with the intent of moving to a more production-ready solution in the future.\nFor this tutorial, you need to have Terraform and Docker installed on your local desktop, along with access to a Proxmox server. We\u0026rsquo;ll provision a simple VM to generate a Terraform state file, example code is available below and the backend configuration is available under each section, MinIO or Postgres. You will also need a Proxmox template, ubuntu20, to deploy the VM - see below for creating a simple template.\nSetting up a permanent MinIO or PostgreSQL server in Proxmox is beyond the scope of this post. However, 3rd party solutions exist for deploying MinIO on NAS servers, e.g. TrueNAS and Synology, and the web is full of tutorials on deploying PostgreSQL in LXC containers. MinIO #MinIO is a highly scalable S3-compatible object store with a RESTful API. It\u0026rsquo;s open source and licensed under AGPLv3 with the option for commercial licensing. While it is designed as a distributed storage solution, a single disk deployment is possible. As a Terraform backend, MinIO supports versioning and object locking which is useful for tracking and reviewing changes to Terraform state. While it does not support state locking, MinIO\u0026rsquo;s intuitive user-interface is a great way explore using remote backends.\nMinIO Docker Image # Since the original post, MINIO_ACCESS_KEY and MINIO_SECRET_KEY have been deprecated in favor of MINIO_ROOT_USER and MINIO_ROOT_PASSWORD. Additionally, versioning and object locking are available for single disk deployments. For this example, we\u0026rsquo;ll use the Bitnami MinIO image and set the root user access using the environment variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD. Additionally, we\u0026rsquo;ll set the region to us-east-1 and expose the container locally on ports 9000 and 9001. Run the following command to start the container:\ndocker run -d --name minio-server \\ --env MINIO_ROOT_USER=\u0026#34;root\u0026#34; \\ --env MINIO_ROOT_PASSWORD=\u0026#34;rootsecret\u0026#34; \\ --env MINIO_REGION=\u0026#34;us-east-1\u0026#34; \\ --publish 9000:9000 \\ --publish 9001:9001 \\ bitnami/minio:latest Next, create a bucket named staging with versioning and object locking enabled:\n# create bucket with object locking enabled docker exec minio-server mc mb --with-lock local/staging # enable versioning docker exec minio-server mc version enable local/staging # enable 30 day retention policy docker exec minio-server mc retention set --default compliance 30d local/staging You can confirm the server is running and view all available buckets by using the MinIO client, mc:\n$ docker exec minio-server mc ls local [2022-02-20 17:26:06 UTC] 0B staging/ Generating Access Keys #While you can use the root username and password to access the MinIO server, let\u0026rsquo;s create an additional user with read and write permissions for Terraform to use. For simplicity, set the username to terraform and password to terraformsecret. Feel free to generate a new access key using the CLI or GUI directions below.\nUsing the CLI #To create an access key using the MinIO client, run:\n# create a new user docker exec minio-server mc admin user add local terraform terraformsecret # add permissions to the user docker exec minio-server mc admin policy attach local readwrite --user terraform For MinIO versions prior to v2023-03-20, you\u0026rsquo;ll need to use the following command to set the permissions:\n# add permissions to the user docker exec minio-server mc admin policy set local readwrite user=terraform Using the GUI #To create an access key using the MinIO server\u0026rsquo;s web interface, open your browser at http://localhost:9001 and login using the root credentials from MINIO_ROOT_USER and MINIO_ROOT_PASSWORD. Note: If you didn\u0026rsquo;t set the root credentials using environment variables from above, the default values are minio and miniosecret.\nUnder the Identity menu, click Users. Click on the Create User button and fill in the form with the username, terraform, and password, terraformsecret. Check the box for readwrite permissions, and click Save.\nConfiguring Terraform To Use MinIO #Now that we have a MinIO server running in Docker, let\u0026rsquo;s configure Terraform to use it. Using the example code below, add the following s3 backend configuration to the terraform block:\n# main.tf terraform { # PROVIDER CONFIGURATION FROM BELOW backend \u0026#34;s3\u0026#34; { bucket = \u0026#34;staging\u0026#34; key = \u0026#34;terraform.tfstate\u0026#34; endpoint = \u0026#34;http://localhost:9000\u0026#34; region = \u0026#34;us-east-1\u0026#34; access_key = \u0026#34;terraform\u0026#34; secret_key = \u0026#34;terraformsecret\u0026#34; force_path_style = true skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true # required with Terraform v1.6+ skip_region_validation = true } } Go ahead and run terraform apply to build the example VM and generate the initial state file. Terraform will create a new terraform.tfstate file inside the staging bucket, let\u0026rsquo;s take a look at it using the MinIO client head command:\n# view the terraform state file in MinIO $ docker exec minio-server mc head local/staging/terraform.tfstate { \u0026#34;version\u0026#34;: 4, \u0026#34;terraform_version\u0026#34;: \u0026#34;1.1.5\u0026#34;, \u0026#34;serial\u0026#34;: 2, \u0026#34;lineage\u0026#34;: \u0026#34;87f396c1-0aea-4493-f52c-eb3cd0a18b58\u0026#34;, \u0026#34;outputs\u0026#34;: { \u0026#34;id\u0026#34;: { \u0026#34;value\u0026#34;: \u0026#34;pve/qemu/100\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; }, You can also Preview the state file from the MinIO GUI:\nWith versioning and object locking enabled, try changing vcpu count to 2 in the main.tf file and run terraform apply again. Within the GUI, you should see a new version of the state file.\nAfter you are done exploring, go ahead and destroy the VM and remove the MinIO server:\n# destroy the VM terraform destroy # stop the MinIO server docker stop minio-server \u0026amp;\u0026amp; docker rm minio-server Limitations of Using MinIO #Not all the AWS S3 configuration options, S3 backend documentation, are supported by MinIO. Notability, state locking via a DynamoDB-like database is not supported, thus this approach is not suitable for multi-user environments.\nPostreSQL #PostgreSQL is an open source relational database management system (RDBMS) that supports standard SQL language. As a Terraform backend, it supports state locking but not versioning and object locking. Directly viewing the state information is possible with simple SQL queries to the database, along with calls to terraform state \u0026lt;pull/show\u0026gt;.\nPostgreSQL Docker Image #We\u0026rsquo;ll use the offical Postgres image to create a server with the default user postgres having the password password123, and a new terraform database. Additionally, we\u0026rsquo;ll expose the container on port 5432 on the host machine.\ndocker run -d --name postgres \\ --user postgres \\ --env POSTGRES_PASSWORD=password123 \\ --env POSTGRES_DB=terraform \\ --publish 5432:5432 \\ postgres:latest Configuring Terraform To Use PostgreSQL #Using the example code below, add the following pg backend configuration to the terraform block:\n# main.tf terraform { # PROVIDER CONFIGURATION FROM BELOW backend \u0026#34;pg\u0026#34; { conn_str = \u0026#34;postgres://postgres:password123@localhost:5432/terraform?sslmode=disable\u0026#34; } } Adding sslmode=disable to the connection string will disable SSL connections and avoid the error pq: SSL is not enabled on the server. After running terraform init, Terraform will create a new schema terraform_remote_state inside the database. You can configure the schema name by setting schema_name in the backend block. For additional configuration options see pg backend documentation.\nGo ahead and run terraform apply to build the example VM and generate the initial state information. Let\u0026rsquo;s take a look inside the database, the state information will be stored in the terraform database under the terraform_remote_state schema in the state table, use the following commands to view it:\n# connect to the container docker exec -it postgres psql Now, using psql commands inside the container:\n-- list the databases postgres=# \\l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+------------+------------+----------------------- postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres + | | | | | postgres=CTc/postgres terraform | postgres | UTF8 | en_US.utf8 | en_US.utf8 | (4 rows) -- connect to the terraform database postgres=# \\c terraform -- list the columns in the table terraform=# \\d terraform_remote_state.states; Table \u0026#34;terraform_remote_state.states\u0026#34; Column | Type | Collation | Nullable | Default --------+--------+-----------+----------+------------------------------------------- id | bigint | | not null | nextval(\u0026#39;global_states_id_seq\u0026#39;::regclass) name | text | | | data | text | | | Indexes: \u0026#34;states_pkey\u0026#34; PRIMARY KEY, btree (id) \u0026#34;states_by_name\u0026#34; UNIQUE, btree (name) \u0026#34;states_name_key\u0026#34; UNIQUE CONSTRAINT, btree (name) -- view the terraform state information terraform=# SELECT \u0026#34;data\u0026#34; FROM terraform_remote_state.states; data ------------------------------------------------------------------ ... State Locking in PostgreSQL #One of the advantages of using PostgreSQL as a backend is that it provides state locking, ensuring only a single user can modify state at a time. You can view this in action by querying the pg_locks table while running Terraform. For example, run the following query in a separate terminal window while running terraform apply:\npostgres=# SELECT * FROM pg_locks; Example output:\nlocktype | database |...| classid | objid | objsubid | virtualtransaction | pid | mode | granted |...| ------------+----------+---+---------+-------+----------+--------------------+-----+-----------------+---------+---+ advisory | 16384 |...| 0 | 1 | 1 | 3/0 | 96 | ExclusiveLock | t |...| You will see a transient advisory lock appearing in the pg_locks table. This lock is removed when Terraform completes its run. To confirm the lock is for the terraform database, try running the following query while Terraform is processing state changes:\nSELECT pl.database, pl.locktype, pl.mode, psa.datid, psa.datname FROM pg_locks pl JOIN pg_stat_activity psa ON pl.database = psa.datid WHERE psa.datname = \u0026#39;terraform\u0026#39;; It should return a similar result:\ndatabase | locktype | mode | datid | datname ----------+----------+---------------+-------+----------- 16384 | advisory | ExclusiveLock | 16384 | terraform (1 row) Cleaning Up #Once you are finished, clean-up your environment by running the following:\n# destroy the VM terraform destroy # stop \u0026amp; remove the Postgres container docker stop postgres \u0026amp;\u0026amp; docker rm postgres Recap #Overall, we tested two different backends for storing terraform state. Using MinIO as a S3 backend, we were able to store state in a environment specific manner using pre-configured buckets, for example staging. Additionally, with versioning and object locking enabled on the bucket, we retained the history of all terraform runs and could easily view and rollback to a specific version using the MinIO GUI. While MinIO does not offer state locking, you do have the ability to encrypt data at rest. If you are interested in adding bucket encryption see MinIO Docs: Data Encryption (SSE) for more details.\nWith Postgres, we gained state locking, which is great for multi-user environments. In this post, we hard-coded the database username and password into connection parameter in the main.tf file, however, this is not a good practice for production environments. Instead, consider using environment variables to pass in the database credentials via PGUSER \u0026amp; PGPASSWORD. If state locking is a requirement for your local environment, also consider using the Kubernetes backend, as it allows you to store state inside the cluster\u0026rsquo;s etcd store and supports state locking.\nHopefully, you’ve found this exploration of remote backends useful. Thanks for reading!\nFull Code #Template Code #For more elaborate templates, checkout the post: Golden Images and Proxmox Templates Using cloud-init\n# SSH into your PVE server user@desktop:~$ ssh root@pve Use the following commands to create a new template:\n# change into the ISO directory cd /var/lib/vz/template/iso # download the image wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img # create a virtual machine qm create 9000 --name ubuntu20 --machine q35 --scsihw virtio-scsi-pci # import the ubuntu image as the root disk qm disk import 9000 ubuntu-20.04-server-cloudimg-amd64.img local-lvm # attach and set the root disk as boot drive qm set 9000 --boot order=scsi0 --scsi0 \u0026#34;local-lvm:vm-9000-disk-0\u0026#34; # add cloud-init config drive qm set 9000 --ide2 local-lvm:cloudinit --citype nocloud # convert the vm to a template qm template 9000 Terraform File #Telmate Proxmox Provider #Telmate Proxmox v2.9 only works with Proxmox v7. For Proxmox v8 support, use the BPG Proxmox provider and code below.\n# main.tf terraform { required_providers { proxmox = { source = \u0026#34;Telmate/proxmox\u0026#34; version = \u0026#34;~\u0026gt; 2.9.0\u0026#34; } } # ADD BACKEND CONFIGURATION HERE } provider \u0026#34;proxmox\u0026#34; { pm_api_url = \u0026#34;https://pve.example.com/api2/json\u0026#34; pm_api_token_id = \u0026#34;terraform@pve!token\u0026#34; pm_api_token_secret = \u0026#34;\u0026lt;TOKEN_VALUE\u0026gt;\u0026#34; } resource \u0026#34;proxmox_vm_qemu\u0026#34; \u0026#34;proxmox_vm\u0026#34; { target_node = \u0026#34;pve\u0026#34; vmid = 100 name = \u0026#34;vm-example\u0026#34; clone = \u0026#34;ubuntu20\u0026#34; full_clone = true os_type = \u0026#34;cloud-init\u0026#34; cores = \u0026#34;1\u0026#34; memory = \u0026#34;1024\u0026#34; agent = 1 disk { type = \u0026#34;scsi\u0026#34; slot = 0 storage = \u0026#34;local-lvm\u0026#34; size = \u0026#34;8G\u0026#34; format = \u0026#34;raw\u0026#34; cache = \u0026#34;writeback\u0026#34; iothread = 0 ssd = 1 discard = \u0026#34;ignore\u0026#34; } scsihw = \u0026#34;virtio-scsi-pci\u0026#34; bootdisk = \u0026#34;scsi0\u0026#34; network { model = \u0026#34;virtio\u0026#34; bridge = \u0026#34;vmbr0\u0026#34; tag = \u0026#34;1\u0026#34; } # cloud-init settings sshkeys = file(\u0026#34;/home/$USER/.ssh/id_ed25519.pub\u0026#34;) ipconfig0 = \u0026#34;ip=192.168.1.100/24,gw=192.168.1.1\u0026#34; } BPG Proxmox Provider #Works with Proxmox VE v8, see BPG Proxmox for additional configuration options.\n# main.tf terraform { required_providers { proxmox = { source = \u0026#34;bpg/proxmox\u0026#34; version = \u0026#34;\u0026gt;=0.39.0\u0026#34; } } # ADD BACKEND CONFIGURATION HERE } provider \u0026#34;proxmox\u0026#34; { endpoint = \u0026#34;https://pve.example.com/api2/json\u0026#34; api_token = \u0026#34;terraform@pve!token=\u0026lt;TOKEN_VALUE\u0026gt;\u0026#34; } resource \u0026#34;proxmox_virtual_environment_vm\u0026#34; \u0026#34;vm\u0026#34; { node_name = \u0026#34;pve\u0026#34; vm_id = 100 name = \u0026#34;vm-example\u0026#34; agent { enabled = true } clone { vm_id = 9000 full = true } cpu { cores = 1 type = \u0026#34;host\u0026#34; } memory { dedicated = 1024 floating = 1024 } network_device { bridge = \u0026#34;vmbr0\u0026#34; vlan_id = \u0026#34;1\u0026#34; } disk { datastore_id = \u0026#34;local-lvm\u0026#34; interface = \u0026#34;scsi0\u0026#34; size = 8 file_format = \u0026#34;raw\u0026#34; cache = \u0026#34;writeback\u0026#34; iothread = false ssd = true discard = \u0026#34;on\u0026#34; } # cloud-init settings initialization { user_account { keys = [file(\u0026#34;~/.ssh/id_ed25519.pub\u0026#34;)] } ip_config { ipv4 { address = \u0026#34;192.168.1.100/24\u0026#34; gateway = \u0026#34;192.168.1.1\u0026#34; } } } # Cloud-init SSH keys will cause a forced replacement, see: # https://github.com/bpg/terraform-provider-proxmox/issues/373 lifecycle { ignore_changes = [initialization[\u0026#34;user_account\u0026#34;], ] } } Troubleshooting #Errors Using MinIO #Initializing the backend... Error refreshing state: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. This error will occur when running terraform init and is caused by an invalid MinIO access key. Confirm that the access key is correct in the GUI and try again.\nError: Retrieving AWS account details: AWS account ID not previously found and failed retrieving via all available methods. This is caused by Terraform attempting to contact the AWS IAM API to pull an account id, starting with Terraform v1.6+ you will need to set skip_requesting_account_id = true in your s3 backend config.\nmc: \u0026lt;ERROR\u0026gt; Unable to make bucket `local/staging`. Access Denied. mc: \u0026lt;ERROR\u0026gt; Unable to enable versioning. Access Denied. mc: \u0026lt;ERROR\u0026gt; Remote bucket `local/staging` does not support locking If you copied the MinIO bucket code from this post, a new-line character may have been added at the end of code in some terminals. Delete it and re-run the commands.\nErrors Using Postgres #pq: SSL is not enabled on the server This error is due to a missing SSL certificate on the Postgres server. Add sslmode=disable to the connection string, conn_str, to skip the SSL check and connect via HTTP. Alternatively, you can configure PostgreSQL to use TLS see this GitHub Gist and Postgres Docs: SSL Support for more information.\nReferences #Terraform:\nTerraform Terraform State Terraform Documentation State Locking Terraform Docs: Postgres Backend Terraform Docs: S3 Backend DynamoDB State Locking MinIO:\nMinIO MinIO Documentation MinIO Docs: User Management MinIO Docs: Bucket Versioning MinIO Docs: MinIO Object Locking MinIO Docs: Data Encryption (SSE) MinIO Docs: Network Encryption (TLS) MinIO Client Docker Hub: minio/minio Docker Hub: bitnami/minio PostgreSQL:\nPostgreSQL PostgreSQL Documentation Postgres Docs: Advisory Locks Postgres Docs: pg_locks Postgres Docs: SSL Support Docker Hub: postgresql/postgres ","date":"February 21, 2022","permalink":"https://www.trfore.com/posts/storing-terraform-state-in-minio-and-postgresql/","section":"Posts","summary":"Secure your Terraform state files locally using MinIO or PostgreSQL","title":"Storing Terraform State in MinIO and PostgreSQL"},{"content":" Development on the Telmate plugin is currently in flux, as Telmate is no longer in business. Version \u0026lt;= v2.9 are incompatible with Proxmox 8, so you\u0026rsquo;ll need to use v3 - see the updated post for Telmate PVE 8 support. Also, I\u0026rsquo;ve switched to using BPG provider, see the latest post on using Terraform BPG Provider with Proxmox 8. Intro #In this tutorial, we’ll explore using Terraform to provision virtual machines on Proxmox. We\u0026rsquo;ll cover granting Terraform limited access to the Proxmox API, then create a reusable module for the Telmate Proxmox provider plugin and use it to create a three node K3s cluster. If you are familiar with Terraform, feel free to skip the background section and jump straight into creating the module.\nFull code available at the GitHub repo (link) Terraform #Terraform is an open-source infrastructure as code (IaC) tool that allows you to define, provision, and manage infrastructure resources across various cloud platforms and on-premises environments. It uses a declarative configuration language, HashiCorp Configuration Language (HCL), to define the desired state of your infrastructure. One of the key differences to other IaC tools is that Terraform stores infrastructure state in a single file, terraform.tfstate, allowing you to use version control systems to track changes, revert to previous states, and maintain synchronization of environments between end-users.\nTerraform consist of two parts: core and plugins. Core is the main program, written in Go, that is responsible for reading configuration files and modules; creating resource graphs and plans; managing state; and communicating with plugins via remote procedure calls. While, plugins are Go binaries that extend the functionality of the core tool by adding new providers and provisioners. You can find plugins for all the major cloud-providers and hypervisors in the Terraform Registry. There are several plugins for Proxmox, however, we are going to focus on using Telmate/Terraform-Provider-Proxmox for this tutorial.\nState Storage #Terraform state is a crucial component of infrastructure management, as Terraform uses it to determine the changes required to reach the desired state. The state file should be stored using a secure remote backend, as it may contain sensitive data, such as database credentials. Additionally, consider using one that supports state locking, to ensure that only one user can modify the state at any given time.\nFor this post, we\u0026rsquo;ll use the default local backend, keeping the state file on the local filesystem and focus on provisioning. However, this is not recommended for production environments as it uses JSON format and stores the majority of information in plain text. If you interested in storing state securely and locally, see the post: Storing Terraform State in MinIO and PostgreSQL.\nTerraform Modules #A module is simply a collection of Terraform configuration files in a single directory. Now taking this idea further, it allows you to organize your configuration files into logical groupings to: 1) increase code-reuse and reduce duplication across your codebase; 2) encapsulate complex infrastructure configurations; and 3) share configurations across team members and the broader community.\nFor this post, we\u0026rsquo;ll create a complex folder structure that mimics one you might see in a production environment, but rather than creating folders for different environments (e.g., dev, staging, prod), we\u0026rsquo;ll just use a single homelab/ folder to store our K3s cluster configuration. Moreover, we\u0026rsquo;ll use the standard structure for creating local modules and include a examples/ folder along with the modules/ folder.\nterraform-telmate-proxmox/ ├── examples │ ├── lxc │ └── vm │ ├── main.tf │ └── variables.tf ├── homelab │ └── k3s-cluster │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── modules │ ├── lxc │ └── vm │ ├── main.tf │ ├── outputs.tf │ ├── README.md │ └── variables.tf Now before we get into writing the module, it\u0026rsquo;s important to cover what a module should accomplish. General advice is that modules should increase the abstraction of your infrastructure configuration, rather than just a simple wrapper for a single resource. Typically example of this might be a module that creates a virtual network consisting of multiple resources, for example: firewall, load balancer, and several subnets.\nIn our case we\u0026rsquo;ll create a module that slightly goes against this convention, as the majority of Proxmox workflows consist of creating a VM or LXC container and the provider only exposes these two resources. Yet, we\u0026rsquo;ll still encapsulate the complexity of creating a virtual machine and redefine several of the default values from the provider to suit our infrastructure requirements. Additionally, it will handle the conditional addition of several attributes with complex values depending on the value of select variables.\nGrant Terraform Access to Proxmox #Telmate plugin works by making API request to the Proxmox Virtual Environment (PVE) endpoint. For this tutorial we\u0026rsquo;ll use the Proxmox CLI to create a new role, group with permissions, and user; then generate an access token for the terraform user. We\u0026rsquo;ll also only allow permissions for specific paths in Proxmox, /storage and /vms. For additional configuration options see Proxmox Wiki: User Management or pveum help.\nFrom the PVE CLI enter the following commands:\n# create role pveum role add TerraformUser -privs \u0026#34;Datastore.AllocateSpace \\ Datastore.Audit VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \\ VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk \\ VM.Config.HWType VM.Config.Memory VM.Config.Network \\ VM.Config.Options VM.Migrate VM.Monitor VM.PowerMgmt\u0026#34; # create group pveum group add terraform-users # add permissions pveum acl modify /storage -group terraform-users -role TerraformUser pveum acl modify /vms -group terraform-users -role TerraformUser # create user \u0026#39;terraform\u0026#39; pveum useradd terraform@pve -groups terraform-users # generate a token pveum user token add terraform@pve token -privsep 0 The last command will output a token value similar to the following, which we will use to validate Terraform with Proxmox during the final provisioning step.\n┌──────────────┬──────────────────────────────────────┐ │ key │ value │ ╞══════════════╪══════════════════════════════════════╡ │ full-tokenid │ terraform@pve!token │ ├──────────────┼──────────────────────────────────────┤ │ info │ {\u0026#34;privsep\u0026#34;:\u0026#34;0\u0026#34;} │ ├──────────────┼──────────────────────────────────────┤ │ value │ 782a7700-4010-4802-8f4d-820f1b226850 │ └──────────────┴──────────────────────────────────────┘ Creating a Terraform Module #In this tutorial, we will create a single module, vm. The following sections discuss the logic behind each file and Terraform code using snippets with complete code examples available at the GitHub repo.\nMain File # With the release of Terraform v1.3.0, setting object attributes as optional with default values is now possible. Additionally, the experimental [module_variable_optional_attrs] option is now depreciated, see issue #31355. We\u0026rsquo;ll start by creating the main.tf file, which is the entry point for our module. Here we will define the required provider plugins and minimal Terraform version in the terraform block. We\u0026rsquo;ll also create a single resource block that uses the resource type proxmox_vm_qemu from the Telmate provider to create a new virtual machine.\nWithin this resource we\u0026rsquo;ll set several attributes including three required settings: target_node, clone, and vmid. The target_node and clone attributes take string values that reference the Proxmox node and template name to clone from, respectively. The vmid attribute takes a number value which is the ID of the virtual machine we want to create. We\u0026rsquo;ll set all attributes using variables that we will define in the next section.\nEach VM will have at minimum a single disk, but we\u0026rsquo;ll also want to provide the flexibility to add additional disk. The provider allows us to do this by defining multiple disk block within our resource block. To avoid manually duplicating a block of code for each disk, we\u0026rsquo;ll use a dynamic block to create multiple disk blocks based on the number of disks requested by the user. Within the dynamic block, it iterates through a disks variable that we\u0026rsquo;ll define later, however, this is a list of objects that each contain values to populate the disk attributes within the content block.\nLastly, we\u0026rsquo;ll use several ternary conditional expressions to set cloud-init attributes depending upon whether the user provides an SSH key or sets a static IP address. The syntax for this type of expression is if CONDITION ? TRUE_VALUE : FALSE_VALUE. For example with sshkeys, if the variable ci_ssh_key is set, then use the built-in function file() to pass the contents as a string, else set the value to null.\n# modules/vm/main.tf terraform { required_version = \u0026#34;\u0026gt;=1.3.0\u0026#34; required_providers { proxmox = { source = \u0026#34;Telmate/proxmox\u0026#34; version = \u0026#34;~\u0026gt; 2.9.0\u0026#34; } } } resource \u0026#34;proxmox_vm_qemu\u0026#34; \u0026#34;vm\u0026#34; { target_node = var.node vmid = var.vm_id ... clone = var.template_name ... dynamic \u0026#34;disk\u0026#34; { for_each = var.disks content { type = disk.value.disk_interface slot = disk.value.disk_slot storage = disk.value.disk_storage size = disk.value.disk_size ... } } ... # cloud-init config ciuser = var.ci_user sshkeys = (var.ci_ssh_key != null ? file(\u0026#34;${var.ci_ssh_key}\u0026#34;) : null) ... ipconfig0 = (var.ci_ipv4_cidr != null ? \u0026#34;ip=${var.ci_ipv4_cidr},gw=${var.ci_ipv4_gateway}\u0026#34; : \u0026#34;ip=dhcp\u0026#34;) ... } For a list of all available attributes, see Telmate documentation.\nVariables File #HCL variables are used to define inputs that will be passed into the module and that you would like to: 1) set default values for; 2) mask sensitive values from being displayed in logs or output; 3) enforce type constraints on; and/or 4) validate values. These inputs can be either required or optional, depending on if a default value is defined in the variable block. Terraform will also enforce that all required variables are provided when running terraform \u0026lt;plan/apply/destroy\u0026gt;, and met the type constraints requirements and pass any custom validation rules. In the example below, node and vm_id are required variables, while bios is optional. Additionally, bios inputs are limited to either seabios or ovmf, otherwise an error will be raised.\nTerraform allows you to set their values in a number of ways, including using the Terraform CLI, environmental variables, and configuration files (*.tfvars). We\u0026rsquo;ll discuss these methods in more detail later.\n# module/vm/variables.tf variable \u0026#34;node\u0026#34; { description = \u0026#34;Name of Proxmox node to provision VM on, e.g. \u0026#39;pve\u0026#39;.\u0026#34; type = string } variable \u0026#34;vm_id\u0026#34; { description = \u0026#34;ID number for new VM.\u0026#34; type = number } ... variable \u0026#34;bios\u0026#34; { description = \u0026#34;VM bios, setting to \u0026#39;ovmf\u0026#39; will automatically create a EFI disk.\u0026#34; type = string default = \u0026#34;seabios\u0026#34; validation { condition = contains([\u0026#34;seabios\u0026#34;, \u0026#34;ovmf\u0026#34;], var.bios) error_message = \u0026#34;Invalid bios setting. Valid options: \u0026#39;seabios\u0026#39; or \u0026#39;ovmf\u0026#39;.\u0026#34; } } ... Above in the main.tf file, we created a dynamic block for the disk block. Now we need to create a corresponding disks variable that will be a list of objects, list(object()). Terraform objects are simply a map of key-value pairs, in which you can specify type constraints for each key attribute including: type, default value, and optionality. We can also define a default object for the variable that will be used if no disk are specified by the user. In the example below, the default disk object is one that is 8GB in size and in the slot, scsi0, which matches the boot disk drive in the Proxmox template that we\u0026rsquo;ll create later.\n# module/vm/variables.tf ... variable \u0026#34;disks\u0026#34; { description = \u0026#34;Terraform object with disk configurations.\u0026#34; type = list(object({ disk_interface = optional(string, \u0026#34;scsi\u0026#34;) disk_slot = optional(number, 0) disk_storage = optional(string, \u0026#34;local-lvm\u0026#34;) disk_size = optional(string, \u0026#34;8G\u0026#34;) disk_format = optional(string, \u0026#34;raw\u0026#34;) disk_cache = optional(string, \u0026#34;writeback\u0026#34;) disk_backup = optional(number, 0) disk_iothread = optional(number, 0) disk_ssd = optional(number, 1) disk_discard = optional(string, \u0026#34;on\u0026#34;) } )) default = [{ disk_interface = \u0026#34;scsi\u0026#34; disk_slot = 0 disk_storage = \u0026#34;local-lvm\u0026#34; disk_size = \u0026#34;8G\u0026#34; disk_format = \u0026#34;raw\u0026#34; disk_cache = \u0026#34;writeback\u0026#34; disk_backup = 0 disk_iothread = 0 disk_ssd = 1 disk_discard = \u0026#34;on\u0026#34; }] } ... Output File #In the last module file, we define two output variables that will be used to retrieve the instance ID and IPv4 address of the virtual machine. The value attribute references the provider resource, proxmox_vm_qemu; the simple but unique identifier for the resource, vm; and an expression that is exposed by the Telmate provider, e.g. id.\n# module/vm/outputs.tf output \u0026#34;id\u0026#34; { description = \u0026#34;Instance VM ID\u0026#34; value = proxmox_vm_qemu.vm.id } output \u0026#34;public_ipv4\u0026#34; { description = \u0026#34;Instance Public IPv4 Address\u0026#34; value = proxmox_vm_qemu.vm.default_ipv4_address } Using the Module: Building a K3s Cluster # Code to create K3s cluster available in examples/k3s-cluster (link). Feel free to copy it into your own homelab directory. Now that we have a module that creates a virtual machine, let\u0026rsquo;s use it to create three VMs and use Terraform to create a K3s cluster.\nConfiguring the Provider #We\u0026rsquo;ll start by adding the provider configuration to homelab/k3s-cluster/main.tf. We\u0026rsquo;ll use the same Terraform block configuration as we did in the vm module, but this time we\u0026rsquo;ll add a provider block that defines access to the Proxmox API.\n# homelab/k3s-cluster/main.tf terraform { required_providers { proxmox = { source = \u0026#34;Telmate/proxmox\u0026#34; version = \u0026#34;~\u0026gt; 2.9.0\u0026#34; } } } provider \u0026#34;proxmox\u0026#34; { pm_api_url = var.pve_api_url pm_api_token_id = var.pve_token_id pm_api_token_secret = var.pve_token_secret } ... Let\u0026rsquo;s create a few variables to mask sensitive API token values from Terraform logs, the homelab/k3s-cluster/variables file contains the following:\n# homelab/k3s-cluster/variables.tf variable \u0026#34;pve_token_id\u0026#34; { sensitive = true } variable \u0026#34;pve_token_secret\u0026#34; { sensitive = true } variable \u0026#34;pve_api_url\u0026#34; { description = \u0026#34;Proxmox API Endpoint, e.g. \u0026#39;https://pve.example.com/api2/json\u0026#39;\u0026#34; type = string sensitive = true validation { condition = can(regex(\u0026#34;(?i)^http[s]?://.*/api2/json$\u0026#34;, var.pve_api_url)) error_message = \u0026#34;Proxmox API Endpoint Invalid. Check URL - Scheme and Path required.\u0026#34; } } Using the Module #In order to use the vm module, we\u0026rsquo;ll create a module block within the homelab/k3s-cluster/main.tf file and set the source to the modules/vm/ folder. To create multiple VM instances, we\u0026rsquo;ll use the for_each argument and pass a map of the VM instances we want to the child vm module. Specifically, we\u0026rsquo;ll only set values that are unique for a given instance, and set all common values in the root k3s_cluster module.\n# homelab/k3s-cluster/main.tf ... module \u0026#34;k3s_cluster\u0026#34; { source = \u0026#34;../../modules/vm\u0026#34; for_each = tomap({ \u0026#34;k3s-controller\u0026#34; = { id = 210 ipv4_cidr = \u0026#34;192.168.1.210/24\u0026#34; ipv4_gateway = \u0026#34;192.168.1.1\u0026#34; }, \u0026#34;k3s-node1\u0026#34; = { id = 211 ipv4_cidr = null # Use DHCP ipv4_gateway = null # Use DHCP }, \u0026#34;k3s-node2\u0026#34; = { id = 212 ipv4_cidr = null # Use DHCP ipv4_gateway = null # Use DHCP }, }) node = \u0026#34;pve\u0026#34; # required vm_id = each.value.id # required vm_name = each.key # optional template_name = \u0026#34;ubuntu20\u0026#34; # required vcpu = 2 # optional memory = 4096 # optional ci_custom_data = \u0026#34;vendor=local:snippets/vendor-data.yaml\u0026#34; # optional ci_ssh_key = \u0026#34;~/.ssh/id_ed25519.pub\u0026#34; # optional, add SSH key to \u0026#39;default\u0026#39; user ci_ipv4_cidr = each.value.ipv4_cidr # optional ci_ipv4_gateway = each.value.ipv4_gateway # optional } ... Using Terraform to Run a Shell Command #Lastly, we can use Terraform to generate a random string that we\u0026rsquo;ll use as the K3s join token and to run the K3s install script. To pass the IP address of our K3s control node to the install script, we\u0026rsquo;ll create a local variable, controller_ip, that captures the module output public_ipv4 for the control node.\n# homelab/k3s-cluster/main.tf ... locals { controller_ip = module.k3s_cluster[\u0026#34;k3s-controller\u0026#34;].public_ipv4 agent_ips = { for k, v in module.k3s_cluster : k =\u0026gt; v.public_ipv4 if k != \u0026#34;k3s-controller\u0026#34; } } ... resource \u0026#34;random_string\u0026#34; \u0026#34;k3s_token\u0026#34; { length = 64 special = false } ... resource \u0026#34;null_resource\u0026#34; \u0026#34;agents\u0026#34; { for_each = local.agent_ips connection { host = each.value user = \u0026#34;ubuntu\u0026#34; } provisioner \u0026#34;remote-exec\u0026#34; { inline = [ \u0026#34;curl -sfL https://get.k3s.io | K3S_URL=https://${local.controller_ip}:6443 K3S_TOKEN=${random_string.k3s_token.result} sh -\u0026#34; ] } } Now, let\u0026rsquo;s bring it all together and deploy our K3s cluster using Terraform.\nProvisioning using Terraform #Proxmox Template: Ubuntu 20.04 #Let\u0026rsquo;s create a simple Ubuntu 20.04 template in Proxmox to use as the base OS for our cluster nodes. If you already have a template available, change the clone value to the name of your template; otherwise, SSH into your Proxmox node and run the following commands as a privileged user:\n# change into the ISO directory cd /var/lib/vz/template/iso # download the image wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img # create a virtual machine qm create 9000 --name ubuntu20 --machine q35 --scsihw virtio-scsi-pci # import the ubuntu image as the root disk qm disk import 9000 ubuntu-20.04-server-cloudimg-amd64.img local-lvm # attach and set the root disk as boot drive qm set 9000 --boot order=scsi0 --scsi0 \u0026#34;local-lvm:vm-9000-disk-0\u0026#34; # add cloud-init config drive qm set 9000 --ide2 local-lvm:cloudinit --citype nocloud # convert the vm to a template qm template 9000 For more details on creating Proxmox templates, see Proxmox: Cloud-Init Support or checkout the post Golden Images and Proxmox Templates Using cloud-init.\nProvision #Finally, we will use Terraform to provision the three virtual machines and run the K3s scripts. We will define the API variables in-line using the -var flag. If you created a token using the directions above, the pve_api_token_id value is terraform@pve!token and the pve_api_token_secret value is similar to the output from above, e.g. 782a7700-4010-4802-8f4d-820f1b226850. Your pve_api_url value may be something similar to https://pve.example.com/api2/json if your node is behind a proxy, otherwise you can use the IP address, e.g. http://192.168.1.100/api2/json.\n# create a terraform plan terraform plan \\ -var=\u0026#39;pve_api_url=https://pve.example.com/api2/json\u0026#39; \\ -var=\u0026#39;pve_api_token_id=TOKEN\u0026#39; \\ -var=\u0026#39;pve_api_token_secret=SECRET\u0026#39; \\ -out tfplan # build the cluster terraform apply tfplan Test #Once provisioning is complete connect to the controller node and view the cluster using kubectl.\n# connect to the controller node user@desktop:~$ ssh ubuntu@$(terraform output -raw controller_ip) # list the active nodes ubuntu@k3s-controller:~$ sudo kubectl get nodes NAME STATUS ROLES AGE VERSION k3s-controller Ready control-plane,master 51s v1.23.3+k3s1 k3s-node1 Ready \u0026lt;none\u0026gt; 44s v1.23.3+k3s1 k3s-node2 Ready \u0026lt;none\u0026gt; 41s v1.23.3+k3s1 Remove #Once you are finished with the cluster you can tear it down using terraform destroy.\n# destroy the cluster terraform destroy \\ -var=\u0026#39;pve_api_url=https://pve.example.com/api2/json\u0026#39; \\ -var=\u0026#39;pve_api_token_id=TOKEN\u0026#39; \\ -var=\u0026#39;pve_api_token_secret=SECRET\u0026#39; Recap #Terraform is a powerful IaC tool that simplifies infrastructure management, promotes collaboration, and enhances efficiency. Here, we created a module to simplify provisioning virtual machines on Proxmox VE and used it to create three nodes for our K3s cluster. We also used Terraform\u0026rsquo;s built-in resources to run the shell commands to create the cluster and generate a K3s join token. Furthermore, because we used Terraform, the k3s_token is stored in the terraform.tfstate file, which allows you to add additional nodes to the cluster without destroying the cluster and rebuilding it - test this out by adding an additional VM instance and re-running terraform apply. Additionally, because Terraform captures the IP addresses of the VMs we created, setting a static IP address for the control node is not necessary and works with DHCP.\nHopefully, you\u0026rsquo;ve found this tutorial useful, feel free to use it and the companion repo as a reference for your own infrastructure. Thanks for reading!\nReferences # Github: trfore/terraform-telmate-proxmox Terraform:\nTerraform Terraform Documentation HCL language HCL types Github: Hashicorp/Terraform Github: Example Terraform .gitignore HashiCorp Learn: Modules Hashicorp Docs: Locals HashiCorp Learn: Modules - Overview Terraform Providers / Plugins:\nTelmate/Terraform-Provider-Proxmox Telmate Documentation Other Resources:\nTerraform: Up \u0026amp; Running ","date":"February 20, 2022","permalink":"https://www.trfore.com/posts/provisioning-proxmox-vms-with-terraform/","section":"Posts","summary":"Master Terraform modules to provision a K3s cluster on Proxmox with a single command","title":"Provisioning Proxmox VMs with Terraform"},{"content":"","date":null,"permalink":"https://www.trfore.com/tags/ansible/","section":"Tags","summary":"","title":"Ansible"},{"content":"Create Ansible User in Proxmox ## create role pveum role add AnsibleUser --privs \u0026#34;Datastore.AllocateSpace \\ VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \\ VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk \\ VM.Config.HWType VM.Config.Memory VM.Config.Network \\ VM.Config.Options VM.Console VM.Monitor VM.PowerMgmt\u0026#34; # create group pveum group add ansible-users # add permissions pveum acl modify / -group ansible-users -role AnsibleUser # create user \u0026#39;ansible\u0026#39; pveum useradd ansible@pve -groups ansible-users # generate a token pveum user token add ansible@pve token -privsep 0 Module: proxmox #Quick notes:\nAnsible Docs: community.general.proxmox Module: proxmox_kvm #Quick notes:\nAnsible Docs: community.general.proxmox_kvm Provides a way to provision VMs on PVE from backups, templates or create new VMs. Alternative to Terraform, but without a state file. Run the plays from a localhost or remote host that is not a Proxmox node, as the task will access the Proxmox API endpoint. The task requires the python libraries: proxmoxer and requests. However, only the Ansible controller needs these dependencies installed. You cannot set the api_* variables in a file, this leads to code duplication in the playbook. Example Code #Variable File:\n# proxmox_vars.yml pve_host: \u0026#34;192.168.100.1:8006/api2/json\u0026#34; pve_node: \u0026#34;proxmox\u0026#34; pve_storage: \u0026#34;local-lvm\u0026#34; pve_token_id: \u0026#34;{{ vault_pve_token_id }}\u0026#34; pve_token_secret: \u0026#34;{{ vault_pve_token_secret }}\u0026#34; pve_user: \u0026#34;ansible@pve\u0026#34; Vault File:\n# vault.yml vault_pve_token_id: \u0026#34;token\u0026#34; vault_pve_token_secret: \u0026#34;782a7700-4010-4802-8f4d-820f1b226850\u0026#34; Example Playbook:\nvmid: PVE template ID. clone: PVE template name, required if vmid is set, but no validation occurs so the value is arbitrary. name: name of new VM, if unset defaults to 'Copy-of-VM-[PVE_TEMPLATE_NAME]'. # playbook.yml --- - name: Create VM from Proxmox Template hosts: - localhost vars_files: - proxmox_vars.yml tasks: - name: Create VM from Proxmox Template delegate_to: localhost block: - name: Clone Proxmox Template community.general.proxmox_kvm: node: \u0026#34;{{ pve_node }}\u0026#34; api_host: \u0026#34;{{ pve_host }}\u0026#34; api_user: \u0026#34;{{ pve_user }}\u0026#34; api_token_id: \u0026#34;{{ pve_token_id }}\u0026#34; api_token_secret: \u0026#34;{{ pve_token_secret }}\u0026#34; vmid: 9000 # template ID clone: foobar # no validation, arbitrary value to trigger clone process name: new-vm-name storage: \u0026#34;{{ pve_storage }}\u0026#34; timeout: 60 register: proxmox_vm - name: VM - Start, Stop, Remove community.general.proxmox_kvm: node: \u0026#34;{{ pve_node }}\u0026#34; api_host: \u0026#34;{{ pve_host }}\u0026#34; api_user: \u0026#34;{{ pve_user }}\u0026#34; api_token_id: \u0026#34;{{ pve_token_id }}\u0026#34; api_token_secret: \u0026#34;{{ pve_token_secret }}\u0026#34; vmid: \u0026#34;{{ proxmox_vm.vmid }}\u0026#34; state: started # started, stopped, absent timeout: 60 References #Ansible:\nAnsible Docs: community.general.proxmox_kvm https://github.com/ansible-collections/community.general/blob/main/plugins/modules/proxmox_kvm.py https://github.com/ansible-collections/community.general/blob/main/plugins/module_utils/proxmox.py Ansible Docs: community.general.proxmox Proxmox:\nProxmox API Docs Python:\nProxmoxer Requests ","date":"December 18, 2021","permalink":"https://www.trfore.com/posts/ansible-modules-for-proxmox/","section":"Posts","summary":"","title":"Ansible Modules for Proxmox"},{"content":"Code Examples \u0026amp; Posts #All post content is available only on this site, trfore.com. The majority of code examples are available on Github and licensed accordingly. For examples that only appear on the site, the code is licensed under Apache 2.0 License. Code examples pulled from other sources are marked as such and retain the original licenses with attribution to the original authors. I make an effort to update code examples in the event of significant modifications; however, it is possible that certain posts and commands may become obsolete due to the nature of technology.\nPublications #I was fortunate to work with creative people and ask interesting questions about how our brain works, here is a short list of my publications:\nAcetylcholine Modulates Cerebellar Granule Cell Spiking by Regulating the Balance of Synaptic Excitation and Inhibition (2020)\nASTN2 modulates synaptic strength by trafficking and degradation of surface proteins (2018)\nIntersectional strategies for cell type specific expression and transsynaptic labeling (2014)\nEpsin 1 Promotes Synaptic Growth by Enhancing BMP Signal Levels in Motoneuron Nuclei (2013)\nTranscriptome Profiling Following Neuronal and Glial Expression of ALS-Linked SOD1 in Drosophila (2013)\nWebsite #Powered by Hugo \u0026amp; Congo. Hosted on Github \u0026amp; Cloudflare.\n","date":null,"permalink":"https://www.trfore.com/about/","section":"trfore","summary":"","title":"About"},{"content":"","date":"February 28, 2019","permalink":"https://www.trfore.com/contact/","section":"trfore","summary":"","title":"Contact"},{"content":"Below is a collection of software and books that have greatly influenced the way I work and think. I primarily keep this updated for myself and for any new graduate student starting their journey into the life sciences.\nFor the books that are freely available to the public, I have included those direct links. Many of these books are available via academic subscriptions and some of the programming books are available via membership to ACM. Manning, No Starch Press, and O\u0026rsquo;Reilly often post book bundles on Humble Bundle that are DRM-free.\nSoftware # Joplin, fantastic note-taking app with inline code support; end-to-end encryption; and synchronization via a cloud provider or local server. Rstudio / Posit, best IDE for R with support for python. VScode, general all-around code editor. Zotero, open-source PDF and reference manager. Books #Data Analytics # Feature Engineering and Selection: A Practical Approach for Predictive Models (free) Introduction to Statistical Learning (free) Statistical Rethinking, Bayesian data analysis Statistics with R: Bayesian Statistics (free) The Elements of Statistical Learning: Data Mining, Inference, and Prediction (free) R Language # Advance R (free) R for Data Science (free) Tidy Modeling with R (free) Neuroscience # Advanced Data Analysis in Neuroscience: Integrating Statistical and Computational Models Programming #Databases # Designing Data-Intensive Applications SQL Indexing and Tuning e-Book (free) Devops # Ansible: Up and Running Container Security, complementary copy available from Aqua Docker: Up and Running Kubernetes: Up and Running, complementary copy available from VMware Tanzu Terraform: Up and Running Python # Architecture Patterns with Python (free) ","date":null,"permalink":"https://www.trfore.com/resources/","section":"trfore","summary":"","title":"Resources"},{"content":"Last updated: September 24, 2024\nSection 1: Who We Are #Our website address is: https://trfore.com.\nSection 2: What Personal Data We Collect and Why We Collect It #Analytics #Usage Data is collected automatically when using the Website.\nUsage Data may include information such as Your Device\u0026rsquo;s Internet Protocol address (e.g. IP address), browser type, browser version, the pages of our Website that You visit, the time and date of Your visit, the time spent on those pages, unique device identifiers and other diagnostic data.\nWhen You access the Website by or through a mobile device, We may collect certain information automatically, including, but not limited to, the type of mobile device You use, Your mobile device unique ID, the IP address of Your mobile device, Your mobile operating system, the type of mobile Internet browser You use, unique device identifiers and other diagnostic data.\nWe may also collect information that Your browser sends whenever You visit our Website or when You access the Website by or through a mobile device.\nComments #When visitors leave comments on the site we collect the data shown in the comments form, and also the visitor’s IP address and browser user agent string to help spam detection.\nAn anonymized string created from your email address (also called a hash) may be provided to the Gravatar service to see if you are using it. The Gravatar service privacy policy is available here. After approval of your comment, your profile picture is visible to the public in the context of your comment.\nContact \u0026amp; Newsletter Forms #To assist with sending You requested information, We use forms to collect and store Your email address. This includes your submitting IP address.\nCookies #If you leave a comment on our site you may opt-in to saving your name, email address and website in cookies. These are for your convenience so that you do not have to fill in your details again when you leave another comment. These cookies will last for one year. Learn more about cookies on Mozilla\u0026rsquo;s website and online tracking on the FTC website.\nEmbedded Content From Other Websites #Articles on this site may include embedded content (e.g. articles, images, links, videos, etc.). Embedded content from other websites behaves in the exact same way as if the visitor has visited the other website.\nThese websites may collect data about you, use cookies, embed additional third-party tracking, and monitor your interaction with that embedded content, including tracing your interaction with the embedded content if you have an account and are logged in to that website.\nSection 3: Who We Share Your Data With #Service Providers and others who help with our business operations and assist in the delivery of our products and services including, but not limited to, application development, site hosting, maintenance, data analysis, infrastructure provision, IT services, customer service, email delivery services, payment processing, marketing, analytics, and enforcement of our Terms of Service Agreement and other agreements;\nOther users of the site to identify you to anyone to whom you send messages or make comments through the Services;\nPersons or entities with whom you consent to have your Personal Data shared;\nThird parties in order to prevent damage to our property (tangible and intangible), for safety reasons, or to collect amounts owed to us; and\nThird parties as we believe necessary or appropriate, in any manner permitted under applicable law, including laws outside your country of residence to: comply with legal process; respond to requests from public and government authorities, including public and government authorities outside your country of residence; enforce our Terms of Service Agreement and other agreements; protect our operations; protect our rights, privacy, safety or property, and/or that of our affiliates, you, or others; and allow us to pursue available remedies or limit the damages that we may sustain.\nWe will never sell, rent, or lease Your Personal Data to a third party.\nSection 4: How Long We Retain Your Data #If you leave a comment, the comment and its metadata are retained indefinitely. This is so we can recognize and approve any follow-up comments automatically instead of holding them in a moderation queue.\nFor users that register on our website, we also store the personal information they provide in their user profile. All users can see, edit, or delete their personal information at any time (except they cannot change their username). Website administrators can also see and edit that information.\nAnalytics data is retained indefinitely. Contact forms and comments cookies are held for one year.\nSection 5: What Rights You Have over Your Data #If You have an account on this site, or have left comments, You can request to receive an exported file of the personal data We hold about you, including any data You have provided to Us. You can also request that We erase any personal data we hold about you.\nThis does not include any data We are obliged to keep for administrative, legal, or security purposes.\nSection 6: Where We Send Your Data #Visitor comments may be checked through an automated spam detection service. Third parties have access to your data as noted within this agreement.\nSection 7: Additional Information #We will retain your Personal Data for the period necessary to fulfill the purposes outlined in this Privacy Policy unless a longer retention period is required or allowed by law.\nUSERS UNDER 13 YEARS OF AGE #Our Services are not directed to and We do not knowingly collect Personal Data from children under the age of 13. If We become aware that a child under the age of 13 has provided Us with Personal Data, We will take steps to remove such data. If You become aware that Your child has provided Us with Personal Data without Your consent, please contact us. By using the Services, You are representing to Us that You are not under the age of 13.\nProtection of Personal Data #We use reasonable and appropriate physical, electronic, and administrative safeguards to protect personal data from loss, misuse and unauthorized access, disclosure, alteration and destruction, taking into account the nature of the Personal Data and risks involved in processing that information.\nNotice and Choice #i. The types of Personal Data We collect and why We collect it, are described in Section 2 of this Privacy Policy.\nii. See section 5 to remove Your data from this site.\nChanges to This Privacy Policy #We may change this Privacy Policy at any time. The most recent version of the Privacy Policy is indicated by the “Last Updated” date at the top of the Privacy Policy. All changes are effective immediately upon posting. Please review this Privacy Policy frequently to stay updated on changes that may affect you. Your continued use of this website signifies your continuing consent to be bound by this Privacy Policy.\nContact Information #If you have any questions about this Privacy Policy, You can contact us.\n","date":"January 1, 0001","permalink":"https://www.trfore.com/privacy/","section":"trfore","summary":"","title":"Privacy Policy"}]