Storing Terraform State in MinIO and PostgreSQL

Intro #

As discussed in the previous post, storing Terraform state in a secure remote backend is important as it typically contains sensitive data, such as database credentials. Additionally, using a backend that supports state locking is important in multi-user environments to ensure that only one user can modify the state at any given time. In this post we will demonstrate using MinIO or PostgreSQL with Terraform to securely store the state within your local environment. To keep things simple we will use docker containers to test each backend, but with the intent of moving to a more production-ready solution in the future.

For this tutorial, you need to have Terraform and Docker installed on your local desktop, along with access to a Proxmox server. We’ll provision a simple VM to generate a Terraform state file, example code is available below and the backend configuration is available under each section, MinIO or Postgres. You will also need a Proxmox template, ubuntu20, to deploy the VM - see below for creating a simple template.

MinIO #

MinIO is a highly scalable S3-compatible object store with a RESTful API. It’s open source and licensed under AGPLv3 with the option for commercial licensing. While it is designed as a distributed storage solution, a single disk deployment is possible. As a Terraform backend, MinIO supports versioning and object locking which is useful for tracking and reviewing changes to Terraform state. While it does not support state locking, MinIO’s intuitive user-interface is a great way explore using remote backends.

MinIO Docker Image #

For this example, we’ll use the Bitnami MinIO image and set the root user access using the environment variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD. Additionally, we’ll set the region to us-east-1 and expose the container locally on ports 9000 and 9001. Run the following command to start the container:

docker run -d --name minio-server \
    --env MINIO_ROOT_USER="root" \
    --env MINIO_ROOT_PASSWORD="rootsecret" \
    --env MINIO_REGION="us-east-1" \
    --publish 9000:9000 \
    --publish 9001:9001 \
    bitnami/minio:latest

Next, create a bucket named staging with versioning and object locking enabled:

# create bucket with object locking enabled
docker exec minio-server mc mb --with-lock local/staging

# enable versioning
docker exec minio-server mc version enable local/staging

# enable 30 day retention policy
docker exec minio-server mc retention set --default compliance 30d local/staging

You can confirm the server is running and view all available buckets by using the MinIO client, mc:

$ docker exec minio-server mc ls local
[2022-02-20 17:26:06 UTC]     0B staging/

Generating Access Keys #

While you can use the root username and password to access the MinIO server, let’s create an additional user with read and write permissions for Terraform to use. For simplicity, set the username to terraform and password to terraformsecret. Feel free to generate a new access key using the CLI or GUI directions below.

Using the CLI #

To create an access key using the MinIO client, run:

# create a new user
docker exec minio-server mc admin user add local terraform terraformsecret

# add permissions to the user
docker exec minio-server mc admin policy attach local readwrite --user terraform

For MinIO versions prior to v2023-03-20, you’ll need to use the following command to set the permissions:

# add permissions to the user
docker exec minio-server mc admin policy set local readwrite user=terraform

Using the GUI #

To create an access key using the MinIO server’s web interface, open your browser at http://localhost:9001 and login using the root credentials from MINIO_ROOT_USER and MINIO_ROOT_PASSWORD. Note: If you didn’t set the root credentials using environment variables from above, the default values are minio and miniosecret.

Under the Identity menu, click Users. Click on the Create User button and fill in the form with the username, terraform, and password, terraformsecret. Check the box for readwrite permissions, and click Save.

Configuring Terraform To Use MinIO #

Now that we have a MinIO server running in Docker, let’s configure Terraform to use it. Using the example code below, add the following s3 backend configuration to the terraform block:

# main.tf
terraform {
  # PROVIDER CONFIGURATION FROM BELOW
  backend "s3" {
    bucket   = "staging"
    key      = "terraform.tfstate"
    endpoint = "http://localhost:9000"
    region   = "us-east-1"

    access_key = "terraform"
    secret_key = "terraformsecret"

    force_path_style            = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true # required with Terraform v1.6+
    skip_region_validation      = true
  }
}

Go ahead and run terraform apply to build the example VM and generate the initial state file. Terraform will create a new terraform.tfstate file inside the staging bucket, let’s take a look at it using the MinIO client head command:

# view the terraform state file in MinIO
$ docker exec minio-server mc head local/staging/terraform.tfstate
{
  "version": 4,
  "terraform_version": "1.1.5",
  "serial": 2,
  "lineage": "87f396c1-0aea-4493-f52c-eb3cd0a18b58",
  "outputs": {
    "id": {
      "value": "pve/qemu/100",
      "type": "string"
    },

You can also Preview the state file from the MinIO GUI:

With versioning and object locking enabled, try changing vcpu count to 2 in the main.tf file and run terraform apply again. Within the GUI, you should see a new version of the state file.

After you are done exploring, go ahead and destroy the VM and remove the MinIO server:

# destroy the VM
terraform destroy

# stop the MinIO server
docker stop minio-server && docker rm minio-server

Limitations of Using MinIO #

Not all the AWS S3 configuration options, S3 backend documentation, are supported by MinIO. Notability, state locking via a DynamoDB-like database is not supported, thus this approach is not suitable for multi-user environments.

PostreSQL #

PostgreSQL is an open source relational database management system (RDBMS) that supports standard SQL language. As a Terraform backend, it supports state locking but not versioning and object locking. Directly viewing the state information is possible with simple SQL queries to the database, along with calls to terraform state <pull/show>.

PostgreSQL Docker Image #

We’ll use the offical Postgres image to create a server with the default user postgres having the password password123, and a new terraform database. Additionally, we’ll expose the container on port 5432 on the host machine.

docker run -d --name postgres \
    --user postgres \
    --env POSTGRES_PASSWORD=password123 \
    --env POSTGRES_DB=terraform \
    --publish 5432:5432 \
    postgres:latest

Configuring Terraform To Use PostgreSQL #

Using the example code below, add the following pg backend configuration to the terraform block:

# main.tf
terraform {
  # PROVIDER CONFIGURATION FROM BELOW
  backend "pg" {
    conn_str = "postgres://postgres:password123@localhost:5432/terraform?sslmode=disable"
  }
}

Adding sslmode=disable to the connection string will disable SSL connections and avoid the error pq: SSL is not enabled on the server. After running terraform init, Terraform will create a new schema terraform_remote_state inside the database. You can configure the schema name by setting schema_name in the backend block. For additional configuration options see pg backend documentation.

Go ahead and run terraform apply to build the example VM and generate the initial state information. Let’s take a look inside the database, the state information will be stored in the terraform database under the terraform_remote_state schema in the state table, use the following commands to view it:

# connect to the container
docker exec -it postgres psql

Now, using psql commands inside the container:

-- list the databases
postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 terraform | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)

-- connect to the terraform database
postgres=# \c terraform

-- list the columns in the table
terraform=# \d terraform_remote_state.states;
                       Table "terraform_remote_state.states"
 Column |  Type  | Collation | Nullable |                  Default
--------+--------+-----------+----------+-------------------------------------------
 id     | bigint |           | not null | nextval('global_states_id_seq'::regclass)
 name   | text   |           |          |
 data   | text   |           |          |
Indexes:
    "states_pkey" PRIMARY KEY, btree (id)
    "states_by_name" UNIQUE, btree (name)
    "states_name_key" UNIQUE CONSTRAINT, btree (name)

-- view the terraform state information
terraform=# SELECT "data" FROM terraform_remote_state.states;
                             data
------------------------------------------------------------------
...

State Locking in PostgreSQL #

One of the advantages of using PostgreSQL as a backend is that it provides state locking, ensuring only a single user can modify state at a time. You can view this in action by querying the pg_locks table while running Terraform. For example, run the following query in a separate terminal window while running terraform apply:

postgres=# SELECT * FROM pg_locks;

Example output:

  locktype  | database |...| classid | objid | objsubid | virtualtransaction | pid |      mode       | granted |...|
------------+----------+---+---------+-------+----------+--------------------+-----+-----------------+---------+---+
 advisory   |    16384 |...|       0 |     1 |        1 | 3/0                |  96 | ExclusiveLock   | t       |...|

You will see a transient advisory lock appearing in the pg_locks table. This lock is removed when Terraform completes its run. To confirm the lock is for the terraform database, try running the following query while Terraform is processing state changes:

SELECT pl.database, pl.locktype, pl.mode, psa.datid, psa.datname
FROM pg_locks pl
JOIN pg_stat_activity psa ON pl.database = psa.datid
WHERE psa.datname = 'terraform';

It should return a similar result:

 database | locktype |     mode      | datid |  datname
----------+----------+---------------+-------+-----------
    16384 | advisory | ExclusiveLock | 16384 | terraform
(1 row)

Cleaning Up #

Once you are finished, clean-up your environment by running the following:

# destroy the VM
terraform destroy

# stop & remove the Postgres container
docker stop postgres && docker rm postgres

Recap #

Overall, we tested two different backends for storing terraform state. Using MinIO as a S3 backend, we were able to store state in a environment specific manner using pre-configured buckets, for example staging. Additionally, with versioning and object locking enabled on the bucket, we retained the history of all terraform runs and could easily view and rollback to a specific version using the MinIO GUI. While MinIO does not offer state locking, you do have the ability to encrypt data at rest. If you are interested in adding bucket encryption see MinIO Docs: Data Encryption (SSE) for more details.

With Postgres, we gained state locking, which is great for multi-user environments. In this post, we hard-coded the database username and password into connection parameter in the main.tf file, however, this is not a good practice for production environments. Instead, consider using environment variables to pass in the database credentials via PGUSER & PGPASSWORD. If state locking is a requirement for your local environment, also consider using the Kubernetes backend, as it allows you to store state inside the cluster’s etcd store and supports state locking.

Hopefully, you’ve found this exploration of remote backends useful. Thanks for reading!

Full Code #

Template Code #

For more elaborate templates, checkout the post: Golden Images and Proxmox Templates Using cloud-init

# SSH into your PVE server
user@desktop:~$ ssh root@pve

Use the following commands to create a new template:

# change into the ISO directory
cd /var/lib/vz/template/iso

# download the image
wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img

# create a virtual machine
qm create 9000 --name ubuntu20 --machine q35 --scsihw virtio-scsi-pci
# import the ubuntu image as the root disk
qm disk import 9000 ubuntu-20.04-server-cloudimg-amd64.img local-lvm
# attach and set the root disk as boot drive
qm set 9000 --boot order=scsi0 --scsi0 "local-lvm:vm-9000-disk-0"
# add cloud-init config drive
qm set 9000 --ide2 local-lvm:cloudinit --citype nocloud
# convert the vm to a template
qm template 9000

Terraform File #

Telmate Proxmox Provider #

Telmate Proxmox v2.9 only works with Proxmox v7. For Proxmox v8 support, use the BPG Proxmox provider and code below.

# main.tf
terraform {
  required_providers {
    proxmox = {
      source  = "Telmate/proxmox"
      version = "~> 2.9.0"
    }
  }
  # ADD BACKEND CONFIGURATION HERE
}

provider "proxmox" {
  pm_api_url          = "https://pve.example.com/api2/json"
  pm_api_token_id     = "terraform@pve!token"
  pm_api_token_secret = "<TOKEN_VALUE>"
}

resource "proxmox_vm_qemu" "proxmox_vm" {
  target_node = "pve"
  vmid        = 100
  name        = "vm-example"
  clone       = "ubuntu20"
  full_clone  = true
  os_type     = "cloud-init"
  cores       = "1"
  memory      = "1024"
  agent       = 1

  disk {
    type     = "scsi"
    slot     = 0
    storage  = "local-lvm"
    size     = "8G"
    format   = "raw"
    cache    = "writeback"
    iothread = 0
    ssd      = 1
    discard  = "ignore"
  }
  scsihw   = "virtio-scsi-pci"
  bootdisk = "scsi0"

  network {
    model  = "virtio"
    bridge = "vmbr0"
    tag    = "1"
  }

  # cloud-init settings
  sshkeys   = file("/home/$USER/.ssh/id_ed25519.pub")
  ipconfig0 = "ip=192.168.1.100/24,gw=192.168.1.1"
}

BPG Proxmox Provider #

Works with Proxmox VE v8, see BPG Proxmox for additional configuration options.

# main.tf
terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = ">=0.39.0"
    }
  }
  # ADD BACKEND CONFIGURATION HERE
}

provider "proxmox" {
  endpoint  = "https://pve.example.com/api2/json"
  api_token = "terraform@pve!token=<TOKEN_VALUE>"
}

resource "proxmox_virtual_environment_vm" "vm" {
  node_name = "pve"
  vm_id     = 100
  name      = "vm-example"

  agent {
    enabled = true
  }

  clone {
    vm_id = 9000
    full  = true
  }

  cpu {
    cores = 1
    type  = "host"
  }

  memory {
    dedicated = 1024
    floating  = 1024
  }

  network_device {
    bridge  = "vmbr0"
    vlan_id = "1"
  }

  disk {
    datastore_id = "local-lvm"
    interface    = "scsi0"
    size         = 8
    file_format  = "raw"
    cache        = "writeback"
    iothread     = false
    ssd          = true
    discard      = "on"
  }

  # cloud-init settings
  initialization {
    user_account {
      keys = [file("~/.ssh/id_ed25519.pub")]
    }

    ip_config {
      ipv4 {
        address = "192.168.1.100/24"
        gateway = "192.168.1.1"
      }
    }
  }

  # Cloud-init SSH keys will cause a forced replacement, see:
  # https://github.com/bpg/terraform-provider-proxmox/issues/373
  lifecycle {
    ignore_changes = [initialization["user_account"], ]
  }
}

Troubleshooting #

Errors Using MinIO #

Initializing the backend...
Error refreshing state: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.

This error will occur when running terraform init and is caused by an invalid MinIO access key. Confirm that the access key is correct in the GUI and try again.

Error: Retrieving AWS account details: AWS account ID not previously found and failed retrieving via all available methods.

This is caused by Terraform attempting to contact the AWS IAM API to pull an account id, starting with Terraform v1.6+ you will need to set skip_requesting_account_id = true in your s3 backend config.

mc: <ERROR> Unable to make bucket `local/staging`. Access Denied.
mc: <ERROR> Unable to enable versioning. Access Denied.
mc: <ERROR> Remote bucket `local/staging` does not support locking

If you copied the MinIO bucket code from this post, a new-line character may have been added at the end of code in some terminals. Delete it and re-run the commands.

Errors Using Postgres #

pq: SSL is not enabled on the server

This error is due to a missing SSL certificate on the Postgres server. Add sslmode=disable to the connection string, conn_str, to skip the SSL check and connect via HTTP. Alternatively, you can configure PostgreSQL to use TLS see this GitHub Gist and Postgres Docs: SSL Support for more information.

References #

Terraform:

• Terraform
• Terraform State
• Terraform Documentation
  • State Locking
• Terraform Docs: Postgres Backend
• Terraform Docs: S3 Backend
  • DynamoDB State Locking

MinIO:

• MinIO
• MinIO Documentation
  • MinIO Docs: User Management
  • MinIO Docs: Bucket Versioning
  • MinIO Docs: MinIO Object Locking
  • MinIO Docs: Data Encryption (SSE)
  • MinIO Docs: Network Encryption (TLS)
• MinIO Client
• Docker Hub: minio/minio
• Docker Hub: bitnami/minio

PostgreSQL:

• PostgreSQL
• PostgreSQL Documentation
  • Postgres Docs: Advisory Locks
  • Postgres Docs: pg_locks
  • Postgres Docs: SSL Support
• Docker Hub: postgresql/postgres