Terraform on Proxmox: one VM from code, with drift and a rebuild

Goal

So far I built the lab’s virtual machines with a shell script and configured them with Ansible. A script says what to do. Terraform says what should exist, compares that with what does exist, and changes only the difference. That is how fleets of machines are usually kept consistent.

I wanted to describe one virtual machine in Terraform, give Terraform only the permissions it needs, change the machine by hand to see how drift shows up, and destroy and rebuild it from code.

This is the fourth post in Part 4. It uses one throwaway machine. The three lab VMs were not touched.

Terms used in this post

These build on the earlier glossaries.

  • Infrastructure as code: describing machines and networks in files kept in version control, and creating them from those files.
  • Terraform: a tool that reads those files, compares them with the real infrastructure, and makes the changes.
  • Provider: a plugin that lets Terraform talk to one system. Here, one for the Proxmox API.
  • State: Terraform’s record of the objects it created. It is how it knows what exists.
  • Plan: the list of changes Terraform would make, shown before it makes them.
  • Drift: a difference between the real machine and the code, usually from a manual change.
  • API token: a credential for an API that can carry its own, smaller set of permissions.
  • Template: a prepared virtual machine that new machines are cloned from.
  • Cloud-init: the first-boot mechanism that sets a machine’s address, user and keys.

What I built

  • A separate Linux machine runs Terraform. It can reach the hypervisor’s API. The state file is local to it and is not in git.
  • A restricted identity. I created a dedicated user on the hypervisor with an API token, a resource pool, and a custom role. The role was granted on the pool, on the two storage areas, on the network zone, and on the one VM ID the code manages. I did not use the administrator account.
  • A template. I made one Debian 12 template by hand with the hypervisor’s command line, from the same cloud image the lab VMs use.
  • The code. Four small files: the provider and its version, the variables, one VM resource, and two outputs. The VM is cloned from the template, has 1 vCPU, 1 GB of memory and an 8 GB disk, and gets its address and SSH key from cloud-init. Its ID, 214, is the last three digits of its address.

How it works

How Terraform changes a VM Diagram: the code and state, the plan, the apply, and the VM it manages.

terraform plan asks the hypervisor what exists, compares that with the files, and prints the difference. terraform apply carries out a saved plan. The state file maps each resource in the code to the real object, so a second plan can say “no changes” or show exactly one field that differs.

Commands and results

  • First build. Applying took 10 seconds, including the clone and the start. The machine answered SSH a few seconds later.
  • No change. A second plan reported “No changes. Your infrastructure matches the configuration.”
  • Drift. I changed the VM’s memory to 2048 MB with the hypervisor’s command line. The next plan showed one change, dedicated = 2048 -> 1024, and applying it put the configuration back to 1024 MB.
  • A change in code. Setting 1536 MB in the variables made the plan show 1024 -> 1536 and nothing else.
  • Rebuild. I created a file on the machine, then destroyed and rebuilt it. Destroy and apply took 13 seconds together, and the machine answered SSH 26 seconds after the destroy started. The file was gone, because the disk was a fresh clone.
  • Scope of the token. With the token, a read of the managed VM worked. A read of the configuration of VM 300 and of the status of VM 301 returned HTTP 403, “Permission check failed”.

Challenges

  • Least privilege took iterations. The token needed permissions I did not expect: permission to use the network from the node (to query a download URL), and a permission on the exact VM ID, because creating a machine checked that path before the machine joined the pool. Each missing permission showed up as a 403 from one API call, and the messages named the privilege but not always the path that needed it. A tightly scoped automation identity is worth it, but expect to build it by testing.
  • A destroy that hung. My first terraform destroy waited for a graceful shutdown, and the machine has no guest agent. Sixteen minutes later I interrupted Terraform. The shutdown task on the hypervisor was still running and held a lock on the VM, so every later change to it failed with a lock timeout until I stopped the task by hand. The provider’s documentation describes this case. Without the guest agent, Proxmox uses ACPI for shutdown, and for some VMs the shutdown does not work and the VM gets stuck on destroying. It recommends stop_on_destroy = true, which stops the VM instead of shutting it down. I set stop_on_destroy, and destroys then took one to three seconds. In a fleet, anything that waits on a machine that never answers needs a timeout and a forced path.
  • The image import needed more access than the API gives. Terraform could not import a downloaded disk image with the API token alone, because the provider copies the disk over SSH as a privileged user. Giving automation root SSH to the hypervisor was a larger grant than I wanted, so I cloned from a template. That moves the image preparation outside Terraform, into a hand-made template, and the template is a thing that needs its own process in a fleet.
  • A rebuilt machine looks like an impostor. After a rebuild at the same address, SSH refused to connect because the host key had changed. That is the correct behaviour. In a fleet, rebuilds need the old host keys retired in the same step.

What this lab cannot show is remote state shared by a team, a review process for plans, or many machines in one apply.

Differences with a production setup

  • State. Mine is a local file on one machine. Teams keep state in a shared backend with locking, so two people cannot apply at once, and protect it, because it can hold secrets.
  • Review. In production a plan is usually produced by a pipeline, reviewed on a pull request, and applied by the pipeline. I ran everything by hand.
  • Existing machines. I did not import the three lab VMs into Terraform. Bringing machines that were built by hand under management is most of the real work.
  • Modules and scale. One VM resource is not a module with inputs for a rack of machines.
  • TLS. I turned off certificate verification for the hypervisor’s self-signed certificate. A real setup trusts a proper certificate.
  • Secrets. The token lives in a file with restricted permissions on one machine. A real setup would use a secrets manager and short-lived credentials.
  • Self-service. Some teams let developers request databases or buckets through Kubernetes objects with Crossplane, and a portal such as Backstage puts those requests behind a web page. I used neither, and one engineer ran Terraform by hand.
  • Delivery from git. A pull-based tool such as Argo CD or Flux applies what is in the repository to a cluster, which covers Kubernetes objects. Terraform covers machines and networks, and teams often use both.

FAQ

What is the difference between Terraform and Ansible?

Terraform creates and changes infrastructure objects such as machines and networks, and records them in state. Ansible configures what is inside a machine. In this lab, Terraform would create a VM and Ansible would install Slurm or k3s on it.

What is drift, and how does Terraform find it?

Drift is when the real object no longer matches the code. Terraform refreshes its view from the API when it plans, so a manual change appears as a difference to put back.

Why not just use the administrator account?

Because anything that has it can change or delete every machine. A token limited to one pool and one VM ID could not even read the other VMs.

Why use a template?

It makes new machines quick and identical. It also needs its own care: someone has to rebuild it when the base image changes.

Next

Take the quiz to test your understanding: Part 4 quiz

Part 4 closes with a Go command line tool for the platform.

Comments