Packer unable to connect to proxmox vm over ssh

Packer
1

Hello,

I’m running packer (version 1.10.0) directly on the host of a proxmox (8.0.4) node with the following configuration:

// Define the variables required for Proxmox connection
variable "proxmox_token_id" {
  type        = string
  description = "Token ID for Proxmox API"
}

variable "proxmox_token_secret" {
  type        = string
  description = "Token Secret for Proxmox API"
}

variable "proxmox_url" {
  type    = string
  default = "https://pve.example.com:8006/api2/json"
}

variable "proxmox_username" {
  type    = string
  default = "root@pam!packer-builder"
}

// Configure Packer with the required Proxmox plugin
packer {
  required_plugins {
    proxmox = {
      version = "1.1.6"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

// Define the source configuration for cloning a Proxmox VM
source "proxmox-clone" "my-template" {
  proxmox_url               = var.proxmox_url
  username                  = var.proxmox_username
  token                     = var.proxmox_token_secret
  insecure_skip_tls_verify  = true
  node                      = "pve-training"
  vm_id                     = "109"
  clone_vm_id               = 1000
  full_clone                = true
  memory                    = 4096
  cores                     = 2
  sockets                   = 1
  qemu_agent                = true
  ssh_username              = "devops"
  ssh_timeout               = "20m"
  scsi_controller           = "virtio-scsi-pci"

  network_adapters {
    bridge = "vmbr1"
    model = "virtio"
  }

  ipconfig {
    ip = "10.13.13.26/24"
    gateway = "10.13.13.1"
  }

  nameserver = "9.9.9.9"
}

// Define the build configuration
build {
  sources = ["source.proxmox-clone.my-template"]
  provisioner "shell" {
    inline = ["touch /root/test"]
  }
}

This is an Ubuntu 22.04 cloud image and I’m provisioning a public key through cloud-init for the user “devops”.
Packer waits for the qemu guest agent to publish its IP and when this happens, packer gives this error:

2023/12/23 01:21:09 packer-plugin-proxmox_v1.1.6_x5.0_linux_amd64 plugin: 2023/12/23 01:21:09 [INFO] Attempting SSH connection to 10.13.13.26:22...
2023/12/23 01:21:09 packer-plugin-proxmox_v1.1.6_x5.0_linux_amd64 plugin: 2023/12/23 01:21:09 [DEBUG] reconnecting to TCP connection for SSH
2023/12/23 01:21:09 packer-plugin-proxmox_v1.1.6_x5.0_linux_amd64 plugin: 2023/12/23 01:21:09 [DEBUG] handshaking with SSH
2023/12/23 01:21:09 packer-plugin-proxmox_v1.1.6_x5.0_linux_amd64 plugin: 2023/12/23 01:21:09 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/12/23 01:21:09 packer-plugin-proxmox_v1.1.6_x5.0_linux_amd64 plugin: 2023/12/23 01:21:09 [DEBUG] Detected authentication error. Increasing handshake attempts.

In the ssh logs of the virtual machine I see:

Dec 23 01:23:05 ubuntu sshd[2231]: error: kex_exchange_identification: Connection closed by remote host
Dec 23 01:23:05 ubuntu sshd[2231]: Connection closed by 10.13.13.1 port 43494
Dec 23 01:23:05 ubuntu sshd[2232]: Connection closed by authenticating user devops 10.13.13.1 port 43504 [preauth]

10.13.13.1 is the proxmox node and where Packer is run from.
Connecting through ssh from the host as root using the devops user in the virtual machine works perfectly:

root@pve:~# ssh devops@10.13.13.26
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-86-generic x86_64)
[...]

Any ideas what’s going on? I’ve spent quite a lot of time on this and I’m failing to understand why Packer doesn’t work when everything else around ssh seems to be perfectly ok. Any help/suggestion is appreciated.

ssh client on proxmox is: 1:9.2p1-2+deb12u1
ssh server on the virtual machine is: 1:8.9p1-3ubuntu0.4

2

When adding -debug, I see that Packer seems to be creating ephemeral ssh keys for some reason instead of just using the existing private key to authenticate:

2023/12/23 01:34:33 packer-plugin-proxmox_v1.1.6_x5.0_linux_amd64 plugin: 2023/12/23 01:34:33 using token auth
==> proxmox-clone.my-template: Creating ephemeral key pair for SSH communicator...
==> proxmox-clone.my-template: Created ephemeral SSH key pair for communicator
    proxmox-clone.my-template: Saving communicator private key for debug purposes: my-template.pem

How can I stop it from doing this?

3

I realised that switching to password authentication works, but I need to make relatively ugly changes to the cloudimage through cloud-init, meaning:

runcmd:
  - sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
  - systemctl restart sshd

So if I then use ssh_password, Packer is able to connect. I still don’t understand what’s happening and why Packer isn’t using the ssh key provided for the root user.

4

Ok, so now I realise that Packer automatically changes the cloud-init drive in Proxmox and adds the user and the ssh key through this ephemeral ssh key pair (and it configures the networktoo, if this is set up in the hcl configuration file) – these settings can be added in the GUI in Proxmox and it’s “the short version” of cloud-init that Proxmox offers. The problem with that is that you cannot provision too much with that version in Proxmox, as it’s too simple. So, for example, you cannot install any packages, such as qemu-guest-agent.

So if you want to install qemu-guest-agent, you need to set a cicustom that points to a full-blown user-data configuration file. If you do that, the virtual machine will ignore the settings in the ‘short’ cloud-init version that Packer uses and it will just read whatever you’ve set in cicustom. Except for the network part, which in cloud-init is considered a separate source. So while you can still configure the network with Packer under these circumstances, the user-related settings (user name, ssh key) will be ignored.

So Packer creates this key pair that is simply unusable, because I need to use a cicustom to install qemu-guest-agent, as this is what Packer uses to identity the virtual machine’s IP.

So is there any way I can tell Packer to stop generating this silly ephemeral ssh key pair and use the existing one?