Packer KVM build issue- SSH handshake error: Timeout during SSH handshake

Packer
1

Hi All,

When I am trying to build new kvm image using qcow2 file, its getting stuck at
2024/04/03 01:06:43 ui: ESC[1;32m==> qemu: Waiting for SSH to become available…ESC[0m
2024/04/03 01:06:43 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:2857…
2024/04/03 01:06:43 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2024/04/03 01:06:43 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2024/04/03 01:07:43 packer-builder-qemu plugin: [DEBUG] SSH handshake err: Timeout during SSH handshake
2024/04/03 01:07:50 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:2857…
2024/04/03 01:07:50 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2024/04/03 01:07:50 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2024/04/03 01:08:50 packer-builder-qemu plugin: [DEBUG] SSH handshake err: Timeout during SSH handshake
2024/04/03 01:08:57 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:2857…
2024/04/03 01:08:57 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2024/04/03 01:08:57 packer-builder-qemu plugin: [DEBUG] handshaking with SSH

Troubleshooting steps followed:

  1. Manually spin up vm using qcow2 file
  2. Tested sshd service
  3. Disabled firewall
  4. Enable PermitRootLogin yes
  5. Provided correct user (root) and password(****) which I can use to access this vm
  6. after making changes shutdown vm and tested with updated qcow2 file, still its same error.
    Could you please suggest.

Here is my packer file:

{
“variables”: {
“ssh_passwd”: “******”
},

“builders”: [
{

  "type": "qemu",
  "format": "qcow2",
  "accelerator": "kvm",
  "vm_name": "oracle8-image_latest.qcow2",
  "qemu_binary": "/usr/libexec/qemu-kvm",
  "iso_urls": "/u02/tmp/oracle8-image.qcow2",
  "iso_checksum": "e84ceced009c15bf58b5c52f2375cc3bdddea1d2418216d3c6314903b825ffe5df02a89fbbd4",
  "output_directory": "/u02/tmp",
  "disk_image": "true",
  "disk_size": "60G",
  "use_backing_file": "false",
  "communicator": "ssh",
  "ssh_pty": "true",
  "ssh_username": "root",
  "ssh_password": "ssh_passwd",
  "ssh_wait_timeout": "30m",
  "shutdown_command": "echo '{{ user `ssh_passwd` }}' | sudo -S /sbin/halt -h -p",
  "headless": "true",
  "memory": "8192",
  "cpus": "2"
}

],
}

Note: i am able to access vm using same qcow2 file /u02/tmp/oracle8-image.qcow2 using virsh console with same root password.

Could anyone please suggest.

2

More detailed logs:

2024/04/03 05:03:25 packer-builder-qemu plugin: Looking for available communicator (SSH, WinRM, etc) port between 2222 and 4444
2024/04/03 05:03:25 packer-builder-qemu plugin: Found available port: 3483 on IP: 127.0.0.1
==> qemu: Found port for communicator (SSH, WinRM, etc): 3483.
==> qemu: Pausing after run of step ‘stepPortForward’. Press enter to continue.
2024/04/03 05:03:27 packer-builder-qemu plugin: Looking for available port between 5900 and 6000 on 127.0.0.1
==> qemu: Looking for available port between 5900 and 6000 on 127.0.0.1
2024/04/03 05:03:27 packer-builder-qemu plugin: Found available port: 5988 on IP: 127.0.0.1
2024/04/03 05:03:27 packer-builder-qemu plugin: Found available VNC port: 5988 on IP: 127.0.0.1
==> qemu: Pausing after run of step ‘stepConfigureVNC’. Press enter to continue.
2024/04/03 05:03:29 packer-builder-qemu plugin: Qemu --version output: QEMU emulator version 4.2.0 (qemu-kvm-4.2.0-59.module_el8.5.0+1063+c9b9feff.1)
2024/04/03 05:03:29 packer-builder-qemu plugin: Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
2024/04/03 05:03:29 packer-builder-qemu plugin: Qemu version: 4.2.0
==> qemu: Starting VM, booting disk image
qemu: view the screen of the VM, connect via VNC without a password to
qemu: vnc://127.0.0.1:5988
2024/04/03 05:03:29 packer-builder-qemu plugin: Qemu Builder has no floppy files, not attaching a floppy.
qemu: The VM will be run headless, without a GUI. If you want to
2024/04/03 05:03:29 packer-builder-qemu plugin: Executing /usr/libexec/qemu-kvm: string{“-device”, “virtio-net,netdev=user.0”, “-m”, “8192M”, “-drive”, “file=/u02/tmp/oracle8-image.qcow2,if=virtio,cache=writeback,discard=ignore,format=qcow2”, “-netdev”, “user,id=user.0,hostfwd=tcp::3483-:22”, “-machine”, “type=pc,accel=kvm”, “-name”, “oracle8-image_latest.qcow2”, “-smp”, “cpus=4,sockets=4”, “-vnc”, “127.0.0.1:88”, “-boot”, “c”}
qemu: view the screen of the VM, connect via VNC without a password to
qemu: vnc://127.0.0.1:5988
2024/04/03 05:03:29 packer-builder-qemu plugin: Started Qemu. Pid: 1012529
==> qemu: Pausing after run of step ‘stepRun’. Press enter to continue.
==> qemu: Pausing after run of step ‘stepConfigureQMP’. Press enter to continue.
==> qemu: Waiting 1m0s for boot…
==> qemu: Connecting to VM via VNC (127.0.0.1:5988)
2024/04/03 05:04:36 packer-builder-qemu plugin: Connected to VNC desktop: QEMU (oracle8-baselineimage_latest.qcow2)
==> qemu: Typing the boot command over VNC…
==> qemu: Pausing after run of step 'boot_command: '. Press enter to continue.
==> qemu: Pausing after run of step ‘stepTypeBootCommand’. Press enter to continue.
qemu: Not using a NetBridge – skipping StepWaitGuestAddress
==> qemu: Pausing after run of step ‘stepWaitGuestAddress’. Press enter to continue.
2024/04/03 05:04:57 packer-builder-qemu plugin: [INFO] Waiting for SSH, up to timeout: 30m0s
==> qemu: Using ssh communicator to connect: 127.0.0.1
==> qemu: Waiting for SSH to become available…
2024/04/03 05:04:57 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:3483…
2024/04/03 05:04:57 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2024/04/03 05:04:57 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2024/04/03 05:05:57 packer-builder-qemu plugin: [DEBUG] SSH handshake err: Timeout during SSH handshake
2024/04/03 05:06:04 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:3483…
2024/04/03 05:06:04 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2024/04/03 05:06:04 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2024/04/03 05:07:04 packer-builder-qemu plugin: [DEBUG] SSH handshake err: Timeout during SSH handshake
2024/04/03 05:07:11 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:3483…
2024/04/03 05:07:11 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2024/04/03 05:07:11 packer-builder-qemu plugin: [DEBUG] handshaking with SSH

3

Issue has been resolved.

Troubleshooting steps you need to follow:

  1. It takes time to boot vm from ISO file (it took almost 20 mins), so till vm is up it keeps giving error as SSH handshake failed. As soon as vm is up , error turns into Handshake complete “connected to SSH!”. So always check progress by connecting to vm console using vncviewer with port given in logs. For ex in my case it was 127.0.0.1:5988 and monitor progress.
  2. It could fail to get kickstart file, make sure path provided is correct
  3. Boot process could stuck, if there is any issue with kickstart file(like any wrong parameter is passed to command)