How to reboot VM with Packer

Packer
1

Hi,

This post extends my post from yesterday:

The VM where packer is running:

# packer --version
1.8.3

# qemu-system-x86_64 --version
QEMU emulator version 5.2.0 (Debian 1:5.2+dfsg-11+deb11u2)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

# uname -a
Linux packer02 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

I am trying to reboot the VM during provisioning. I am installing a newer Kernel and want to reboot to be able to remove the old one, which cannot be removed while running.
I tried multiple ways but unfortunately none of them worked for me.
I am using the latest Debian Bullseye ISO to create our private cloud image, and up to the reboot everything works fine.

Here is the relevant part of the hcl template:

  provisioner "shell" {
    environment_vars = [
      "BUILD_NAME=${var.build_name}",
      "IMAGE_DATE=${var.datestamp}.${var.build_id}"
    ]
    execute_command   = "echo 'vagrant' | {{ .Vars }} sudo -S -E bash '{{ .Path }}'"
    pause_before      = "2s"
    expect_disconnect = "true"
    scripts           = [
      "provision_scripts/grub.sh",
      "provision_scripts/apt.sh",
      "provision_scripts/kernel.sh",
      "provision_scripts/reboot.sh"
    ]
  }

  provisioner "shell" {
    environment_vars = [
      "BUILD_NAME=${var.build_name}",
      "IMAGE_DATE=${var.datestamp}.${var.build_id}"
    ]
    execute_command  = "echo 'vagrant' | {{ .Vars }} sudo -S -E bash '{{ .Path }}'"
    pause_before     = "60s"
    scripts          = [
      "provision_scripts/install.sh",
      "provision_scripts/configure.sh",
      "provision_scripts/setup.sh",
      "provision_scripts/cleanup.sh",
      "provision_scripts/zerodisk.sh"
    ]
  }

reboot.sh content:

#!/usr/bin/env bash
set -x
set -e

log()
{
  logger --tag "${1}"
}

log "Reboot and use the cloud Kernel"
#dpkg-reconfigure --frontend noninteractive openssh-server
#systemctl enable ssh.service
systemctl stop sshd.service
/usr/bin/nohup shutdown -r now &
#nohup shutdown -r now < /dev/null > /dev/null 2>&1 &
exit 0

Unfortunately packer breaks after the reboot without running the first script in the second provisioner with a Timeout during SSH handshake. The VM is up and running and sshd is listening to port 22 within 20 seconds after the reboot.

Relevant log:

==> qemu.debian_test_gnt: + /usr/bin/nohup shutdown -r now
2022/10/27 08:01:03 [INFO] 0 bytes written for 'stdout'
2022/10/27 08:01:03 [INFO] 199 bytes written for 'stderr'
2022/10/27 08:01:03 [INFO] RPC client: Communicator ended with: 0
2022/10/27 08:01:03 [INFO] RPC endpoint: Communicator ended with: 0
2022/10/27 08:01:03 packer-builder-qemu plugin: [INFO] RPC endpoint: Communicator ended with: 0
2022/10/27 08:01:03 packer-provisioner-shell plugin: [INFO] 0 bytes written for 'stdout'
2022/10/27 08:01:03 packer-provisioner-shell plugin: [INFO] 199 bytes written for 'stderr'
2022/10/27 08:01:03 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2022/10/27 08:01:03 [INFO] (telemetry) ending shell
2022/10/27 08:01:03 [INFO] (telemetry) Starting provisioner shell
==> qemu.debian_test_gnt: Pausing 1m0s before the next provisioner...
==> qemu.debian_test_gnt: Provisioning with shell script: provision_scripts/install.sh
2022/10/27 08:02:03 packer-provisioner-shell plugin: Opening provision_scripts/install.sh for reading
2022/10/27 08:02:03 packer-provisioner-shell plugin: [INFO] 824 bytes written for 'uploadData'
2022/10/27 08:02:03 [INFO] 824 bytes written for 'uploadData'
2022/10/27 08:02:03 packer-builder-qemu plugin: [DEBUG] Opening new ssh session
2022/10/27 08:02:03 packer-builder-qemu plugin: [ERROR] ssh session open error: 'EOF', attempting reconnect
2022/10/27 08:02:03 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/10/27 08:02:03 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2022/10/27 08:03:03 packer-provisioner-shell plugin: Retryable error: Error uploading script: Timeout during SSH handshake
2022/10/27 08:03:05 packer-provisioner-shell plugin: [INFO] 824 bytes written for 'uploadData'
2022/10/27 08:03:05 [INFO] 824 bytes written for 'uploadData'
2022/10/27 08:03:05 packer-builder-qemu plugin: [DEBUG] Opening new ssh session
2022/10/27 08:03:05 packer-builder-qemu plugin: [ERROR] ssh session open error: 'client not available', attempting reconnect
2022/10/27 08:03:05 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/10/27 08:03:05 packer-builder-qemu plugin: [DEBUG] handshaking with SSH

I tried the inline shell provisioner, rebooting from the kernel.sh and now with a dedicated script without success.

 Build 'qemu.debian_test_gnt' errored after 12 minutes 51 seconds: Error uploading script: Timeout during SSH handshake

Thanks

2

I am running into the exact same issue.

  provisioner "shell" {
    inline = ["reboot"]
    execute_command = "echo 'ubuntu' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    expect_disconnect = true
    pause_after = "60s"
  }

According to the docs expect_disconnect should prevent this issue. But as you and I have found, it doesn’t work as expected. (Bug?)

Try adding ssh_read_write_timeout to the build section .

ssh_read_write_timeout (duration string | ex: “1h5m2s”) - The amount of time to wait for a remote command to end. This might be useful if, for example, packer hangs on a connection after a reboot. Example: 5m . Disabled by default

Related