CentOS7 qemu VM Stuck After Configured

Hello,

I’ve been trying to create a basic CentOS7 qcow2 image. Packer spins up a VM from the ISO specified, and I can connect to the running instance thru VNC. When the VM gets to the point where it runs the post instructions, the VM just stops there, even with no post instructions in the kickstart file. Eventually Packer times out and I get:

==> kvm: Timeout waiting for SSH.

I assume the VM is supposed to reboot for Packer to ssh in. Any suggestions would be greatly appreciated.

Packer version: 1.5.1
OS: Ubuntu 18.04 LT

Packer template:

{
“variables”: {
“build_number”: “{{timestamp}}”,
“centos_arch”: “x86_64”,
“iso_url”: “http://mirror.cs.vt.edu/pub/CentOS/7.7.1908/isos/x86_64/CentOS-7-x86_64-Everything-1908.iso”,
“iso_checksum_url”: “http://mirror.cs.vt.edu/pub/CentOS/7.7.1908/isos/x86_64/sha256sum.txt”,
“iso_checksum_type”: “sha256”,
“username”: “root”,
“password”: “temp12345”,
“timeout”: “40m”,
“headless”: “true”,
“http”: “http”,
“ks_file”: “ks.cfg”,
“disk_size”: “22528”,
“memory”: “5120”,
“cpus”: “4”
},
“builders”: [
{
“name”: “kvm”,
“type”: “qemu”,
“iso_url”: “{{ user iso_url}}”,
“iso_checksum_url”: “{{user iso_checksum_url}}”,
“iso_checksum_type”: “{{user iso_checksum_type}}”,
“ssh_username”: “{{user username}}”,
“ssh_password”: “{{user password}}”,
“ssh_timeout”: “{{user timeout}}”,
“headless”: “{{user headless}}”,
“shutdown_command”: “shutdown -P now”,
“http_directory”: “{{ user http }}”,
“boot_command”: [
" text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/{{user ks_file}}"
],
“disk_size”: “{{ user disk_size }}”,
“memory”: “{{ user memory }}”,
“cpus”: “{{ user cpus }}”
}
]
}

Kickstart file:

# Install OS instead of upgrade
install
# Keyboard layouts
keyboard 'us'
# Root password
rootpw temp12345
# System timezone
timezone America/New_York
# Reboot after installation
reboot --eject
# poweroff
# System language
lang en_US.UTF-8
# Firewall configuration
firewall --enable --ssh
# System authorization information
auth  --useshadow  --passalgo=sha512
# Use graphical install
#graphical
text
firstboot --disabled
# SELinux configuration
selinux --disabled
# Networking
network  --bootproto=dhcp --onboot=on --device=eth0 --ipv6=auto
# Do not configure the X Window System
skipx
# Accept EULA without user interaction
eula --agreed
# System bootloader configuration
bootloader --location=mbr
# Partition clearing information
clearpart --all
zerombr
# Disk partitioning information
autopart
# User setup
user --name=centos --groups=centos --password=centos
# Services
services --enabled=NetworkManager,sshd
# Packages
%packages --nobase
@core
-*firmware
-b43-openfwwf
-efibootmgr
-audit*
-libX*
-fontconfig
-freetype
sudo
openssh-clients
openssh-server
make
perl
kernel-firmware
kernel-devel
%end

Can you please provide a logfile using
PACKER_LOG=1 PACKER_LOG_PATH=packer.log packer build your-template.json?

The log file is below. I’ve removed most of the attempts to connect thru ssh, but kept the top and the bottom portion of the log. I did cancel the build, so you’ll see those log entries reflect that towards the bottom. Thanks for the help!

packer.log.txt (16.3 KB)

Is it possible to root@127.0.0.1:2922 during build, and if, can you debug the network configuration in this session?

This hint is interesting, too:

%post
sed -i "s/#PermitRootLogin yes/PermitRootLogin yes/g" /etc/ssh/sshd_config
%end

Seems like a connection is established, but the authentication part is failing.

jorge@DESKTOP:~/Image-Build$ ssh -vvv -p 2922 -o PreferredAuthentications=password root@127.0.0.1
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /home/jorge/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving “127.0.0.1” port 2922
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2922.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jorge/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
ssh_exchange_identification: Connection closed by remote host

I looked for the /etc/sshd_config, but there seems to be only an anaconda file there (/etc/sshd_config.anaconda). I’m starting to think maybe my kickstart file is not complete…

Is it asking for a password? It seems the configuration is expecting a ssh-key authentication only.

Its not asking for a password when attempting to ssh in, it just fails. Even though I added the option for password authentication, and set the root password in the kickstart file. Yeah, seems like I may have to add steps to add a public key to /root/.ssh/authorized_keys, maybe as a %post directive?

Or try to figure out why it’s not pretending to expect a password. :wink: I’ve no experiences with Kickstart, because I am using presseding. But the %post-step sounds good.

Yea, i need to get better kickstart :slight_smile: I’ll see how the %post option goes. Thanks for the feedback!

You are welcome :+1: Hope we can fix that.

Ok, so after a couple of hours I found the issue. By default RedHat based systems do not setup ssh during installation by default. One must add the inst.sshd boot option which:

inst.sshd
Starts the sshd service during the installation, which allows you to connect to the system during the installation using SSH and monitor its progress. For more information on SSH, see the ssh(1) man page and the corresponding chapter in the Red Hat Enterprise Linux 7 System Administrator’s Guide. By default, sshd is only automatically started on IBM Z, and on other architectures, sshd is not started unless the inst.sshd option is used.

After updating my boot_command like this:

"boot_command": [
          "<tab> inst.sshd inst.text inst.ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/{{user `ks_file`}}<enter><wait>"
        ],

Packer was able to ssh in. But, it connects prematurely and doesn’t give the installation a chance to complete…

So, I’m testing with enabling ssh during the %post phase in the kickstart file, acting as a sort of latch for Packer to ssh in.

Post installation tasks

%post
mv /etc/ssh/sshd_config.anaconda /etc/ssh/sshd_config
systemctl start sshd
%end

This was a fun one :stuck_out_tongue_winking_eye:

1 Like

Congrats!

Concerning the premature connect: https://packer.io/docs/templates/communicator.html#pause_before_connecting
Maybe you could give it an additional try. :blush:

1 Like

after making the changes to the boot command in json file : inst.sshd
still i am getting an error “==> vsphere-iso: Waiting for SSH to become available…
2020/04/10 05:52:53 packer-builder-vsphere-iso plugin: [INFO] Waiting for SSH, up to timeout: 3m0s
2020/04/10 05:53:08 packer-builder-vsphere-iso plugin: [DEBUG] TCP connection to SSH ip/port failed: dial tcp 10.0.68.221:22: i/o timeout
2020/04/10 05:53:28 packer-builder-vsphere-iso plugin: [DEBUG] TCP connecti”

i have made the “pause_before_connecting=10 min” and add the post installation steps as below but same issu
%post
mv /etc/ssh/sshd_config.anaconda /etc/ssh/sshd_config
systemctl start sshd
%end

can i have the kickstart file as example to see where i am messing it up.

my kickstart file ia as below:

Basic Configuration

lang en_US

keyboard fr

cmdline

eula --agreed

timezone Europe/Paris --isUtc

network --onboot yes --device ens192 --bootproto static --ip=10.0.68.221 --netmask=255.255.255.0 --gateway= 10.0.68.1 --nameserver=172.30.0.150 --hostname=centos.local --noipv6

services --enabled=iptables,rsyslog,sshd,NetworkManager

user --groups=wheel --homedir=/home/itops --name=itops --iscrypted --password=$1$c2dnIt5L$UsKvFG/u0Mzh7VBHfOR3c0

rootpw --iscrypted $1$c2dnIt5L$UsKvFG/u0Mzh7VBHfOR3c0

#platform x86, AMD64, or Intel EM64T

reboot

Installation

text

cdrom

BootLoader

bootloader --location=mbr --timeout=10 --append=“rhgb quiet crashkernel=auto”

zerombr

sshpw --username root $1$c2dnIt5L$UsKvFG/u0Mzh7VBHfOR3c0 --iscrypted

Partition

clearpart --all --initlabel --drives=sda

autopart --type=lvm

Auth

auth --passalgo=sha512 --useshadow

Disable selinux & Fw

selinux --disabled

firewall --disabled

No GUI

skipx

firstboot --disable

Post installation

%post --nochroot --interpreter=/bin/bash

yum install -y vim

mv /etc/ssh/sshd_config.anaconda /etc/ssh/sshd_config

systemctl start sshd

%end

Packages

%packages

@^minimal

@core

-usbutils

%end

reboot --eject

Hi!

So adding that extra flag to the boot command did allow Packer to log in, but it turns out that it is still the incorrect way of doing things because that allowed ssh access prematurely (before Anaconda configured the system). What actually ended up fixing my problem was running my Packer build on a physical system, rather than in a virtualized system such as an EC2 instance. When I setup Packer on a physical system my Packer config and kickstart worked nicely! I ended up adjusting a couple of things, but that was my actual fix. I think it had to do with the availability of actual kvm support rather than qemu which is software based.

My kickstart file is below:
# Install OS instead of upgrade
install

# Keyboard layouts
keyboard us

# Root password
rootpw temp12345

# System timezone
timezone --utc UTC

# Reboot after installation
reboot --eject

# System language
lang en_US.UTF-8

# Firewall configuration
firewall --enable --ssh

# System authorization information
auth  --useshadow  --passalgo=sha512

# Use text install
text

# Dont' start the Initial Setup application upon first sytem boot
firstboot --disabled

# SELinux configuration
selinux --permissive

# Networking
network  --bootproto=dhcp --onboot=on --device=eth0 --ipv6=auto

# Do not configure the X Window System
skipx

# Accept EULA without user interaction
eula --agreed

# System bootloader configuration
bootloader --location=mbr

# Partition clearing information
clearpart --all
zerombr

# Partition setup
part /boot --fstype=xfs --size 500
part pv.01 --fstype=xfs --size 1 --grow

# LVM setup
volgroup System pv.01
logvol swap           --vgname=System --fstype=swap --name=swap --recommended
logvol /home          --vgname=System --fstype=xfs --size=1907 --name=home
logvol /tmp           --vgname=System --fstype=xfs --size=1907 --name=tmp
logvol /var           --vgname=System --fstype=xfs --size=4768 --name=var
logvol /var/log       --vgname=System --fstype=xfs --size=4768 --name=var_log
logvol /var/log/audit --vgname=System --fstype=xfs --size=1907 --name=var_log_audit
logvol /var/tmp       --vgname=System --fstype=xfs --size=1907 --name=var_tmp
logvol /              --vgname=System --fstype=xfs --size=1 --grow --name=root

# User setup
# TODO: Test if no password is set and if adding a key still work
user --name=centos --groups=centos

# Services
services --enabled=NetworkManager,sshd

# Packages
%packages
@^Basic Web Server
%end

%post --log=/root/ks-post.log

yum install -y python3 epel-release
pip3 install passlib

yum install -y python2-pip
pip install --upgrade pip
pip install passlib
%end

My Packer template is below:

{
“variables”: {
“build_number”: “{{timestamp}}”,
“centos_arch”: “x86_64”,
“iso_url”: “http://mirror.cs.vt.edu/pub/CentOS/7.7.1908/isos/x86_64/CentOS-7-x86_64-Everything-1908.iso”,
“iso_checksum_url”: “http://mirror.cs.vt.edu/pub/CentOS/7.7.1908/isos/x86_64/sha256sum.txt”,
“iso_checksum_type”: “sha256”,
“ssh_username”: “root”,
“ssh_password”: “{{ env PACKER_SSH_PASSWORD }}”,
“timeout”: “30m”,
“headless”: “true”,
“http”: “http”,
“ks_file”: “ks.cfg”,
“disk_size”: “22528”,
“memory”: “5120”,
“cpus”: “4”
},
“sensitive-variables”: [“ssh_password”],
“builders”: [
{
“name”: “STIG-Base”,
“type”: “qemu”,
“accelerator”: “kvm”,
“iso_url”: “{{ user iso_url}}”,
“iso_checksum_url”: “{{user iso_checksum_url}}”,
“iso_checksum_type”: “{{user iso_checksum_type}}”,
“ssh_username”: “{{user ssh_username}}”,
“ssh_password”: “{{ user ssh_password }}”,
“ssh_timeout”: “{{user timeout}}”,
“headless”: “{{user headless}}”,
“shutdown_command”: “shutdown -P now”,
“http_directory”: “{{ user http }}”,
“boot_command”: [
" inst.text inst.ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/{{user ks_file}}"
],
“disk_size”: “{{ user disk_size }}”,
“memory”: “{{ user memory }}”,
“cpus”: “{{ user cpus }}”,
“vnc_port_min”: 5957,
“vnc_port_max”: 5957
}
]
}

pause_before_connecting

actually is a setting to tell Packer to wait that configured amount of time after its initial connection by the way. your issue seems to be that the initial ssh connection is not even established. have you tried increasing

ssh_timeout

to give Packer the longer than default 10m to keep trying ssh?

I had to do this because the initial setup took a bit longer than 10m.

let me increase the ssh_timeout and see if that helps