RHEL/CENTOS 7 builds randomly failing

Hello,
We have an AWS Codebuild job that had been working fine for about a year, then started randomly failing. Some days it works, somedays it fails. Our RHEL 8 jobs which use same format are working fine.

So the jist of it: We start packer to build our RHEL 7 AMI. It starts fine, we install some packages, and do a yum update. Then we do an init 6, wait, and then log back in to perform some additional tasks. It’s during this mid build reboot that the intermittent failure occurs…
Looking at the debug output, this is what’s happening, notice the 16 minute gap in the log files…

2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [ERROR] Remote command exited without exit status or exit signal.
2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 2300218
2022/01/12 13:13:21 [INFO] 318739 bytes written for ‘stdout’
2022/01/12 13:13:21 [INFO] 0 bytes written for ‘stderr’
2022/01/12 13:13:21 [INFO] RPC client: Communicator ended with: 2300218
2022/01/12 13:13:21 [INFO] RPC endpoint: Communicator ended with: 2300218
2022/01/12 13:13:21 packer-provisioner-shell plugin: [INFO] 0 bytes written for ‘stderr’
2022/01/12 13:13:21 packer-provisioner-shell plugin: [INFO] 318739 bytes written for ‘stdout’
2022/01/12 13:13:21 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 2300218
2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: ‘EOF’, attempting reconnect
2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [DEBUG] handshake complete!
2022/01/12 13:13:21 packer-builder-amazon-ebs plugin: [INFO] no local agent socket, will not connect agent
2022/01/12 13:13:22 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: rm -f /tmp/script_5223.sh
2022/01/12 13:13:22 [INFO] RPC client: Communicator ended with: 0
2022/01/12 13:13:22 [INFO] RPC endpoint: Communicator ended with: 0
2022/01/12 13:13:22 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2022/01/12 13:13:22 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2022/01/12 13:13:22 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/12 13:13:22 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: rm -f
2022/01/12 13:13:22 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2022/01/12 13:13:22 [INFO] RPC client: Communicator ended with: 0
2022/01/12 13:13:22 [INFO] RPC endpoint: Communicator ended with: 0
2022/01/12 13:13:22 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2022/01/12 13:13:22 [INFO] (telemetry) ending shell
2022/01/12 13:13:22 [INFO] (telemetry) Starting provisioner shell
==> amazon-ebs: Pausing 10s before the next provisioner…
==> amazon-ebs: Provisioning with shell script: /packer/packer-shell4185226075
2022/01/12 13:13:32 packer-provisioner-shell plugin: Opening /packer/packer-shell4185226075 for reading
2022/01/12 13:13:32 packer-provisioner-shell plugin: [INFO] 2706 bytes written for ‘uploadData’
2022/01/12 13:13:32 [INFO] 2706 bytes written for ‘uploadData’
2022/01/12 13:13:32 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
***** Notice there is a 16 minute gap here!!!
2022/01/12 13:29:18 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: 'ssh: unexpected packet in response to channel open: ', attempting reconnect
2022/01/12 13:29:18 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/12 13:29:18 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
2022/01/12 13:29:18 packer-builder-amazon-ebs plugin: [DEBUG] handshake complete!
2022/01/12 13:29:18 packer-builder-amazon-ebs plugin: [INFO] no local agent socket, will not connect agent
2022/01/12 13:29:19 packer-builder-amazon-ebs plugin: [DEBUG] Starting remote scp process: scp -vt /tmp
2022/01/12 13:29:19 packer-builder-amazon-ebs plugin: [DEBUG] Started SCP session, beginning transfers…
2022/01/12 13:29:19 packer-builder-amazon-ebs plugin: [DEBUG] Copying input data into temporary file so we can read the length
2022/01/12 13:29:19 packer-provisioner-shell plugin: Retryable error: Error uploading script: Error copying input data into local temporary file. Check that TEMPDIR has enough space. Please see https://www.packer.io/docs/other/environment-variables#tmpdirfor more info. Error: connection reset
2022/01/12 13:29:19 [INFO] (telemetry) ending shell
==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present…

This is what a successful build looks like:
amazon-ebs: + init 6
2022/01/10 13:11:24 packer-builder-amazon-ebs plugin: [ERROR] Remote command exited without exit status or exit signal.
2022/01/10 13:11:24 [INFO] 303047 bytes written for ‘stdout’
2022/01/10 13:11:24 [INFO] 0 bytes written for ‘stderr’
2022/01/10 13:11:24 [INFO] RPC client: Communicator ended with: 2300218
2022/01/10 13:11:24 [INFO] RPC endpoint: Communicator ended with: 2300218
2022/01/10 13:11:24 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 2300218
2022/01/10 13:11:24 packer-provisioner-shell plugin: [INFO] 303047 bytes written for ‘stdout’
2022/01/10 13:11:24 packer-provisioner-shell plugin: [INFO] 0 bytes written for ‘stderr’
2022/01/10 13:11:24 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 2300218
2022/01/10 13:11:24 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/10 13:11:24 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: ‘EOF’, attempting reconnect
2022/01/10 13:11:24 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/10 13:11:24 packer-builder-amazon-ebs plugin: [ERROR] reconnection error: dial tcp 10.211.191.163:22: connect: connection refused
2022/01/10 13:11:24 packer-provisioner-shell plugin: Retryable error: Error removing temporary script at /tmp/script_964.sh: dial tcp 10.211.191.163:22: connect: connection refused
2022/01/10 13:11:26 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/10 13:11:26 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: ‘client not available’, attempting reconnect
2022/01/10 13:11:26 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/10 13:11:41 packer-builder-amazon-ebs plugin: [ERROR] reconnection error: dial tcp 10.211.191.163:22: i/o timeout
2022/01/10 13:11:41 packer-provisioner-shell plugin: Retryable error: Error removing temporary script at /tmp/script_964.sh: dial tcp 10.211.191.163:22: i/o timeout
2022/01/10 13:11:43 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/10 13:11:43 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: ‘client not available’, attempting reconnect
2022/01/10 13:11:43 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/10 13:11:43 packer-builder-amazon-ebs plugin: [ERROR] reconnection error: dial tcp 10.211.191.163:22: connect: connection refused
2022/01/10 13:11:43 packer-provisioner-shell plugin: Retryable error: Error removing temporary script at /tmp/script_964.sh: dial tcp 10.211.191.163:22: connect: connection refused
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: ‘client not available’, attempting reconnect
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [DEBUG] handshake complete!
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [INFO] no local agent socket, will not connect agent
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: rm -f /tmp/script_964.sh
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2022/01/10 13:11:45 [INFO] RPC client: Communicator ended with: 0
2022/01/10 13:11:45 [INFO] RPC endpoint: Communicator ended with: 0
2022/01/10 13:11:45 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: rm -f
2022/01/10 13:11:45 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2022/01/10 13:11:45 [INFO] RPC client: Communicator ended with: 0
2022/01/10 13:11:45 [INFO] RPC endpoint: Communicator ended with: 0
2022/01/10 13:11:45 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2022/01/10 13:11:45 [INFO] (telemetry) ending shell
2022/01/10 13:11:45 [INFO] (telemetry) Starting provisioner shell
==> amazon-ebs: Pausing 10s before the next provisioner…
==> amazon-ebs: Provisioning with shell script: /packer/packer-shell3624160795
2022/01/10 13:11:55 packer-provisioner-shell plugin: Opening /packer/packer-shell3624160795 for reading
2022/01/10 13:11:55 packer-provisioner-shell plugin: [INFO] 2706 bytes written for ‘uploadData’
2022/01/10 13:11:55 [INFO] 2706 bytes written for ‘uploadData’
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] Starting remote scp process: scp -vt /tmp
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] Started SCP session, beginning transfers…
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] Copying input data into temporary file so we can read the length
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] scp: Uploading script_5662.sh: perms=C0644 size=2706
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] SCP session complete, closing stdin pipe.
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] Waiting for SSH session to complete.
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] scp stderr (length 32): Sink: C0644 2706 script_5662.sh
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: chmod 0755 /tmp/script_5662.sh
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2022/01/10 13:11:55 [INFO] RPC client: Communicator ended with: 0
2022/01/10 13:11:55 [INFO] RPC endpoint: Communicator ended with: 0
2022/01/10 13:11:55 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2022/01/10 13:11:55 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session

Any idea what’s going on?