Shell script not executing fully but don't see non zero exit code

Packer
1

Hi,
I have a basic shell script provisioner. It doesn’t seem to execute completely. using latest packer version. Why do I don’t see echo messages in the log output. If I login into packer instance with -debug, I can execute the shell script to completion without issue.
Thanks a lot for your time and advise.
Venkatesh

{
builder {…}
{
“type”: “shell”,
“script”: “./agent_install.sh”
},
}

agent_install.sh script content is
#!/usr/bin/bash -e
set -ex
echo “cd /var/mps/mps_images”
cd /var/mps/mps_images
echo “tar xvzf build-masagent-13.0-66.33.tgz”
tar xvzf build-masagent-13.0-66.33.tgz
chmod +x /var/mps/mps_images/installmasagent
echo “/var/mps/mps_images/installmasagent”
/var/mps/mps_images/installmasagent
echo “done executing agent_install *******”

Debug output with PACKER_LOG=1

==> amazon-ebs: Provisioning with shell script: ./agent_install.sh
2020/09/03 12:21:03 packer-provisioner-shell plugin: Opening ./agent_install.sh for reading
2020/09/03 12:21:03 [INFO] 334 bytes written for ‘uploadData’
2020/09/03 12:21:03 packer-provisioner-shell plugin: [INFO] 334 bytes written for ‘uploadData’
2020/09/03 12:21:03 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2020/09/03 12:21:03 packer-builder-amazon-ebs plugin: [DEBUG] Starting remote scp process: scp -vt /tmp
2020/09/03 12:21:03 packer-builder-amazon-ebs plugin: [DEBUG] Started SCP session, beginning transfers…
2020/09/03 12:21:03 packer-builder-amazon-ebs plugin: [DEBUG] Copying input data into temporary file so we can read the length
2020/09/03 12:21:03 packer-builder-amazon-ebs plugin: [DEBUG] scp: Uploading script_5593.sh: perms=C0644 size=334
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] SCP session complete, closing stdin pipe.
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] Waiting for SSH session to complete.
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] scp stderr (length 31): Sink: C0644 334 script_5593.sh
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: chmod 0755 /tmp/script_5593.sh
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:04 [INFO] RPC client: Communicator ended with: 0
2020/09/03 12:21:04 [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:04 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: chmod +x /tmp/script_5593.sh; PACKER_BUILDER_TYPE=‘amazon-ebs’ PACKER_BUILD_NAME=‘amazon-ebs’ /tmp/script_5593.sh
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:04 [INFO] 0 bytes written for ‘stdout’
2020/09/03 12:21:04 [INFO] 0 bytes written for ‘stderr’
2020/09/03 12:21:04 [INFO] RPC client: Communicator ended with: 0
2020/09/03 12:21:04 [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:04 packer-provisioner-shell plugin: [INFO] 0 bytes written for ‘stdout’
2020/09/03 12:21:04 packer-provisioner-shell plugin: [INFO] 0 bytes written for ‘stderr’
2020/09/03 12:21:04 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: rm -f /tmp/script_5593.sh
2020/09/03 12:21:04 [INFO] RPC client: Communicator ended with: 0
2020/09/03 12:21:04 [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:04 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
2020/09/03 12:21:04 packer-builder-amazon-ebs plugin: [DEBUG] starting remote command: rm -f
2020/09/03 12:21:05 packer-builder-amazon-ebs plugin: [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:05 [INFO] RPC client: Communicator ended with: 0
2020/09/03 12:21:05 [INFO] RPC endpoint: Communicator ended with: 0
2020/09/03 12:21:05 [INFO] (telemetry) ending shell
2020/09/03 12:21:05 [INFO] (telemetry) Starting provisioner shell
2020/09/03 12:21:05 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0

2

With interactive mode (-debug) I can execute the same uploaded script in /tmp directory without any issues and it runs every command successfully. With Ubuntu source AMI, same script works fine as well but difference b/w freebsd and ubuntu is that in Ubuntu I see execution commands and logs with PACKER_LOG=1 With freebsd there is nothing (even for other basic unix commands).
Has it got to do with execute_command?
chmod +x {{ .Path }}; env {{ .Vars }} {{ .Path }
I dont’ understand but I tried the above but didn’t make any difference. The bash is the default shell on our source AMI.

The bash is at /usr/bin/bash unlike on ubuntu. the version of bash is 3.2.57(1)
packer version used is 1.6.1 on mac

3

It could just be that with freeBSD the tty isn’t working to pipe the standard out back to Packer; have you verified that the script definitely isn’t running, for example by creating a “~/finished.txt” file at the end of your script that you can check for?

4

Thanks for the tips. I tried doing touch file before and after execution. Still I didn’t see any file created. Definitely the script wasn’t executing as the untaring takes some time and is followed by reboot. Initially I though the default shell was sh but it wasn’t and some home grown which behaves like sh. I tried changing default shell for the login, but I couldn’t as it is stripped down os. Adding it didn’t help as the code seems to overwrite /etc/passwd file and change the shell. I couldn’t add user as well thinking I will create an account with bash shell.

Any tips around execute_command. Thanks

5

Sure. execute_command is a template engine. so the stuff in those curly braces are variables that get replaced with builder information. {{.Path}} represents the path to the script where Packer uploaded it on the remote machine, and {{.Vars}} represents any environment variables you have set, or that Packer sets for you.

The default execute_command should be chmod +x {{ .Path }}; {{ .Vars }} {{ .Path }}. Are you adding the “env”? If not where are you seeing it?

In your logs, I see that the execute_command gets interpolated to chmod +x /tmp/script_5593.sh; PACKER_BUILDER_TYPE=‘amazon-ebs’ PACKER_BUILD_NAME=‘amazon-ebs’ /tmp/script_5593.sh, and this is what gets called. That looks correct to me. Maybe your homegrown sh relative doesn’t like that style of variable declaration? You could set the execute_command to not use the {{.Vars}} – something super simple like “execute_command”: “{{ .Path }}” to start. Ideally you want to make execute_command mimic exactly what you ran from the instance when you called --debug.

6

Thanks once again for a clear explanation. Will try execute_command as suggested. However I tried running this directly in -debug mode
“chmod +x /tmp/script_5593.sh; PACKER_BUILDER_TYPE=‘amazon-ebs’ PACKER_BUILD_NAME=‘amazon-ebs’ /tmp/script_5593.sh”

This worked. This script takes couple of minutes. So can’t understand from the log 2020/09/03 12:21:04 that it instantaneously executed it (in my recent run 1 second) and completed it and deleted the the script in /tmp directory

7

one interesting observation is /tmp/script_5593.sh has 0644 permission during debug after it completed that shell provisioner. There was chmod 0755 /tmp/script_5593.sh but didn’t appear to have taken effect. I am also little confused as at 2020/09/03 12:21:04 it is calling “rm -f /tmp/script_5593.sh”. but when it reaches the interactive debug after this step, when I login into packer build I still see the file /tmp/script_5593.sh but with 0644 permission. Is it marked for deletion but not deleted? if I reboot the file is no longer there.

8

Maybe the packer user doesn’t have permission to modify and remove that file.

It’s probably gone after the reboot because it is in the tmp directory.

What are the permissions on the script locally?

9

Thanks once again for helping me. The permission of the local script is 755 and after it is uploaded it is 644. There is a umask of 022. I changed local script to 777 so that after upload it becomes 755. However it didn’t make the difference. How do I execute the uploaded file without packer wrapping the command into a temp script? Like I do using terraform remote shell executioner?

The packer user is same as I login to the packer instance being created and I use same credentials. There is only 1 user for login. I still see the file in /tmp directory after it complete that provisioner. Agree, after reboot (which gets triggered after runnign script manually or even if I do reboot without running it deletes the file). so still can’t explain it can’t do chmod +x or delete file in that provisioner step.