Packer + Vault - tmp ssh key location correction

We are trying to build a packer base for template creation in vmware. Packer with built out integration with Hashi Vault.

I have 80% of the updated keys directed to vault. Vault seems to pull down keys / values as environment shell variables. Then packer just pulls against those when it populates fields as it runs. But the deployment of ubuntu 20 (as example) fails as the post deployment runs and it calls "/tmp/ansible-key4165246055"


==> vsphere-iso.linux-ubuntu: Pausing after run of step 'StepConnect'. Press enter to continue. ==> vsphere-iso.linux-ubuntu: Pausing before the next provisioner . Press enter to continue. 
==> vsphere-iso.linux-ubuntu: Provisioning with Ansible...
    vsphere-iso.linux-ubuntu: Not using Proxy adapter for Ansible run:
    vsphere-iso.linux-ubuntu:   Using ssh keys from Packer communicator...
==> vsphere-iso.linux-ubuntu: Executing Ansible: ansible-playbook -e packer_build_name="linux-ubuntu" -e packer_builder_type=vsphere-iso -e packer_http_addr=172.13.100.146:8046 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars display_skipped_hosts=false --extra-vars PACKER_BUILDER_USERNAME=<sensitive> --extra-vars PACKER_BUILDER_KEY_PUBLIC=***** --extra-vars PACKER_ANSIBLE_USER_USERNAME=<sensitive> --extra-vars PACKER_ANSIBLE_KEY_PUBLIC=***** --extra-vars ANSIBLE_USER='<sensitive>' --extra-vars ANSIBLE_SECRET=***** --extra-vars ANSIBLE_PASSWORD_ENCRYPTED=***** -e ansible_ssh_private_key_file=/tmp/ansible-key4165246055 -i /tmp/packer-provisioner-ansible444158725 /home/nerd/git/gitlab/ps-lab-vmware/packer/ansible/main.yml
    vsphere-iso.linux-ubuntu:
    vsphere-iso.linux-ubuntu: PLAY [all] *********************************************************************
    vsphere-iso.linux-ubuntu:
    vsphere-iso.linux-ubuntu: TASK [Gathering Facts] *********************************************************
    vsphere-iso.linux-ubuntu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '172.13.100.178' (ED25519) to the list of known hosts.\r\nLoad key \"/tmp/ansible-key4165246055\": error in libcrypto\r\n<sensitive>@172.13.100.178: Permission denied (publickey,password).", "unreachable": true}
    vsphere-iso.linux-ubuntu:

From another shell I can see that the file is empty

nerd@laptop:~/git/gitlab/ps-lab-vmware$ cat /tmp/ansible-key4165246055

nerd@laptop:~/git/gitlab/ps-lab-vmware$ 

Question:

  1. Where in the packer build structure does it set that variable so I can replace with the correct. in linux-ubuntu.pkr.hcl → Ansible stanza calls key: " “–extra-vars”, “PACKER_ANSIBLE_KEY_PUBLIC=‘${local.packer_ansible_key_public}’”, But if this is populating "/tmp/ansible-key4165246055" … I think something is wrong

I am running into a very similar issue. I am attempting to use the ansible provisioner with the proxy disabled in order to use a reboot command in a CentOS 7 build. I receive a very similar error in my logs. When I cat the temporary key created via Packer, it is blank. I have also attempted to create my own keys and pass them into ssh_host_key_file and ssh_authorized_key_file, but the files were not overriding how Packer set the ansible extra vars.

If I use the proxy, everything connects fine. I simply need to connect to the machine WITHOUT proxying traffic through Packer.

==> vsphere-iso.linux_iso: Provisioning with Ansible...
    vsphere-iso.linux_iso: 	Using ssh keys from Packer communicator...
    vsphere-iso.linux_iso: Not using Proxy adapter for Ansible run:
    vsphere-iso.linux_iso: 	Using ssh keys from Packer communicator...
2022/10/10 20:59:19 packer-provisioner-ansible plugin: Creating inventory file for Ansible run...
==> vsphere-iso.linux_iso: Executing Ansible: ansible-playbook -e packer_build_name="linux_iso" -e packer_builder_type=vsphere-iso -e packer_http_addr=10.32.8.253:0 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars repos_path=/runner_dir/image-pipeline/image-pipeline/repos image_name=c7-it-advanced-v0.0.0-20221010 date=20221010 platform=vmware os_distro=c7 app_type=it image_type=advanced --tags vmware --skip-tags level2 -vvv -e ansible_ssh_private_key_file=/tmp/ansible-key3491725608 -i /tmp/packer-provisioner-ansible2398177769 /runner_dir/image-pipeline/image-pipeline/packer_provisioners/site.yml
    vsphere-iso.linux_iso: ansible-playbook 2.9.22
    vsphere-iso.linux_iso:   config file = None
    vsphere-iso.linux_iso:   configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    vsphere-iso.linux_iso:   ansible python module location = /runner_dir/image-pipeline/image-pipeline/venv/ansible2.9/lib/python3.10/site-packages/ansible
    vsphere-iso.linux_iso:   executable location = /runner_dir/image-pipeline/image-pipeline/venv/ansible2.9/bin/ansible-playbook
    vsphere-iso.linux_iso:   python version = 3.10.6 (main, Aug  3 2022, 07:09:11) [GCC 9.4.0]
    vsphere-iso.linux_iso: No config file found; using defaults
    vsphere-iso.linux_iso: host_list declined parsing /tmp/packer-provisioner-ansible2398177769 as it did not pass its verify_file() method
    vsphere-iso.linux_iso: script declined parsing /tmp/packer-provisioner-ansible2398177769 as it did not pass its verify_file() method
    vsphere-iso.linux_iso: auto declined parsing /tmp/packer-provisioner-ansible2398177769 as it did not pass its verify_file() method
    vsphere-iso.linux_iso: Parsed /tmp/packer-provisioner-ansible2398177769 inventory source with ini plugin
    vsphere-iso.linux_iso: Skipping callback 'actionable', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'counter_enabled', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'debug', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'dense', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'dense', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'full_skip', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'json', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'minimal', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'null', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'oneline', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'selective', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'skippy', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'stderr', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'unixy', as we already have a stdout callback.
    vsphere-iso.linux_iso: Skipping callback 'yaml', as we already have a stdout callback.
    vsphere-iso.linux_iso:
    vsphere-iso.linux_iso: PLAYBOOK: site.yml *************************************************************
    vsphere-iso.linux_iso: 1 plays in /runner_dir/image-pipeline/image-pipeline/packer_provisioners/site.yml
    vsphere-iso.linux_iso:
    vsphere-iso.linux_iso: PLAY [Packer Provisioners] *****************************************************
    vsphere-iso.linux_iso:
    vsphere-iso.linux_iso: TASK [Gathering Facts] *********************************************************
    vsphere-iso.linux_iso: task path: /runner_dir/image-pipeline/image-pipeline/packer_provisioners/site.yml:1
    vsphere-iso.linux_iso: <10.228.40.140> ESTABLISH SSH CONNECTION FOR USER: root
    vsphere-iso.linux_iso: <10.228.40.140> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/tmp/ansible-key3491725608"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 '-o IdentitiesOnly=yes' -o ControlPath=/root/.ansible/cp/fe9ac226fc 10.228.40.140 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
    vsphere-iso.linux_iso: <10.228.40.140> (255, b'', b'Warning: Permanently added \'10.228.40.140\' (ECDSA) to the list of known hosts.\r\nLoad key "/tmp/ansible-key3491725608": invalid format\r\nroot@10.228.40.140: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n')
    vsphere-iso.linux_iso: fatal: [default]: UNREACHABLE! => {
    vsphere-iso.linux_iso:     "changed": false,
    vsphere-iso.linux_iso:     "msg": "Failed to connect to the host via ssh: Warning: Permanently added '10.228.40.140' (ECDSA) to the list of known hosts.\r\nLoad key \"/tmp/ansible-key3491725608\": invalid format\r\nroot@10.228.40.140: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).",
    vsphere-iso.linux_iso:     "unreachable": true
    vsphere-iso.linux_iso: }
    vsphere-iso.linux_iso:
    vsphere-iso.linux_iso: PLAY RECAP *********************************************************************
    vsphere-iso.linux_iso: default                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
    vsphere-iso.linux_iso:
2022/10/10 20:59:20 [INFO] (telemetry) ending ansible
==> vsphere-iso.linux_iso: Provisioning step had errors: Running the cleanup provisioner, if present...