Unable to ansible provision with ssh key only with ssh_pass and become_pass

When creating my template vm on proxmox to provision it with ansible it fails to connect with ssh key. I can get it to work by passing the ssh_pass and become_pass to the extra arguments.

Packer: 1.7.8
Ansible: 2.12.1
Python: 3.10.1
Proxmox: 7.1-4

proxmox.ubuntu-2004: Executing Ansible: ansible-playbook -e packer_build_name="ubuntu-2004" -e packer_builder_type=proxmox -e packer_http_addr=10.0.2.57:8698 --ssh-extra-args '-o IdentitiesOnly=yes' -v --extra-vars ansible_python_interpreter=/usr/bin/python -e ansible_ssh_private_key_file=/tmp/ansible-key4158079046 -i /tmp/packer-provisioner-ansible1476003741 /home/alteredtech/Coding/infrastructure/packer/ubuntu-20.04/playbook/packer-ubuntu-20-04.yml
    proxmox.ubuntu-2004: Using /home/alteredtech/Coding/infrastructure/packer/ubuntu-20.04/playbook/ansible.cfg as config file
    proxmox.ubuntu-2004:
    proxmox.ubuntu-2004: PLAY [Provision Image] *********************************************************
    proxmox.ubuntu-2004:
    proxmox.ubuntu-2004: TASK [Gathering Facts] *********************************************************
    proxmox.ubuntu-2004: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '10.0.2.199' (ED25519) to the list of known hosts.\r\nLoad key \"/tmp/ansible-key4158079046\": invalid format\r\npacker@10.0.2.199: Permission denied (publickey,password).", "unreachable": true}
    proxmox.ubuntu-2004:
    proxmox.ubuntu-2004: PLAY RECAP *********************************************************************
    proxmox.ubuntu-2004: default                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
    proxmox.ubuntu-2004:

Here is the full pastebin of the packer log.
This is my main pkr.hcl file.
My variables pkr.hcl file.
Then my preseed.cfg file.

I asked this question on reddit and someone did get it working with their own system but we are not using the same starting point. They are using clone while I am using an iso. Then the only other difference is they are using a vault file so they might have info on that that might be passing the ssh_pass and become_pass that I am unaware of.

Reading the documentation I shouldn’t have to pass the ssh_pass and become_pass since I am already providing that info in the source. Or is that where I am wrong and the ansible playbook does need the ssh_pass and become_pass when using something other than cloud images since their user is the root user while on custom images you are not root and as such needs to be passed those parameters?

I got it working like this. Not sure if this is the best way but it is using the SSH_key and ssh_password (even though it should be pulling this from packer source) and not needing the become password.

provisioner "shell" {
    execute_command = "echo 'packer' | sudo -S -E bash '{{ .Path }}'"
    script = "scripts/setup.sh"
  }

  provisioner "ansible" {
    extra_arguments  = ["-v", "-e ansible_ssh_pass=${var.ssh_info["password"]}"]
    ansible_env_vars = ["ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_CONFIG=./playbook/ansible.cfg"]
    playbook_file    = "./playbook/${var.ansible_play}.yml"
    use_proxy        = false
  }

The setup script is like so.

#!/bin/bash

# Enable "packer" user sudo without password
echo "packer ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/packer

Now the next thing to do is have ansible set it so that password based auth is not allowed, add my key to ssh so I can ssh into future vm from my computers.