Qemu/Ubuntu/Cloud-Init SSH Auth issues

Supply an authorized-keys: block in your user-data.yaml under your ssh: block

ssh:
    install-server: true
    allow-pw: true
    authorized-keys:
      - ssh-rsa <encrypted_key> user@host

Fetch your key with:

cat ~/.ssh/id_rsa.pub

I fought this for about a month before giving up and moving to ssh.
Also, you shouldnt need anything in this block to get 22.04 up:

  packages:
    - qemu-guest-agent
    - cloud-initramfs-growroot
    - openssh-server

I found none of these packages to assist me.

I used the following for late commands:

    - "echo 'packer ALL=(ALL) NOPASSWD: ALL' > /target/etc/sudoers.d/test"
    - curtin in-target --target=/target -- chmod 440 /etc/sudoers.d/test