Ansible provisioner

I would like to use the ansible (not local) provisioner with a qemu builder.
I see just about every question results in someone advising to use the ansible-local which means installing ansible into the image first. This is really counter to what I like about ansible, which is NO agent needed.

I followed the packer docs for the ansible provisioner and set up a json. When the image is done, and reboots, ansible errors out because it is waiting for a sudo password.

There doesnt seem to be a way for me to tell the ansible provisioner this, and the docs seem to suggest a temporary ssh key is created for doing this anyway.

I’m sure I am missing something about how the provisioner works, but cannot figure out what it is, whether it is a Debian thing or a qemu thing or?

Example file below that I use regularly to keep an updated Ubuntu AMI on AWS. Hope this helps. Playbook file location is in relation to the packer.json file. This is ran locally on my Mac with Ansible also installed locally.

{
  "variables": {
    "AWS_ACCESS_KEY_ID": "",
    "AWS_SECRET_ACCESS_KEY": ""
  },
  "provisioners": [
    {
      "type": "ansible",
      "playbook_file": "../ansible/baseline.yml"
    }
  ],
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "",
    "secret_key": "",
    "region": "us-east-1",
    "source_ami": "ami-07d0cf3af28718ef8",
    "instance_type": "t2.micro",
    "ssh_username": "ubuntu",
    "force_deregister": "true",
    "force_delete_snapshot": "true",
    "ami_name": "FullMesh-Template-{{timestamp}}"
  }]