Custom ssh connection fails with vagrant builder

I try without success to customize the ssh communicator details doing this :

source "vagrant" "base" {
  source_path = "./builds/"
  output_dir  = "./builds-dev"
  provider    = "libvirt"

  communicator = "ssh"
  ssh_username = "root"
  ssh_password = "changeme"

build {
  sources = ["source.vagrant.base"]

  provisioner "ansible" {
    playbook_file = "./ansible/playbook.yml"

packer (1.7.3) hangs forever waiting for ssh to become available.

I can log into the vm using the ssh details above and the systemctl status sshd reports invalid connections from the vagrant user.
Am I missing something obvious ? (I attempted to define the same thing in json, same result)