Cannot get the Ansible provisioner to work on Alma/RHEL

Hey all, hoping for some inspiration here… I have a collection of Packer scripts which happily create VMware templates using the vsphere-iso builder and I can configure some bits and pieces using the file provisioner and shell provisioner, but I just cannot get the Ansible (not ansible-local!) provisioner to work at all.

I already figured out that the ssh server command is different on RedHat/clones:

/usr/libexec/openssh/sftp-server -e

… but I can’t see any evidence that it is actually spinning up an ssh server at all. I also couldn’t see any evidence that it was creating the files/folders that it said it was, in /home/[user]/.ansible/tmp, including the AnsiballZ_setup.py that it wants to copy to the remote server.

I’ve set local_tmp and remote_tmp variables for ansible, but I guess that’s irrelevant if it’s not getting as far as setting up the ssh server…

Any advice how I can troubleshoot why the ssh server isn’t working gratefully received. I’ve turned on verbose logging, but that’s not helping. I don’t want to revert to the ansible-local provisioner, as this should work!

Steve

Interestingly, I found a log with this in it:

debug1: Will attempt key: /tmp/ansible-key3017709399 explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: KCM:)\n\n
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible\nNo Kerberos credentials available (default cache: KCM:)\n\n
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /tmp/ansible-key3017709399
Load key “/tmp/ansible-key3017709399”: error in libcrypto
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
nisgadmin@10.213.53.55: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

The “Load key “/tmp/ansible-key3017709399”: error in libcrypto” looks significant to me…

In fact, /tmp/ansible-key3017709399 is an empty file. And why is it in /tmp?

Ah well, the empty file is a known issue:

… and you can’t use ssh_authorized_key_file either, that’s broken too:

I give up. I have spent weeks trying to get this to work. Unless someone can provide me with a code snippet in HCL that definitely works I’m going to say that this just doesn’t work on RedHat & it’s derivatives.