Null_resource local_exec - terraform fails, log shows "scp stderr: "Sink: C0644 0 terraform_812172598.sh""

I’m using provisioner local-exec to execute two Ansible playbooks on two EC2 instances. The following code is used to launch the playbooks on both nodes when the variable MultiNode is set to true. I’m not sure if this is the correct way to have the playbooks execute on both nodes.

resource “null_resource” “ExecuteAnsible” {
count = var.MultiNode ? 2 : 1
provisioner “remote-exec” {
connection {
host = coalesce(aws_instance.X360Instance[count.index].public_ip, aws_instance.X360Instance[count.index].private_ip)
user = var.Secrets ? “${local.secrets.TargetUser}” : var.TargetUser
private_key = file(pathexpand("~/.ssh/id_rsa"))
}
inline = [“echo ‘connected!’”]
}

#Execute the PostgresInstaller ansible playbook on the target machine

provisioner “local-exec” {
command = <<EOF
aws --profile {var.profile} ec2 wait instance-status-ok --region {var.RegionMain} --instance-ids ${aws_instance.X360Instance[count.index].id}
ansible-playbook --extra-vars “@AnsiblePlaybooks/Variables.json” AnsiblePlaybooks/PostgresInstaller.yml
EOF
}

#Execute the X360Installer ansible playbook on the target machine

provisioner “local-exec” {
command = <<EOF
aws --profile {var.profile} ec2 wait instance-status-ok --region {var.RegionMain} --instance-ids ${aws_instance.X360Instance[count.index].id}
ansible-playbook --extra-vars “@AnsiblePlaybooks/Variables.json” AnsiblePlaybooks/X360Installer.yml
EOF
}
}

When I do terraform apply, the job actually completes successfully on both nodes, as you can see by the PLAY RECAP:

null_resource.ExecuteAnsible[1] (local-exec): PLAY RECAP *********************************************************************
null_resource.ExecuteAnsible[1] (local-exec): 18.212.63.140 : ok=51 changed=35 unreachable=0 failed=0 skipped=5 rescued=0 ignored=4
null_resource.ExecuteAnsible[1] (local-exec): 54.211.61.102 : ok=51 changed=35 unreachable=0 failed=0 skipped=5 rescued=0 ignored=4
null_resource.ExecuteAnsible[1]: Creation complete after 7m56s [id=468323852023118431]

│ Error: local-exec provisioner error

│ with null_resource.ExecuteAnsible[0],
│ on instances.tf line 219, in resource “null_resource” “ExecuteAnsible”:
│ 219: provisioner “local-exec” {

│ Error running command ’ aws --profile default ec2 wait instance-status-ok --region us-east-1 --instance-ids i-027f4dd2b57ed5797
│ ansible-playbook --extra-vars “@AnsiblePlaybooks/Variables.json” AnsiblePlaybooks/PostgresInstaller.yml
│ ': exit status 2. Output: r to register.",
│ “”,
│ “Waiting for process with pid 20101 to finish.”,
│ “Last metadata expiration check: 0:03:03 ago on Wed 25 May 2022 09:17:20 PM UTC.”,
│ “Package drpm-0.4.1-3.el8.x86_64 is already installed.”,
│ “Dependencies resolved.”,
│ “Nothing to do.”,
│ “Complete!”,
│ “Updating Subscription Management repositories.”,
│ “Unable to read consumer identity”,
│ “”,
│ “This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.”,
│ “”,
│ “Last metadata expiration check: 0:03:07 ago on Wed 25 May 2022 09:17:20 PM UTC.”,
│ “Package createrepo_c-libs-0.17.7-3.el8.x86_64 is already installed.”,
│ “Dependencies resolved.”,
│ “Nothing to do.”,
│ “Complete!”,
│ “Updating Subscription Management repositories.”,
│ “Unable to read consumer identity”,
│ “”,
│ “This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.”,
│ “”,
│ “Last metadata expiration check: 0:03:10 ago on Wed 25 May 2022 09:17:20 PM UTC.”,
│ “Package createrepo_c-0.17.7-3.el8.x86_64 is already installed.”,
│ “Dependencies resolved.”,
│ “Nothing to do.”,
│ “Complete!”
│ ]
PLAY RECAP *********************************************************************
│ 18.212.63.140 : ok=8 changed=2 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0
│ 54.211.61.102 : ok=8 changed=2 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

I can access both nodes, and see that the Playbooks ran correctly and everything installed. I can reach the application. But! the terraform fails. I ran the apply with debugging like " TF_LOG=DEBUG OS_DEBUG=true terraform apply". Tailing the terraform.log shows that the log just stops very early on with a scp stderr: “Sink: C0644 0 terraform_812172598.sh\n” ERROR even though the deployment keeps running. Really hoping @apparentlymart or one of the other devs have some idea, because without the errors recording to the log, I’m pretty lost

2022-05-25T17:19:47.321-0400 [DEBUG] opening new ssh session
2022-05-25T17:19:47.335-0400 [DEBUG] remote command exited with ‘0’: chmod 0777 /tmp/terraform_954179450.sh
2022-05-25T17:19:47.335-0400 [DEBUG] opening new ssh session
2022-05-25T17:19:47.339-0400 [DEBUG] starting remote command: /tmp/terraform_812172598.sh
2022-05-25T17:19:47.353-0400 [DEBUG] starting remote command: /tmp/terraform_954179450.sh
2022-05-25T17:19:47.386-0400 [DEBUG] remote command exited with ‘0’: /tmp/terraform_812172598.sh
2022-05-25T17:19:47.386-0400 [DEBUG] opening new ssh session
2022-05-25T17:19:47.390-0400 [DEBUG] remote command exited with ‘0’: /tmp/terraform_954179450.sh
2022-05-25T17:19:47.390-0400 [DEBUG] opening new ssh session
2022-05-25T17:19:47.395-0400 [DEBUG] Starting remote scp process: ‘scp’ -vt /tmp
2022-05-25T17:19:47.398-0400 [DEBUG] Starting remote scp process: ‘scp’ -vt /tmp
2022-05-25T17:19:47.404-0400 [DEBUG] Started SCP session, beginning transfers…
2022-05-25T17:19:47.405-0400 [DEBUG] Copying input data into temporary file so we can read the length
2022-05-25T17:19:47.406-0400 [DEBUG] Beginning file upload…
2022-05-25T17:19:47.407-0400 [DEBUG] Started SCP session, beginning transfers…
2022-05-25T17:19:47.407-0400 [DEBUG] Copying input data into temporary file so we can read the length
2022-05-25T17:19:47.408-0400 [DEBUG] Beginning file upload…
2022-05-25T17:19:47.427-0400 [DEBUG] SCP session complete, closing stdin pipe.
2022-05-25T17:19:47.427-0400 [DEBUG] Waiting for SSH session to complete.
2022-05-25T17:19:47.435-0400 [DEBUG] SCP session complete, closing stdin pipe.
2022-05-25T17:19:47.435-0400 [DEBUG] Waiting for SSH session to complete.
2022-05-25T17:19:47.436-0400 [ERROR] scp stderr: “Sink: C0644 0 terraform_954179450.sh\n”
2022-05-25T17:19:47.444-0400 [ERROR] scp stderr: “Sink: C0644 0 terraform_812172598.sh\n”
2022-05-25T17:20:40.013-0400 [WARN] Errors while provisioning null_resource.ExecuteAnsible[0] with “local-exec”, so aborting
2022-05-25T17:20:40.017-0400 [ERROR] vertex “null_resource.ExecuteAnsible[0]” error: local-exec provisioner error

TERRAFORM VERSION:
terraform --version
Terraform v1.2.1
on linux_amd64