Failed to instantiate provider to obtain schema. Looking for tips to diagnose the problem

Upon running terraform plan I’m receiving these errors

Error: Failed to instantiate provider “archive” to obtain schema: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Error: Failed to instantiate provider “google” to obtain schema: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Error: Failed to instantiate provider “random” to obtain schema: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

The error message is a little vague which makes me think its masking the real problem. I turned on detailed logging (export TF_LOG=TRACE) and now I see:

2020/10/01 14:18:52 [TRACE] LoadSchemas: retrieving schema for provider type "random"
2020-10-01T14:18:52.973Z [DEBUG] plugin: plugin process exited: path=/root/.terraform.d/plugins/linux_amd64/terraform-provider-google_v3.37.0_x5 pid=1068
2020-10-01T14:18:52.973Z [INFO]  plugin: configuring client automatic mTLS
2020-10-01T14:18:53.007Z [DEBUG] plugin: starting plugin: path=/root/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4 args=[/root/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4]
2020-10-01T14:18:53.008Z [DEBUG] plugin: plugin started: path=/root/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4 pid=1075
2020-10-01T14:18:53.009Z [DEBUG] plugin: waiting for RPC address: path=/root/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4
2020-10-01T14:18:53.032Z [ERROR] plugin.terraform-provider-random_v2.3.0_x4: plugin init error: error="listen unix /tmp/plugin155583172: bind: input/output error" timestamp=2020-10-01T14:18:53.032Z
2020/10/01 14:18:53 [TRACE] LoadSchemas: retrieving schema for provider type "terraform"
2020-10-01T14:18:53.035Z [DEBUG] plugin: plugin process exited: path=/root/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4 pid=1075

So my underlying error seems to be

listen unix /tmp/plugin155583172: bind: input/output error

Haven’t got a clue where to go from here other than pasting that error into google and that hasn’t gotten me very far so far so wondering if anyone here might be able to help.

I don’t know the root cause here but I have figured out how to solve it. My terraform code is in a git repo which is hosted on our gitlab instance. It was cloned to /Users/username/git/gitlabinstancename/group/repo and the directory from which the terraform plan was being run a few other directories deep.

When I cloned the repo to /tmp everything started working. As I said, I don’t understand the root cause of the problem however I know that this fixed it so thought it was worth sharing.

Basically its an issue with the location at which I cloned the repo to and I suspect that the deep location in the directory hierarchy from where I was running the command was contributing to the problem.