Terraform Enterprise: Custom provider is "found" but then "not found"

We are using Terraform Cloud and I have a custom provide that we are using and I’m building it and including it our git repo.

Our file structure looks like this:

├── README.md
├── docker-compose.yaml
├── respositories.tf
├── terraform.d
│   └── plugins
│       ├── darwin_amd64
│       │   └── terraform-provider-cyral_v0.1.0
│       └── linux_amd64
│           └── terraform-provider-cyral_v0.1.0
└── terraform.tf

When I run this locally in a docker container it executes without and issues. When I run it in Terraform Enterprise, I get the following error:
Error: Failed to instantiate provider "cyral" to obtain schema: fork/exec /terraform/terraform.d/plugins/linux_amd64/terraform-provider-cyral_v0.1.0: no such file or directory

Terraform v0.12.24
Configuring remote state backend...
Initializing Terraform configuration...
2020/05/20 00:17:04 [DEBUG] Using modified User-Agent: Terraform/0.12.24 TFC/d33daf519f
2020/05/20 00:17:04 [DEBUG] Using modified User-Agent: Terraform/0.12.24 TFC/d33daf519f
2020/05/20 00:17:04 [INFO] Terraform version: 0.12.24  
2020/05/20 00:17:04 [INFO] Go runtime version: go1.12.13
2020/05/20 00:17:04 [INFO] CLI args: []string{"/usr/local/bin/terraform", "plan", "-out", "/terraform/terraform.tfplan", "-detailed-exitcode", "-var-file=/terraform/terraform.tfvars", "-lock=false", "-parallelism=10"}
2020/05/20 00:17:04 [DEBUG] Attempting to open CLI config file: /tmp/cli.tfrc
2020/05/20 00:17:04 Loading CLI configuration from /tmp/cli.tfrc
2020/05/20 00:17:04 [DEBUG] Using modified User-Agent: HashiCorp Terraform/0.12.24 (+https://www.terraform.io) TFC/d33daf519f
2020/05/20 00:17:04 [INFO] CLI command args: []string{"plan", "-out", "/terraform/terraform.tfplan", "-detailed-exitcode", "-var-file=/terraform/terraform.tfvars", "-lock=false", "-parallelism=10"}
2020/05/20 00:17:04 [DEBUG] Using modified User-Agent: Terraform/0.12.24 TFC/d33daf519f
2020/05/20 00:17:04 [DEBUG] Using modified User-Agent: HashiCorp Terraform/0.12.24 (+https://www.terraform.io) TFC/d33daf519f
2020/05/20 00:17:04 [TRACE] Meta.Backend: built configuration for "remote" backend with hash value 504897656
2020/05/20 00:17:04 [TRACE] Preserving existing state lineage "12f318d2-7de7-124b-a77f-c6d85bf58dd7"
2020/05/20 00:17:04 [TRACE] Preserving existing state lineage "12f318d2-7de7-124b-a77f-c6d85bf58dd7"
2020/05/20 00:17:04 [TRACE] Meta.Backend: working directory was previously initialized for "remote" backend
2020/05/20 00:17:04 [TRACE] Meta.Backend: using already-initialized, unchanged "remote" backend configuration
2020/05/20 00:17:04 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2020/05/20 00:17:04 [TRACE] HTTP client GET request to https://app.terraform.io/.well-known/terraform.json
2020/05/20 00:17:05 [TRACE] Meta.Backend: instantiated backend of type *remote.Remote
2020/05/20 00:17:05 [DEBUG] checking for provider in "."
2020/05/20 00:17:05 [DEBUG] checking for provider in "/usr/local/bin"
2020/05/20 00:17:05 [DEBUG] checking for provider in "terraform.d/plugins/linux_amd64"
2020/05/20 00:17:05 [DEBUG] found provider "terraform-provider-cyral_v0.1.0"
2020/05/20 00:17:05 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2020/05/20 00:17:05 [DEBUG] found valid plugin: "cyral", "0.1.0", "/terraform/terraform.d/plugins/linux_amd64/terraform-provider-cyral_v0.1.0"
2020/05/20 00:17:05 [DEBUG] checking for provisioner in "."
2020/05/20 00:17:05 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/05/20 00:17:05 [DEBUG] checking for provisioner in "terraform.d/plugins/linux_amd64"
2020/05/20 00:17:05 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2020/05/20 00:17:05 [TRACE] Meta.Backend: backend *remote.Remote supports operations
2020/05/20 00:17:05 [INFO] backend/local: starting Plan operation
2020/05/20 00:17:05 [TRACE] backend/local: requesting state manager for workspace "default"
2020/05/20 00:17:05 [TRACE] backend/local: requesting state lock for workspace "default"
2020/05/20 00:17:05 [TRACE] backend/local: reading remote state for workspace "default"
2020/05/20 00:17:05 [TRACE] backend/local: retrieving local state snapshot for workspace "default"
2020/05/20 00:17:05 [TRACE] backend/local: building context for current working directory
2020/05/20 00:17:05 [DEBUG] backend/local: Skipping interactive prompts for variables because input is disabled
2020/05/20 00:17:05 [TRACE] terraform.NewContext: starting
2020/05/20 00:17:05 [TRACE] terraform.NewContext: resolving provider version selections
2020/05/20 00:17:05 [TRACE] terraform.NewContext: loading provider schemas
2020/05/20 00:17:05 [TRACE] LoadSchemas: retrieving schema for provider type "cyral"
2020-05-20T00:17:05.564Z [INFO]  plugin: configuring client automatic mTLS
2020-05-20T00:17:05.600Z [DEBUG] plugin: starting plugin: path=/terraform/terraform.d/plugins/linux_amd64/terraform-provider-cyral_v0.1.0 args=[/terraform/terraform.d/plugins/linux_amd64/terraform-provider-cyral_v0.1.0]

Error: Failed to instantiate provider "cyral" to obtain schema: fork/exec /terraform/terraform.d/plugins/linux_amd64/terraform-provider-cyral_v0.1.0: no such file or directory

I can see that it successfully finds the plugin.

And I have double-checked that the permissions are correct and it’s executable and give that it works in a docker image, I’m pretty sure it’s correctly built as linux_amd64. It’s build with the following env vars GOOS=linux GOARCH=amd64

Thanks

Hi @greenkiwi,

I have seen a very similar case, where the repository was using GitHub LFS and therefore the provider plugin file was not being properly downloaded into the worker, but it was a placeholder instead.

Are you using something similar in your setup ?

If this is not the case, can you share with me a repo, which exhibits the failure, that I can clone and connect to TFC, so I can poke around ?

Regards,
Filip

Hi

I believe we have provided a suitable information on the ticket open:
https://hashicorp.zendesk.com/agent/tickets/28331

I hope you are able to find whats wrong based on what we did share.

update, seems the plugin was dynamic linked to libc musl, and not static linked, this was causing the issue on TFC

Thanks
Alvaro.

1 Like

Hi Filip,

We are encountering this exact issue (git LFS seems to be not properly cloning on the worker). Any chance you were able to get past this with some kind of TFC configuration change?

Thanks,
Pat

related to: Git LFS on Terraform Cloud

For Terraform Cloud or Terraform Enterprise issues, please open a ticket with support.