Local in-house Providers fails with Failed to request discovery document: Get https://x.y.z/.well-known/terraform.json

I have a local in-house provider that is in the following location

/home/terraform/.terraform.d/plugins/myregistry.mycompany.com/cam/camc/0.2.5/linux_amd64/terraform-provider-camc_v0.2.5

The .terraformrc looks like this. We use filesystem_mirror to pull the locally installed provider if available first. If not found locally we would like the engine to go out and fetch the provider from the remote registry.

plugin_cache_dir = “$HOME/.terraform.d/plugin-cache”
disable_checkpoint = true
provider_installation {
filesystem_mirror {
path = “/home/terraform/.terraform.d/plugins”
}
direct {}
}

I have a template with following provider declaration

terraform {
required_version = “>= 0.13”

required_providers {
camc = {
source = “myregistry.mycompany.com/cam/camc
version = “0.2.5”
}
}
}

When I execute this with 0.13.7 or 1.0.1, I see the following error

Could not retrieve the list of available versions for provider
myregistry.mycompany.com/cam/camc: could not connect to myregistry.mycompany.com: Failed to
│ request discovery document: Get
│ “https://myregistry.mycompany.com/.well-known/terraform.json”: dial tcp: lookup
registry.ibm.com on 172.30.0.10:53: read udp
│ 10.254.14.54:46577->172.30.0.10:53: i/o timeout

Since we want the engine to use the provider from local filesystem_mirror we do not have any real registry server set up at location myregistry.mycompany.com. From the trace statement below the engine does find the provider in filemirror_location, then why does it go out to look for terraform.json. Is there a way to prevent this as we do not want to set a registry server.

2022-01-27T17:51:54.858Z [TRACE] getproviders.SearchLocalDirectory: found myregistry.mycompany.com/cam/camc v0.2.5 for linux_amd64 at /home/terraform/.terraform.d/plugins/myregistry.mycompany.com/cam/camc/0.2.5/linux_amd64

Thank you for your time.

Please find below the full trace log:

2022-01-27T17:51:54.848Z [DEBUG] Adding temp file log sink: /tmp/terraform-log946948586 2022-01-27T17:51:54.849Z [INFO] Terraform version: 1.0.1

2022-01-27T17:51:54.849Z [INFO] Go runtime version: go1.16.4

2022-01-27T17:51:54.849Z [INFO] CLI args: string{“/home/terraform/bin/terraform_1.0.1”, “init”, “-backend=false”, “-input=false”, “-get=true”} 2022-01-27T17:51:54.849Z [TRACE] Stdout is not a terminal

2022-01-27T17:51:54.849Z [TRACE] Stderr is not a terminal 2022-01-27T17:51:54.849Z [TRACE] Stdin is not a terminal 2022-01-27T17:51:54.849Z [DEBUG] Attempting to open CLI config file: /stacks/stackjobs/temp_c5727820-7f99-11ec-be06-0d380f39153d/.terraformrc

2022-01-27T17:51:54.849Z [INFO] Loading CLI configuration from /stacks/stackjobs/temp_c5727820-7f99-11ec-be06-0d380f39153d/.terraformrc 2022-01-27T17:51:54.850Z [DEBUG] checking for credentials in “/stacks/stackjobs/temp_c5727820-7f99-11ec-be06-0d380f39153d/.terraform.d/plugins”

2022-01-27T17:51:54.850Z [DEBUG] Explicit provider installation configuration is set

2022-01-27T17:51:54.850Z [TRACE] Selected provider installation method cliconfig.ProviderInstallationFilesystemMirror(“/home/terraform/.terraform.d/plugins”) with includes and excludes

2022-01-27T17:51:54.850Z [TRACE] Selected provider installation method cliconfig.ProviderInstallationDirect with includes and excludes

2022-01-27T17:51:54.852Z [INFO] Checkpoint disabled. Not running.

2022-01-27T17:51:54.852Z [INFO] CLI command args: string{“init”, “-backend=false”, “-input=false”, “-get=true”}

2022-01-27T17:51:54.857Z [TRACE] Meta.Backend: backend has not previously been initialized in this working directory

2022-01-27T17:51:54.857Z [TRACE] backend/local: state manager for workspace “default” will: - read initial snapshot from terraform.tfstate - write new snapshots to terraform.tfstate - create any backup at terraform.tfstate.backup

2022-01-27T17:51:54.857Z [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate

2022-01-27T17:51:54.857Z [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that’s okay

2022-01-27T17:51:54.857Z [TRACE] statemgr.Filesystem: read nil snapshot

2022-01-27T17:51:54.858Z [TRACE] getproviders.SearchLocalDirectory: found myregistry.mycompany.com/cam/camc v0.2.5 for linux_amd64 at /home/terraform/.terraform.d/plugins/myregistry.mycompany.com/cam/camc/0.2.5/linux_amd64

2022-01-27T17:51:54.858Z [TRACE] getproviders.SearchLocalDirectory: found myregistry.mycompany.com/cam/ibmworkloaddeployer v1.0.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/myregistry.mycompany.com/cam/workloaddeployer/1.0.0/linux_amd64

2022-01-27T17:51:54.858Z [TRACE] getproviders.SearchLocalDirectory: found myregistry.mycompany.com/cam/icp-elk v0.2.3 for linux_amd64 at /home/terraform/.terraform.d/plugins/myregistry.mycompany.com/cam/icp-elk/0.2.3/linux_amd64

2022-01-27T17:51:54.858Z [TRACE] getproviders.SearchLocalDirectory: found myregistry.mycompany.com/cam/nsxv v1.0.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/myregistry.mycompany.com/cam/nsxv/1.0.1/linux_amd64

2022-01-27T17:51:54.858Z [TRACE] getproviders.SearchLocalDirectory: found myregistry.mycompany.com/cam/ucd v0.5.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/myregistry.mycompany.com/cam/ucd/0.5.1/linux_amd64

2022-01-27T17:51:54.859Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.138.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/aliyun/alicloud/1.138.0/linux_amd64

2022-01-27T17:51:54.859Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.33.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/aws/2.33.0/linux_amd64

2022-01-27T17:51:54.859Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.44.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/aws/2.44.0/linux_amd64

2022-01-27T17:51:54.859Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.62.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/aws/3.62.0/linux_amd64

2022-01-27T17:51:54.859Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.35.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/1.35.0/linux_amd64

2022-01-27T17:51:54.859Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.80.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/2.80.0/linux_amd64

2022-01-27T17:51:54.860Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.2.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/external/1.2.0/linux_amd64

2022-01-27T17:51:54.860Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.1.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/external/2.1.0/linux_amd64

2022-01-27T17:51:54.860Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.17.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/google/2.17.0/linux_amd64

2022-01-27T17:51:54.860Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.88.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/google/3.88.0/linux_amd64

2022-01-27T17:51:54.860Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.1.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/http/1.1.1/linux_amd64

2022-01-27T17:51:54.860Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.1.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/http/2.1.0/linux_amd64

2022-01-27T17:51:54.860Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.9.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/kubernetes/1.9.0/linux_amd64

2022-01-27T17:51:54.937Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.5.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/kubernetes/2.5.0/linux_amd64

2022-01-27T17:51:54.937Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.4.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/local/1.4.0/linux_amd64

2022-01-27T17:51:54.938Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.1.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/local/2.1.0/linux_amd64

2022-01-27T17:51:54.938Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.1.2 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/null/2.1.2/linux_amd64

2022-01-27T17:51:54.938Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.1.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/null/3.1.0/linux_amd64

2022-01-27T17:51:54.938Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.2.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/random/2.2.1/linux_amd64

2022-01-27T17:51:54.938Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.1.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/random/3.1.0/linux_amd64

2022-01-27T17:51:54.939Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.1.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/tls/2.1.1/linux_amd64

2022-01-27T17:51:54.939Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.1.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/tls/3.1.0/linux_amd64

2022-01-27T17:51:54.939Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.13.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/vsphere/1.13.0/linux_amd64

2022-01-27T17:51:54.939Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.18.3 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/vsphere/1.18.3/linux_amd64

2022-01-27T17:51:54.939Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.0.2 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/hashicorp/vsphere/2.0.2/linux_amd64

2022-01-27T17:51:54.939Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v0.31.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/ibm-cloud/ibm/0.31.0/linux_amd64

2022-01-27T17:51:54.939Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.14.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/ibm-cloud/ibm/1.14.0/linux_amd64

2022-01-27T17:51:54.940Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.32.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/ibm-cloud/ibm/1.32.1/linux_amd64

2022-01-27T17:51:54.940Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.2.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/nutanix/nutanix/1.2.1/linux_amd64

2022-01-27T17:51:54.940Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.39.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/tencentcloudstack/tencentcloud/1.39.0/linux_amd64

2022-01-27T17:51:54.941Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.17.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/terraform-provider-openstack/openstack/1.17.1/linux_amd64

2022-01-27T17:51:54.941Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.34.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/terraform-provider-openstack/openstack/1.34.1/linux_amd64

2022-01-27T17:51:54.941Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.43.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/terraform-provider-openstack/openstack/1.43.1/linux_amd64

2022-01-27T17:51:54.941Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.1.0 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/vmware/nsxt/3.1.0/linux_amd64

2022-01-27T17:51:54.942Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.2.4 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/vmware/nsxt/3.2.4/linux_amd64

2022-01-27T17:51:54.942Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v1.0.2 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/vmware/vra7/1.0.2/linux_amd64

2022-01-27T17:51:54.942Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v2.0.1 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/vmware/vra7/2.0.1/linux_amd64

2022-01-27T17:51:54.942Z [TRACE] getproviders.SearchLocalDirectory: found Terraform Registry v3.0.3 for linux_amd64 at /home/terraform/.terraform.d/plugins/registry.terraform.io/vmware/vra7/3.0.3/linux_amd64

2022-01-27T17:51:54.943Z [DEBUG] Service discovery for myregistry.mycompany.com at https://myregistry.mycompany.com/.well-known/terraform.json

2022-01-27T17:51:54.943Z [TRACE] HTTP client GET request to https://myregistry.mycompany.com/.well-known/terraform.json

Looks like adding exludes prevents the error. Is this the solution?

plugin_cache_dir = “$HOME/.terraform.d/plugin-cache”
disable_checkpoint = true
provider_installation {
filesystem_mirror {
path = “/home/terraform/.terraform.d/plugins”
}
direct {
exclude = [“myregistry.mycompany.com/asterix/asterix”]
}

Wondering why the engine calls “https://myregistry.mycompany.com/.well-known/terraform.json” for a provider found locally? Thank you.

Hi @bhadrim,

Indeed, the first configuration you shared tells Terraform to try searching both locations and take the newest matching version found across both of them.

Your second configuration with exclude tells Terraform to never try to contact the origin registry for that particular provider, which means that only the filesystem_mirror installation method would be available, as you wanted.

If you have no intention to run a real provider registry at myregistry.mycompany.com then you may prefer to exclude that entire host from being used as an origin registry by using the wildcard syntax:

plugin_cache_dir = “$HOME/.terraform.d/plugin-cache”
disable_checkpoint = true
provider_installation {
  filesystem_mirror {
    path = “/home/terraform/.terraform.d/plugins”
  }
  direct {
    exclude = [“myregistry.mycompany.com/*/*”]
  }
}

If you are running Terraform on macOS, Linux, or another Unix-like system (in other words, anything except Windows) then the directory you specified here is one of the implied local mirror directories, and so a different way to get the same result would be to leave the provider_installation block entirely unset, which will cause Terraform to generate a suitable provider installation configuration itself based on the contents of that directory.

Specifically, it will search /home/terraform/.terraform.d/plugins (and all of the other implied mirror directories, if present) and note which provider addresses it encounters in that search. In your case it will find all of the providers you saw mentioned in the “getproviders.SearchLocalDirectory: found …” lines in the log. Then it will generate an effective provider_installation block which has a filesystem_mirror block which only includes each of those providers, and a direct block which excludes each of those providers.

@apparentlymart Thank you for your prompt reply.