Local Provider Installation not working in Terraform 13

piontekdd · March 2, 2021, 2:45pm

I have been struggling with getting a local provider working with Terraform 13. I’ve done a lot of google searches, reading various bugs and fixes etc, but still not able to get this working. The provider is not in the registry yet, but I really need a feature in 13 ( for_each and count in modules). There do not appear to be any errors, which I find strange.

versions.tf

terraform {
  required_version = ">= 0.13"

  required_providers {
    citrixadc = {
      source = "amfam/afi/citrixadc"
      version = "0.12.40"
    }
  }
}

provider.tf

provider "citrixadc" {
  required_providers {
    citrixadc = {
      source = "amfam/afi/citrixadc"
      version = "0.12.40"
    }
  }

  insecure_skip_verify = "true"
}

Location of the local plugin:

# find .terraform -name terraform-provider-citrixadc*
.terraform/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64/terraform-provider-citrixadc_v0.12.40_x1

Terraform init output and TRACE info:

Terraform v0.13.6
+ provider registry.terraform.io/terraform-providers/infoblox v1.1.0

Providers required by configuration:
.
├── provider[amfam/afi/citrixadc] 0.12.40
└── module.vip
    └── provider[registry.terraform.io/terraform-providers/infoblox]

terraform init
Upgrading modules...
Downloading git::ssh://git@gitlab.com/amfament/amfam/cstiac/amfam-tf-mod-infoblox.git?ref=develop for vip...
- vip in /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules/vip

Initializing the backend...

Initializing provider plugins...
- Finding amfam/afi/citrixadc versions matching "0.12.40"...
- Finding latest version of terraform-providers/infoblox...
- Using terraform-providers/infoblox v1.1.0 from the shared cache directory

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
amfam/afi/citrixadc: could not connect to amfam: Failed to request discovery
document: Get "https://amfam/.well-known/terraform.json": dial tcp: lookup
amfam on 10.65.255.10:53: no such host

2021-03-02 08:39:06 [ERROR] 'terraform init' Failed!

2021/03/02 08:39:01 [INFO] Terraform version: 0.13.6  
2021/03/02 08:39:01 [INFO] Go runtime version: go1.14.7
2021/03/02 08:39:01 [INFO] CLI args: []string{"/Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/installs/terraform", "init", "-input=false", "-reconfigure", "-upgrade=true"}
2021/03/02 08:39:01 [DEBUG] Attempting to open CLI config file: /Users/BMP010/.terraformrc
2021/03/02 08:39:01 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021/03/02 08:39:01 [DEBUG] checking for credentials in "/Users/BMP010/.terraform.d/plugins"
2021/03/02 08:39:01 [DEBUG] will search for provider plugins in terraform.d/plugins
2021/03/02 08:39:01 [TRACE] getproviders.SearchLocalDirectory: terraform.d/plugins is a symlink to terraform.d/plugins
2021/03/02 08:39:01 [DEBUG] will search for provider plugins in /Users/BMP010/.terraform.d/plugins
2021/03/02 08:39:01 [TRACE] getproviders.SearchLocalDirectory: /Users/BMP010/.terraform.d/plugins is a symlink to /Users/BMP010/.terraform.d/plugins
2021/03/02 08:39:01 [DEBUG] ignoring non-existing provider search directory /Users/BMP010/Library/Application Support/io.terraform/plugins
2021/03/02 08:39:01 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021/03/02 08:39:01 [INFO] CLI command args: []string{"init", "-input=false", "-reconfigure", "-upgrade=true"}
2021/03/02 08:39:01 [TRACE] ModuleInstaller: installing child modules for . into /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules
2021/03/02 08:39:01 [DEBUG] Module installer: begin vip
2021/03/02 08:39:01 [TRACE] ModuleInstaller: discarding previous record of vip prior to reinstall
2021/03/02 08:39:01 [TRACE] ModuleInstaller: cleaning directory /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules/vip prior to install of vip
2021/03/02 08:39:01 [TRACE] ModuleInstaller: vip address "git::ssh://git@gitlab.com/amfament/amfam/cstiac/amfam-tf-mod-infoblox.git?ref=develop" will be handled by go-getter
2021/03/02 08:39:01 [DEBUG] will download "git::ssh://git@gitlab.com/amfament/amfam/cstiac/amfam-tf-mod-infoblox.git?ref=develop" to /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules/vip
2021/03/02 08:39:01 [TRACE] fetching "git::ssh://git@gitlab.com/amfament/amfam/cstiac/amfam-tf-mod-infoblox.git?ref=develop" to "/Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules/vip"
2021/03/02 08:39:05 [TRACE] ModuleInstaller: vip "git::ssh://git@gitlab.com/amfament/amfam/cstiac/amfam-tf-mod-infoblox.git?ref=develop" was downloaded to /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules/vip
2021/03/02 08:39:05 [DEBUG] Module installer: vip installed at /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules/vip
2021/03/02 08:39:05 [TRACE] modsdir: writing modules manifest to /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/modules/modules.json
2021/03/02 08:39:05 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2021/03/02 08:39:05 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2021/03/02 08:39:05 [DEBUG] New state was assigned lineage "edbdf517-8b7c-c36c-7258-1f30c2f76629"
2021/03/02 08:39:05 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2021/03/02 08:39:05 [TRACE] Meta.Backend: instantiated backend of type <nil>
2021/03/02 08:39:05 [TRACE] providercache.fillMetaCache: scanning directory /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins
2021/03/02 08:39:05 [TRACE] getproviders.SearchLocalDirectory: /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins is a symlink to /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins
2021/03/02 08:39:05 [TRACE] getproviders.SearchLocalDirectory: found amfam/afi/citrixadc v0.12.40 for darwin_amd64 at /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64
2021/03/02 08:39:05 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/terraform-providers/infoblox v1.1.0 for darwin_amd64 at /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64
2021/03/02 08:39:05 [TRACE] providercache.fillMetaCache: including /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64 as a candidate package for amfam/afi/citrixadc 0.12.40
2021/03/02 08:39:05 [TRACE] providercache.fillMetaCache: including /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64 as a candidate package for registry.terraform.io/terraform-providers/infoblox 1.1.0
2021/03/02 08:39:05 [DEBUG] checking for provisioner in "."
2021/03/02 08:39:05 [DEBUG] checking for provisioner in "/Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/installs"
2021/03/02 08:39:05 [DEBUG] checking for provisioner in "terraform.d/plugins/darwin_amd64"
2021/03/02 08:39:05 [DEBUG] checking for provisioner in "/Users/BMP010/.terraform.d/plugins"
2021/03/02 08:39:05 [INFO] Failed to read plugin lock file /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/darwin_amd64/lock.json: open /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/darwin_amd64/lock.json: no such file or directory
2021/03/02 08:39:05 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2021/03/02 08:39:05 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2021/03/02 08:39:05 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2021/03/02 08:39:05 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2021/03/02 08:39:05 [TRACE] statemgr.Filesystem: read nil snapshot
2021/03/02 08:39:05 [TRACE] providercache.fillMetaCache: scanning directory /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins
2021/03/02 08:39:05 [TRACE] getproviders.SearchLocalDirectory: /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins is a symlink to /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins
2021/03/02 08:39:05 [TRACE] getproviders.SearchLocalDirectory: found amfam/afi/citrixadc v0.12.40 for darwin_amd64 at /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64
2021/03/02 08:39:05 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/terraform-providers/infoblox v1.1.0 for darwin_amd64 at /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64
2021/03/02 08:39:05 [TRACE] providercache.fillMetaCache: including /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64 as a candidate package for amfam/afi/citrixadc 0.12.40
2021/03/02 08:39:05 [TRACE] providercache.fillMetaCache: including /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64 as a candidate package for registry.terraform.io/terraform-providers/infoblox 1.1.0
2021/03/02 08:39:05 [DEBUG] Service discovery for amfam at https://amfam/.well-known/terraform.json
2021/03/02 08:39:05 [TRACE] HTTP client GET request to https://amfam/.well-known/terraform.json
2021/03/02 08:39:05 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2021/03/02 08:39:05 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2021/03/02 08:39:06 [DEBUG] GET https://registry.terraform.io/v1/providers/terraform-providers/infoblox/versions
2021/03/02 08:39:06 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/terraform-providers/infoblox/versions
2021/03/02 08:39:06 [TRACE] providercache.fillMetaCache: scanning directory /Users/BMP010/.tfm/cache
2021/03/02 08:39:06 [TRACE] getproviders.SearchLocalDirectory: /Users/BMP010/.tfm/cache is a symlink to /Users/BMP010/.tfm/cache
2021/03/02 08:39:06 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/terraform-providers/infoblox v1.1.0 for darwin_amd64 at /Users/BMP010/.tfm/cache/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64
2021/03/02 08:39:06 [TRACE] providercache.fillMetaCache: including /Users/BMP010/.tfm/cache/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64 as a candidate package for registry.terraform.io/terraform-providers/infoblox 1.1.0
2021/03/02 08:39:06 [TRACE] providercache.Dir.LinkFromOtherCache: linking registry.terraform.io/terraform-providers/infoblox v1.1.0 from existing cache /Users/BMP010/.tfm/cache/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64 to /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64
2021/03/02 08:39:06 [TRACE] providercache.fillMetaCache: scanning directory /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins
2021/03/02 08:39:06 [TRACE] getproviders.SearchLocalDirectory: /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins is a symlink to /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins
2021/03/02 08:39:06 [TRACE] getproviders.SearchLocalDirectory: found amfam/afi/citrixadc v0.12.40 for darwin_amd64 at /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64
2021/03/02 08:39:06 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/terraform-providers/infoblox v1.1.0 for darwin_amd64 at /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64
2021/03/02 08:39:06 [TRACE] providercache.fillMetaCache: including /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64 as a candidate package for amfam/afi/citrixadc 0.12.40
2021/03/02 08:39:06 [TRACE] providercache.fillMetaCache: including /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins/registry.terraform.io/terraform-providers/infoblox/1.1.0/darwin_amd64 as a candidate package for registry.terraform.io/terraform-providers/infoblox 1.1.0
2021/03/02 08:39:06 [TRACE] providercache.fillMetaCache: using cached result from previous scan of /Users/BMP010/git-repositories/IaC/terraform-v12/amfam-tf-mod-netscaler/.terraform/plugins

So near as I can tell, it is seeing my plugin and marking it as a candidate, but there are no other messages in the trace output. The only things I can think of at this point:

The provider isn’t compatible with TF 13 and is silently being discarded as a candidate
I’m doing somethig wrong that isn’t evident to me.

alisdair · March 2, 2021, 3:25pm

Huh, those logs are confusing. I can’t see exactly why this isn’t working, but I have a couple of suggestions.

We don’t recommend installing local copies of providers in the per-project .terraform/ directory. This is considered a local cache to be managed by Terraform. Instead, please try storing your provider under ~/.terraform.d/plugins (noted in the “New Filesystem Layout for Local Copies of Providers” section of the upgrade guide), with the same subdirectory tree you already have.

If you still see the same error, you could try explicitly directing Terraform to use a local mirror location for this provider using a CLI config provider_installation block. This should cause Terraform to look for your provider there instead of in the global implicit cache.

I hope one of these options works for you!

piontekdd · March 2, 2021, 3:55pm

That worked. I swear I tried this earlier. Thanks.
But … there’s always a but

All of our terraform jobs run in Jenkins where the directory structures are created and deleted every time. In previous versions, we were installing the plugins at runtime in /terraform.d/plugins and that worked great. I need to mimic this same behavior in TF 13 to keep the jobs stateless.

It looks like we can still use this folder. I switched to using /terraform.d like we were before, but with the correct/new directory structure and removed from $HOME/.terraform.d/plugins and it looks to be picking things up from the directory now.

2021/03/02 09:59:10 [TRACE] getproviders.SearchLocalDirectory: found amfam/afi/citrixadc v0.12.40 for darwin_amd64 at terraform.d/plugins/amfam/afi/citrixadc/0.12.40/darwin_amd64

apparentlymart · March 2, 2021, 4:55pm

If you’d like to have a separate local mirror directory for each of your Jenkins jobs you can use the explicit provider installation configuration that @alisdair mentioned to override Terraform’s default search locations.

For example, you could have your Jenkins run include generating a .tfrc (CLI configuration) file in the current directory as its first step, along with copying the providers into place:

echo >"$WORKSPACE/workspace.tfrc" <<EOF
provider_installation {
  filesystem_mirror {
    path    = "${WORKSPACE}/terraform-providers"
    include = ["amfam/*/*"]
  }
  direct {
    exclude = ["amfam/*/*"]
  }
}
EOF

(the WORKSPACE references above are intended to be using the Jenkins-provided environment variable which refers to the root of the current workspace. You might have a different systematic way to assign these paths; adjust as needed.)

Then when you run Terraform, set the environment variable TF_CLI_CONFIG_FILE="${WORKSPACE}/workspace.tfrc" to tell Terraform to use this workspace-specific CLI configuration file instead of looking in the usual “global” (really: system-user-specific) locations.

Unfortunately having external software directly modify the .terraform directory contents was never an intentional integration point; Terraform uses that directory as a cache for various purposes and expects to be managing it directly itself. The implementation details of earlier versions seem to have made it seem to work as a local mirror directory, but that wasn’t an intended feature. However, you can use mechanisms like I’ve described above to influence where Terraform looks when it’s doing the operations in terraform init that populate the .terraform directory, and thus get a similar effect to what you had on prior Terraform versions.

boredmonkey · May 23, 2022, 9:28pm

I’m unable to get this thing working with ${WORKSPACE} env variable in Jenkins and found this issue related to env vars terraform rc file Allow setting filesystem_mirror paths relative to the current user's home directory · Issue #27446 · hashicorp/terraform · GitHub. Can you please help me to fix this ?

maxb · May 23, 2022, 11:12pm

The suggested code above uses the bash shell to expand the variable before it is written to the tfrc file at all. If you have accurately followed the above recipe, there will be an absolute path in the tfrc file and Terraform will simply be reading an absolute path as a literal string.

Topic		Replies	Views
Solved: Local provider discovery failing with 0.13 Terraform	2	944	August 25, 2020
0.13-rc1 and local providers Terraform	2	498	November 8, 2021
Terraform v0.13 discovery of providers in the local filesystem Terraform	2	1042	July 27, 2020
Having issue to install a custom provider using provider framework Plugin Development	1	607	November 16, 2022
Terraform provider required Terraform	0	232	January 21, 2021

Local Provider Installation not working in Terraform 13

Related topics