We’re running Terraform in a GitLab CI/CD pipeline on a self hosted instance. In the pipeline, it is not possible for Terraform to download providers on the fly. Because of that, we’d like to pre-provide providers in some directory.
Following these guidelines terraform init command has been set to run: terraform init -input=false -plugin-dir=/opt/terraform/plugins
But — it fails
+ terraform init -input=false -plugin-dir=/opt/terraform/plugins
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
Downloading git::https://company/terraform/modules/kvm.git for kvm...
- kvm in .terraform/modules/kvm
Initializing provider plugins...
- Finding dmacvicar/libvirt versions matching "0.7.6"...
- Finding hashicorp/dns versions matching "3.4.0"...
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ dmacvicar/libvirt: provider registry.terraform.io/dmacvicar/libvirt was not
│ found in any of the search locations
│
│ - /opt/terraform/plugins
╵
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/dns: provider registry.terraform.io/hashicorp/dns was not found
│ in any of the search locations
│
│ - /opt/terraform/plugins
╵
Contents of /opt/terraform/plugins are like this:
+ ls -lR /opt/terraform/plugins
/opt/terraform/plugins:
total 34584
drwxr-xr-x. 3 root root 23 Apr 15 12:38 registry.terraform.io
-rwxr-xr-x. 1 root root 18059264 Dec 12 15:32 terraform-provider-dns_v3.4.0_x5
-rwxr-xr-x. 2 root root 17326080 Nov 20 00:29 terraform-provider-libvirt_v0.7.6
/opt/terraform/plugins/registry.terraform.io:
total 0
drwxr-xr-x. 3 root root 21 Apr 15 12:38 dmacvicar
/opt/terraform/plugins/registry.terraform.io/dmacvicar:
total 0
drwxr-xr-x. 2 root root 47 Apr 15 12:38 libvirt
/opt/terraform/plugins/registry.terraform.io/dmacvicar/libvirt:
total 16920
-rwxr-xr-x. 2 root root 17326080 Nov 20 00:29 terraform-provider-libvirt_v0.7.6
$ terraform -version
Terraform v1.5.7
on linux_amd64
The new structure you described in your second comment is using what the documentation refers to as the “packed layout”, where the packages are represented using their original zip files directly instead of using the contents of those zip files.
Terraform accepts that form but note that it means Terraform will re-extract that zip file each time you use terraform init, because Terraform ultimately needs to be able to run the executable inside.
I’m not entirely sure why your original attempt didn’t work, but I imagine there’s something slightly incorrect about the directory layout you created that is preventing Terraform from understanding it as the “unpacked” layout. Running terraform init with the environment variable TF_LOG=trace will cause Terraform to produce the internal logs that Terraform developers use for debugging, which might give some more clues about how Terraform understood your directory and why it concluded that no packages were available.
I removed it now, but I still get the exact same error.
And I’m now on Terraform v1.8.0.
+ ls -lR /opt/terraform/plugins
/opt/terraform/plugins:
total 34584
-rw-r--r--. 1 root root 4892 Nov 20 00:32 CHANGELOG.md
-rw-r--r--. 1 root root 10126 Nov 20 00:32 LICENSE
-rw-r--r--. 1 root root 6605 Nov 20 00:32 README.md
-rwxr-xr-x. 1 root root 18059264 Dec 12 15:32 terraform-provider-dns_v3.4.0_x5
-rwxr-xr-x. 1 root root 17326080 Nov 20 00:29 terraform-provider-libvirt_v0.7.6
+ env TF_LOG=trace terraform init -input=false -plugin-dir=/opt/terraform/plugins
2024-04-16T12:46:50.487+0200 [INFO] Terraform version: 1.8.0
2024-04-16T12:46:50.487+0200 [DEBUG] using github.com/hashicorp/go-tfe v1.41.0
2024-04-16T12:46:50.487+0200 [DEBUG] using github.com/hashicorp/hcl/v2 v2.20.0
2024-04-16T12:46:50.487+0200 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-04-16T12:46:50.487+0200 [DEBUG] using github.com/zclconf/go-cty v1.14.3
2024-04-16T12:46:50.487+0200 [INFO] Go runtime version: go1.22.1
2024-04-16T12:46:50.487+0200 [INFO] CLI args: []string{"terraform", "init", "-input=false", "-plugin-dir=/opt/terraform/plugins"}
2024-04-16T12:46:50.487+0200 [TRACE] Stdout is not a terminal
2024-04-16T12:46:50.487+0200 [TRACE] Stderr is not a terminal
2024-04-16T12:46:50.487+0200 [TRACE] Stdin is not a terminal
2024-04-16T12:46:50.487+0200 [DEBUG] Attempting to open CLI config file: /home/libvirt2821/.terraformrc
2024-04-16T12:46:50.487+0200 [INFO] Loading CLI configuration from /home/libvirt2821/.terraformrc
2024-04-16T12:46:50.488+0200 [DEBUG] Explicit provider installation configuration is set
2024-04-16T12:46:50.488+0200 [TRACE] Selected provider installation method cliconfig.ProviderInstallationDirect with includes [] and excludes [registry.terraform.io/*/*]
2024-04-16T12:46:50.488+0200 [TRACE] Selected provider installation method cliconfig.ProviderInstallationNetworkMirror("https://artifactory.company.tld/artifactory/api/terraform/terraform-remote/providers/") with includes [] and excludes []
2024-04-16T12:46:50.488+0200 [INFO] CLI command args: []string{"init", "-input=false", "-plugin-dir=/opt/terraform/plugins"}
2024-04-16T12:46:50.492+0200 [TRACE] Meta.Backend: built configuration for "http" backend with hash value 3307601501
2024-04-16T12:46:50.492+0200 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2024-04-16T12:46:50.492+0200 [DEBUG] New state was assigned lineage "498ca57b-86a1-b82f-5713-8a4eee6dca66"
2024-04-16T12:46:50.492+0200 [TRACE] Meta.Backend: moving from default local state only to "http" backend
Initializing the backend...
2024-04-16T12:46:50.492+0200 [DEBUG] checking for provisioner in "/opt/terraform/plugins"
2024-04-16T12:46:50.492+0200 [TRACE] backend/local: state manager for workspace "default" will:
- read initial snapshot from terraform.tfstate
- write new snapshots to terraform.tfstate
- create any backup at terraform.tfstate.backup
2024-04-16T12:46:50.492+0200 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2024-04-16T12:46:50.492+0200 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2024-04-16T12:46:50.492+0200 [TRACE] statemgr.Filesystem: read nil snapshot
2024-04-16T12:46:50.492+0200 [TRACE] Meta.Backend: ignoring local "default" workspace because its state is empty
2024-04-16T12:46:50.493+0200 [DEBUG] New state was assigned lineage "7403f958-ed9d-cf3f-6cc0-1022837fe76f"
2024-04-16T12:46:50.494+0200 [TRACE] Preserving existing state lineage "7403f958-ed9d-cf3f-6cc0-1022837fe76f"
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2024-04-16T12:46:50.496+0200 [TRACE] Meta.Backend: instantiated backend of type *http.Backend
2024-04-16T12:46:50.496+0200 [DEBUG] checking for provisioner in "/opt/terraform/plugins"
2024-04-16T12:46:50.496+0200 [TRACE] Meta.Backend: backend *http.Backend does not support operations, so wrapping it in a local backend
2024-04-16T12:46:50.496+0200 [DEBUG] GET https://scm.company.tld/api/v4/projects/7261/terraform/state/vm
2024-04-16T12:46:50.516+0200 [ERROR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=4670b4da-900f-1457-24d5-4f3c0d53fad1&version=1.8.0": dial tcp 18.165.183.63:443: connect: network is unreachable
2024-04-16T12:46:50.670+0200 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
Initializing modules...
2024-04-16T12:46:50.671+0200 [DEBUG] Module installer: begin my_mod
2024-04-16T12:46:50.671+0200 [TRACE] ModuleInstaller: my_mod is not yet installed
2024-04-16T12:46:50.671+0200 [TRACE] ModuleInstaller: cleaning directory .terraform/modules/my_mod prior to install of my_mod
2024-04-16T12:46:50.671+0200 [TRACE] ModuleInstaller: my_mod address "git::https://scm.company.tld/tci/tools/terraform/modules/my_mod.git?depth=1&ref=v3.0.0" will be handled by go-getter
2024-04-16T12:46:50.671+0200 [TRACE] getmodules: fetching "git::https://scm.company.tld/tci/tools/terraform/modules/my_mod.git?depth=1&ref=v3.0.0" to ".terraform/modules/my_mod"
Downloading git::https://scm.company.tld/tci/tools/terraform/modules/my_mod.git?depth=1&ref=v3.0.0 for my_mod...
2024-04-16T12:46:51.449+0200 [TRACE] ModuleInstaller: my_mod "git::https://scm.company.tld/tci/tools/terraform/modules/my_mod.git?depth=1&ref=v3.0.0" was downloaded to .terraform/modules/my_mod
- my_mod in .terraform/modules/my_mod
2024-04-16T12:46:51.454+0200 [DEBUG] Module installer: my_mod installed at .terraform/modules/my_mod
2024-04-16T12:46:51.454+0200 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json
2024-04-16T12:46:51.460+0200 [DEBUG] init: overriding provider plugin search paths
2024-04-16T12:46:51.460+0200 [DEBUG] will search for provider plugins in [/opt/terraform/plugins]
Initializing provider plugins...
- Finding dmacvicar/libvirt versions matching "0.7.6"...
- Finding hashicorp/dns versions matching "3.4.0"...
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ dmacvicar/libvirt: provider registry.terraform.io/dmacvicar/libvirt was not
│ found in any of the search locations
│
│ - /opt/terraform/plugins
╵
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/dns: provider registry.terraform.io/hashicorp/dns was not found
│ in any of the search locations
│
│ - /opt/terraform/plugins
╵
Looking again at your original directory listings today I think I see the problem: you are missing the path segment that specifies which version of the provider the package is for.
The “unpacked layout” expects the following directory structure:
HOSTNAME/NAMESPACE/TYPE/VERSION/TARGET
…and that’s the path to the directory containing the contents of the provider’s distribution zip file, so there would be at least one executable file under that directory.
So for that libvirt plugin in particular, the correct path for the plugin executable would be:
$ ls -lah /usr/lib/custom-terraform-plugins
-rwxrwxr-x 1 user user 84M Jun 13 15:13 terraform-provider-aws-v1.0.0-x3
-rwxrwxr-x 1 user user 84M Jun 13 15:15 terraform-provider-rundeck-v2.3.0-x3
-rwxrwxr-x 1 user user 84M Jun 13 15:15 terraform-provider-mysql-v1.2.0-x3
And, yes, with that, it DOES work. Only having /opt/terraform/plugins/registry.terraform.io/dmacvicar/libvirt/terraform-provider-libvirt_v0.7.6 was not sufficient.
(Yes, it now failed for dns, but that’s okay. It would work with proper directories.)
The documentation I referred to when I was writing my answer was this one:
The doc you were referring to unfortunately seems to be very stale and is describing how Terraform CLI behaved in Terraform v0.12 and earlier, before there was any support for non-HashiCorp providers. I guess it was missed because that guide is not specifically about provider mirrors but only mentions them as a side-topic. It would probably be better for the guide to just mention the idea of mirrors and then link to the main docs on this topic that I linked above.
That libvirt provider seems to have some other files in its package in addition to just the plugin executable. If that’s true then you should extract the entire zip file into the leaf directory, because otherwise a checksum calculated from the mirror directory won’t match a checksum calculated from the official package, and so you will probably run into problems with dependency lock file mismatches.
(Since the system where you are running Terraform is configured to never use the origin registry that one can’t possibly learn the checksum of the official package anyway, but having an incorrect checksum in the lock file will cause problems if you use this Terraform configuration in a different location later, such as in a local development environment. So less confusing all round to get it right from the start, even if your segregated execution environment won’t care about it today.)
See below.