Terraform 13/14: fails to download newer provider version if older version exists locally

This is a followup on the issue I created in Terraform 13/14: fails to download newer provider version if older version exists locally · Issue #28818 · hashicorp/terraform · GitHub.

I have a older version of google provider (3.64.0) locally $HOME/.terraform.d/plugins/registry.terraform.io/hashicorp/google. I created a new terraform configuration as follows

terraform {
required_providers {
google = {
source = “hashicorp/google”
version = “3.69.0”
}
}
required_version = “>= 0.13”
}

When I execute terraform init I see the following error

Error:
Failed to query available provider packages
Could not retrieve the list of available versions for provider hashicorp/google: no available releases match the given constraints 3.69.0

I believe this might be an issue with the way terraform engine code computes the excludes if the provider exists locally and fails to download the newer version. On T11/12 this used to work fine by downloading the new version under $HOME/.terraform.d/plugins

If I add the following cli configuration on init the new version does get downloaded but will get downloaded even if I have a version locally which is a waste of network traffic.

provider_installation {
direct {
}
}

Has anyone faced this issue? If so can you pls share how you resolved it. Thank you for your time.

Hi @bhadrim,

This behavior has been one where different users have being surprised by the behavior either way: as you say, in older versions Terraform would sometimes (if the newly-requested version doesn’t match the version constraint) ignore what was available locally and go download a new version, which several folks found confusing because they believed that they’d clearly told Terraform to never access the network for that particular plugin.

Consequently, Terraform’s current default behavior is that if you have a provider installed into one of the implied mirror directories then Terraform will assume that you only intend to use local versions of that provider, e.g. because you’ve modified it in some way compared to the upstream and you only want to use your own build.

There are a few ways you can customize this behavior. One way is to continue on the path you were already considering and explicitly tell Terraform to consult both the local mirror directory and origin registries:

provider_installation {
  filesystem_mirror {
    path = "/home/yourself/.terraform.d/plugins"
  }
  direct {}
}

This is different from the automatic configuration Terraform will build for itself in the absense of any explicit configuration, based only on what you have in your implied mirror directory:

provider_installation {
  filesystem_mirror {
    path    = "/home/yourself/.terraform.d/plugins"
    include = ["hashicorp/google"]
  }
  direct {
    exclude = ["hashicorp/google"]
  }
}

Because the first explicit example above doesn’t include those include and exclude configurations that Terraform implicitly inserts when doing automatic behavior, it’ll consult both the local directory and the origin registry and choose whichever one has the newest version matching the constraint.

However, you also mentioned that you are concerned about “wasting network traffic”, which makes me think that your main motivation here might be to avoid re-downloading the same plugins multiple times. In that case, you might be better served by leaving the provider installation methods entirely unmodified (no provider_installation block or local mirror directories at all) and instead enabling a plugin cache directory:

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

If you add that configuration (and create the directory it mentions) then terraform init will treat that directory as a read-through cache for any plugins it needs to install, regardless of the underlying installation method. That means that the first time you select a new version it’ll download it into the cache and then link it from the cache into your working directory. On subsequent installs of the same provider it’ll notice that the cache is already populated and just re-use the existing package directory from local disk.

Once you’re on a newer Terraform version that uses an explicit dependency lock file, both the local mirror technique and the local cache technique unfortunately currently prevent Terraform’s automatic verification of the full set of checksums for a provider already in the cache, because that mechanism requires a network request to the origin registry. Therefore you may need to use terraform providers lock to write multiple platform-specific checksums into the lock file if your team uses Terraform across more than one operating system and CPU architecture.

@ apparentlymart

Hi, Thank you for your reply and the solution. I tried out the following filesystem_mirror solution and this works like a peach if I explicitly set the path.

provider_installation {
  filesystem_mirror {
    path = "/home/terraform/.terraform.d/plugins"
  }
  direct {}
}

But when I use ~/.terraform.d/plugins or $HOME/.terraform.d/plugins the engine fails to find the path. I get the following error. Should the path be set explicitly?
(Note: in plugin_cache_dir we use $HOME/.terraform.d/plugin-cache and that seems to work).

Could not retrieve the list of available versions for provider
hashicorp/google: cannot search $HOME/.terraform.d/plugins: lstat
$HOME/.terraform.d/plugins: no such file or directory

Thank you again for your time.

Hi @bhadrim,

Indeed, newer parts of the CLI configuration format don’t tend to support environment variable substitution like that, because the way that was implemented is incompatible with the modern HCL syntax. CLI configuration remains using the old HCL syntax from Terraform v0.11 and earlier pending some solution to deal with that problem, and so in order to avoid exacerbating the problem further we’ve not introduced any new examples of it.

With that said then, in Terraform as it exists today you’ll need to include the absolute path to the filesystem mirror directory. Perhaps a later version of Terraform will introduce a new approach for referring to environment variables, once we have a solution to how to migrate the format to the new HCL grammar.